Comments:"LEMON – Money Trees (Rap Genius Response to Heroku) | Rap Genius"
URL:http://rapgenius.com/Lemon-money-trees-rap-genius-response-to-heroku-lyrics
Click colored words to read annotations. Highlight white words to create annotations
The Rap Genius family appreciates Heroku’s extensive apology and technical explanation of the queuing problems its Rails customers are facing. This is a solid first step towards rectifying this situation – but it is only a first step. In particular, Rap Genius thinks that Heroku's customers would appreciate if Heroku went into more depth about what they knew about the problem over the last few years. In addition, Heroku should offer its customers at least some of their money back. We love Heroku, they allowed us to focus on building an amazing site, they hooked us up, but they promised a level of service they did not provide.
Heroku claims that though they received reports of unexplained latency over the past couple of years, they weren’t able to figure out the request queuing issue until they read the Rap Genius article. But there is evidence that Heroku knew about the problem for more than 2 years.
On February 16th 2011, Tim Watson sent an email to the Heroku Google Group with the subject “H12 errors, blocked dynos and Heroku website claims”:
The Heroku website claims:http://heroku.com/how/dyno_grid_last#3 "If a dyno is unresponsive for any reason (user bugs, long requests, or high load), other requests will be routed around it."In my experience, this does not seem to be the case. We have several admin features in our app that when requested with certain params, it can take longer then 30s to run. (I am working on ways to get these in check and in the background). When a user trips one of these long running requests, Heroku appears to queue additional requests to this dyno and those requests time out, even though there are plenty of other dynos available to handle that request.Is the statement on the Heroku website true or false? It does not appear that Heroku actively monitors the dynos to see if they are busy with a long running request. Is there a better way to handle this situation? 2 of Heroku’s top brass responded. First Oren Teich, Heroku’s GM & COO: the timeout is only on the routing side.If you have 2 dynos, and 1 is running "forever", then 50% of your requests to your app will timeout.This is expected behavior on Heroku today.I strongly encourage you to use the rack-timeout gem to ensure that your dyno terminates after 30 seconds so you don't get this odd behavior.-- Oren Teich Clearly Oren knew in 2011 that Bamboo’s router was routing requests randomly. If Bamboo’s router were intelligent, 0% of requests would time out in his hypothetical because all requests would be routed to the free dyno.Tim replied:
I am quite surprised this is expected behavior.The documentation on the site is now out of date and is highly misleading. The entire 2 paragraphs of the backlog section are untrue. ( http://devcenter.heroku.com/articles/key-concepts-performance ) The documentation on the site is now out of date and is highly misleading– this is the point at which I would have expected Heroku to investigate the problem, disclose it to customers (including Rap Genius – we didn't find out until years later), and apologize. Instead, Adam Wiggins (Heroku’s CTO) wrote: Tim,You're correct, the routing mesh does not behave in quite the way described by the docs. We're working on evolving away from the global backlog concept in order to provide better support for different concurrency models, and the docs are no longer accurate. The current behavior is not ideal, but we're on our way to a new model which we'll document fully once it's done.In the meantime, you shouldn't have any difficulties as long as you keep your web requests short (less than about 500ms), which is good practice anyway.Sorry for any difficulty or confusion, and thanks for digging in and providing such a detailed analysis.Adam To summarize:• In February 2011, Heroku’s COO and CTO both admitted that the Bamboo router was random and that their documentation was incorrect – i.e., they both admitted that Heroku was misleading customers about its capabilities.
• I think it’s reasonable to assume that Adam and Oren both knew a random router would perform worse than an intelligent router (especially given that at the time, 100% of Heroku's apps were single-threaded Rails apps)
• Therefore, in February 2011 (and for the following 2 years), Heroku’s COO and CTO both knew that Heroku’s documentation incorrectly overstated the abilities of the platform and yet they said nothing about it to their customers until Rap Genius took the issue public.
If Heroku wanted to find out how bad the problem was, they could have:
• Simulated both routing algorithms as Rap Genius did.Now, as Rap Genius is widely known for its expertise in queuing theory, But they could have figured it out eventually and the results would have been devastating
• Observed the performance of Bamboo applications directly. You can’t do this with Heroku’s out-of-the-box logs or add-ons because for some reason they all ignore time spent queuing at the Dyno level, but it’s not hard to change this behavior. In fact, Rap Genius just released a gem that makes New Relic display the actual time your app spends queuing rather than always displaying 0. Heroku could have done this too, and had they done so, the extent of the problem would have been clear immediately
First see https://help.heroku.com/tickets/75238 (logged on 2/5/2013), the ticket in which I reacted to Heroku engineer's comment to me on the phone that Rap Genius requests were queuing at the dyno level. I wrote:
after this phone call with [redacted] I must admit I was pretty shocked that the routing mesh isn't smarter – i.e., why does it send requests to dynos that are already handling requests when there are free dynos available?[...]when I signed up for heroku the behavior was different (and the old behavior was one of the reasons I signed up!!)SO: why did you guys change the behavior here to something that is clearly worse for your users? how can I return to this old behavior? In response, Adam Wiggins (Heroku’s CTO) reached out to me privately in an email (click here to read our entire correspondence). On 2/5/2013 (more than a week before we published the article), he wrote: You're correct that we've made some product decisions over the past few years that have evolved our HTTP routing layer away from the "intelligent routing" approach that we used in 2009. This evolution was based on evidence of behavior and performance of our customer's high-traffic apps. There are a lot of reasons, but the two big ones are 1) the "intelligent routing" doesn't scale, since it relies on distribution locking which effectively destroys parallelism, and 2) it's incompatible with the evented and realtime apps which are increasingly common on the modern web. "Intelligent routing" sounds good, but in the end it wasn't good for our customers. Following this email, I documented for Adam both our simulation results and the many inaccuracies in Heroku’s docs, revealing to him all the same information that went into our final article. Here was Adam’s response: Tom,Thanks for the detailed breakdown of the challenges you're running up against. Getting user perspective is very helpful and I'll apply your feedback as we continue to evolve our product.I'm convinced that the best path forward is for one of your developers to work closely with [redacted] to modernize and optimize your web stack. If you invest this time I think it's very likely you'll end up with an app that performs the way you want it to at a price within your budget.At this point I don't think there's much more I can contribute to the conversation, so I'm going to step out. [redacted] is your point man for technical questions; [redacted] can handle account and billing questions.If you have additional product feedback, please do send it my way. Thanks again for taking the time to speak with me, and I'll do my best to continue to improve the product in ways that serve your needs. Only after we got this response on 2/11/2013 did we realize the only way to resolve this issue would be to go public. We published our article 2 days later.What does Heroku owe its customers now?
Based on Heroku's response so far, it appears their approach to fixing the problem is, “We promised you intelligent routing, we delivered random routing (which is worse than intelligent routing), so we're going to change our documentation to make it clear that we're only promising you random routing.”Others might reply “Even though Bamboo customers didn't get what they paid for, they got something, and, well, where was the SLA saying exactly what that something should be?”
To which the first group would respond: “Sure we got something, but Heroku also TOOK from us without asking. We spent a bunch of time optimizing performance in the wrong places and people using our site got a worse experience than we understood, which caused us to lose users. Finally, Heroku has us locked in: because we've grown in traffic and application complexity, moving to another host is much more difficult now than it would have been 2 years ago”
One thing seems clear though, Heroku must refund the money its customers paid for the New Relic add-on (over the years Rap Genius paid Heroku a total of $63,116.13 for New Relic alone), since it reported incorrect data.
In Rap Genius’s case, this what New Relic showed:
We saw this and spent a ton of time optimizing our Ruby application code and database performance. However, here’s how New Relic should have looked:
