How much Traffic is too much Traffic for CloudFlare? - PhobosLab

Evidence suggests it's 100TB per month.

Before I go into the details I want to state two things first:

• CloudFlare generously provided most of the bandwidth for our site for a year, without any hiccups.
• We (unknowingly) violated their TOS. However, I was assured that was not the reason we were kicked.

So the reason I'm writing this is not because we were kicked (after all, CloudFlare was in the right to do so), but because of how shitty it went down.

For the last 6 years some friends and I have been operating this site that collects images posted in chat rooms on Quakenet IRC. The content gathered is somewhere between reddit and 4chan. It's purely a hobby project - I tried to monetize it through ads some time ago, but failed. Advertising-Networks don't want us as a customer, but I'm fine with that.

The site is pr0gramm.com (NSFW)

Until about a year ago, the site was running fine on a single dedicated server. However, bandwidth consumption was becoming a problem. At times the server's (unmetered) 100mbit/s connection was completely saturated. We've been growing quite a bit.

So we looked for a solution and stumbled upon CloudFlare. Their free plan was exactly what we needed. We didn't want any of the "speed" optimisations, but just their distributed loading to serve our images. It was extremely easy to set up and just worked. In disbelief I commented on HN that I don't understand how they could offer their service for free and that we'd happily pay a few hundred USD (screenshot, orig thread).

In that same comment thread I also expressed my concern that we're somehow using CloudFlare in a way that it was not intended for, but I was assured by CloudFlare's jgrahamc that it's fine. I was later told that jgrahamc's comment was made in error and that our usage was indeed against CloudFlare's TOS (Section 10: Limitation on non-HTML caching).

(Running the whole site through CloudFlare wouldn't make that much sense to us, because it's just one single HTML file, one CSS file and one JS file - about 70kb in total - everything else comes from a JSON API. We never load another HTML file. However, we would happily have enabled it for these files as well, if CloudFlare wanted us to. I suggested this on numerous occasions.)

Everything was running fine until Jan 22. 2013 when suddenly all images on our site didn't load. Opening an image URL directly showed that instead of the image, an HTML site was returned. This site is part of CloudFlare's DOS protection. It's an interstitial that intents to sort out bots.

However, we initially turned off CloudFlare's security settings, because a) if someone wanted to attack us, they'd send automated search requests to our backend and not try to load lots of images and b) CloudFlare just serves images; the interstitials they present would only be presented as broken images.

Logging in into CloudFlare I discovered that the security setting miraculously was changed from the lowest setting ("Essentially off") to the highest ("I'm under attack"). I never received any notification or explanation for the change. I switched it back and everything was running fine again, so I believed it was a fluke.

On Feb 1. CloudFlare completely disabled their service for us and sent all visitors directly to our main Server. The site was now unusably slow so I replaced it with a "Sorry, we'll be back soon" site.

When I logged in into CloudFlare, I was greeted by a notice:

CloudFlare has been temporarily disabled due to a system issue. To ensure there is no performance degradation to your website, we are temporarily routing all traffic directly to your server. Once peak performance is back, we will automatically re-enable CloudFlare.

Again, we were not notified about this by email or any other way.

I waited till the next day, then contacted CloudFlare, asking when we can expect the service to be back. They told us that it's not a "system issue", but that our site was under a "layer 7 attack":

Our operations team routed your site off CloudFlare because was seeing a large layer 7 attack that was negatively impacting other CloudFlare customers. They will review and re-enable the site automatically and no further contact is required.

CloudFlare never enabled our site again.

Curiously, at this time our CloudFlare stats boldly proclaimed "103TB traffic saved by CloudFlare in the last 30 days". Remember the site was disabled on Feb 1. It was the first month our traffic grew beyond 100TB, so I suspect that "layer 7 attack" is a sleazy term for "too much traffic".

On that weekend I talked to their support a bit more and was told that, based on our traffic, we need to be at least on the business plan ($200/mo) - which I would have agreed to - but was assured that someone else would talk to me again on Monday.

One week passed with no answer (I asked a few times), then, yesterday I was contacted again. They apologized for the delay and then told us:

At 100TB/mo., pure file delivery, you'd need to be an Enterprise customer. Let me know if this works within your budget.

The Enterprise plan comes at $3000/mo.

So CloudFlare disabled our site because of a "layer 7 attack", then let us in limbo for two weeks, where we couldn't commit to another solution, only to tell us we need to pay $3000/mo in the end.

We now ordered two servers from Leaseweb in NL, each with 1Gbit uplink and 100TB traffic included to run varnish caches and serve the images. These cost us $200/mo in total.