The Twit Cleaner - Twitter? It’s Not Fun Anymore

This week I worked 62 hours straight. Actually, that’s a lie, I worked 65 hours straight, with 3 hours of naps.

Twitter has stopped being fun.

I started Twit Cleaner in 2009. Given that I haven’t managed to develop a livable income out of it in that time (barring donations from you many wonderful, generous people), it’s pretty obvious I’m not in it for the money. It’s been my primary occupation over that time, and absorbed several thousand hours of coding, designing, building, testing and supporting it. Everything on Twit Cleaner (bar some low level server decisions), I’ve personally done.

I wanted to make Twitter a better place for everyone. I love to code. I love helping people. That’s it.

Unfortunately, the new API is so crippling that Twit Cleaner is unable to continue. If I can’t even maintain the base service, I certainly can’t (won’t) roll out any new functionality I had planned, all of which sits on top of Twitter’s APIs.

So, here’s some things I had in the pipeline that won’t see the light of day:

Filtered follow back– like auto-follow, but filtered through the (user customisable) Twit Cleaner reports , so if they would appear on (your choice of) report sections, you won’t follow them; only if they’re awesome. I’ve had this running on my personal account for a year or so, it’s just needed some UI whipped up around it. Oh, and a billing system.

DM guard– deleting the myriad of DM spam we all receive. Making Direct Messages useful again. Additionally, catching and quarantining all the phishing attacks Twitter suffers, and feeding those back en masse to Twitter, so they can break the links and quickly quell any new attacks that arise, before they spread. This is already partly running on my Twit Cleaner account.

Spam Hunter– a neat game (a core chunk of the back end code already written) to make it fun (and super efficient) to hunt spammers on Twitter. Thus, the masses pool their efforts to remove spam from Twitter (since of course, all this would be collated and automatically fed back to Twitter’s spam team). I reckon, combined? We could make Twitter spam history (or so expensive as not to be worth it for the spammers). I’ve tracked, identified and reported hundreds of thousands of spammers off Twitter.. just while testing this code – not even using it in force.

At the risk of sounding immodest, Twit Cleaner was a stupidly robust system (optimising systems is something I’m really good at).

Most of the downtime I did experience was because I was constantly tweaking and improving things, while working at the outer limit of my skills in some pretty arcane technical areas.

I could happily leave the system running for weeks at a time while I worked on other code. I’d do tech support, but not really touch the engine. It would gracefully handle the plethora of random errors emanating from Twitter’s complex environment; interpret, adjust and retry. Nothing would be lost. Not much would slow.

It was an engineering marvel built, as it were, atop a fail-whaling ship.

And yet the v1.1 Twitter API “upgrade” hit like 20 ice bergs.

Every single Twitter related part of my system started screaming. Yes, some bits I’d made bad choices or coded less-than-perfectly, but when you fix 15 major problem sites, and there’s still more, many of which I genuinely have no idea how I can correct, adjust for or code around? That’s dark.

And when you factor in how smoothly everything ran before this upgrade?

It’s not fun.

Now, to be really clear. I don’t begrudge the decisions Twitter has made with these changes. They’re running a system with a couple of hundred million users. Mine has a couple of hundred thousand. At that many orders of magnitude difference, things change. A lot. With thirty years of tech experience, I marvel that Twitter works at all – let alone as quickly and (relatively) painlessly as it does.

All that said, I’d hoped that dropping the maximum user size down would help things, but quite literally, every part of my system is still complaining (despite a ton of work by me). I’m running out of API calls (how much Twitter will let me do, per 15 minute block) everywhere. Everywhere.

The Twitter API documentation says I should check that. Oh, wait, I’ve run out of API calls to check how many API calls I have left. WTF? Seriously?

And so on.

Multiply that by 60+ hours in 3 days and you start to see why I’m walking away from the dead horse.

I suspect I won’t be alone.

Running Twit Cleaner has been one of the most amazing experiences in my life. I’ve met so many incrediblepeople. I’ve helped, in a small-but-significant way so many to have a better experience on Twitter. I deeply regret that I can’t continue to help improve your lives. It’s not feeding starving children in Africa… but you know, it wasn’t nothing. And wow, you people are amazing. Thank you, from the bottom of my heart.

Thank you also to both Matt and Katy in the spam team at Twitter. It was an extreme pleasure and an honour helping them kick all those bad guys in the ass.

Through all this I’ve learned so, so much.

Perhaps the key thing? Neverplayfootball when someone elseowns the field.

So obvious in hindsight.

Perhaps if I’d known that in Oct 2009, when I started all this, I might have taken a very different path.

But I would have missed so, so much. And I’m incredibly grateful for it all.

From here? I have no idea. I’ll see what the universe brings. A holiday would be nice. In the mean time, of course, I’ll keep learning how to love more and keep posting weird crap on my personal Twitter account, @sidawson.

Take care out there.

Si

[related: Some of the feedback Twit Cleaner has received over the last four years]