OVH (host for Slush's pool and Bitcoin-Central.net) has been compromised

The pool has been hacked. Fortunately I noticed it fast enough, so I made database snapshot seconds before attackers overtake the database machine. I lost some amount of bitcoins, but I'll be able to recover it from my pocket. For now I'm evaluating what's next to do, because all machines in OVH has been compromised and they cannot be trusted anymore.

Full story:
Today at 3pm UTC I noticed that somebody succesfully resetted the password to OVH manager, the place where servers can be managed, restarted to rescue mode etc. I promptly resetted the password at OVH to something different and I also changed password on my email account and checked that there're no other active connections to my mailbox. I have to say that my mailbox is secured by OTP passwords and I take physical security very seriously, so nobody other had an access to my mailbox. I known that password-reset feature is quite popular attack vector, so I made everything possible to prevent it to happen.

By changing the password at OVH, all other sessions using the old credentials are automatically kicked from the Manager. I also cross-checked that nothing wrong happen to the servers at this time. Unfortunately I didn't find a way how the attackers got access to Manager, so I asked OVH support to provide some additional information and restrict Manager access to my IP range.

That's no surprise that OVH didn't respond to this ticket for hours, but at 11pm UTC I realized that there's another succesful password reset at OVH. This is complete mystery to me, because I'm aboslutely sure that nobody else had access to my mailbox and the email with reset link has been untouched (unread, not deleted). I'd say that attacker won't bother by changing status of the email to "unread", but he'd delete the email instead.

This time I realized that the attacker resetted the machine with the wallet to rescue mode, which means that I lost the control to this machine. I was still succesful by logging into the database and I took the snapshot of database and transferred it to safe location. Few seconds since the migration finished, attackers restarted all remaining machines to rescue mode.

So far it looks like yet another inside job, like Linode two years ago. Or attackers found some shortcut how to gain access to Manager without confirming the request from the email. I don't know what's worse option. I'll investigate this issue in detail later and I hope OVH won't close eyes to this.

I can recover the pool to the normal operation tomorrow.

Edit 01:38 UTC: Stratum servers are running on safe servers at Amazon. Mining works for now. I'll setup new database and webserver on trusted machines in few hours, so the pool will be back in full operation.

Fantastic.

https://bitcoin-central.net/