btcd: A Full Alternative Bitcoin Implementation, Written In Go | Bitcoin MagazineBitcoin Magazine

Conformal, a company focused on building open-source software for privacy and security, has revealed their latest project: btcd, a full Bitcoin reimplementation in Google’s programming language Go. The implementation is not yet entirely done, but most core features such as transaction verification, database interaction and network connectivity are tested as working, and the company has released one component of the system for public review: btcwire, a package that focuses on converting Bitcoin protocol messages between btcd’s internal representation of the message and a serialized form suited for direct network transmission, as well as actually sending and receiving these messages over network connections. Conformal provides a more detailed description of btcwire on their website.

The company has also written a blog post explaining their decision to make this alternative implementation; they write:

A number of us at Conformal Systems had been keeping an eye on Bitcoin as passive observers for the past couple years since Bitcoin combines technologies that are already of interest to us: practical use of cryptography, distributed systems, and electronic payments. In January 2013 I had one of our developers, David Hill, attempt to port bitcoind and its GUI to Bitrig, an OS that several of our developers forked from OpenBSD. David encountered several problems with porting to Bitrig and in the process found issues with unit tests, non-portable functions and seeding of a PRNG. While pushing to get the port complete, it was clear that it would take a lot more effort than usual to complete this port. After seeing these issues with the porting, I felt that the Bitcoin ecosystem could use an alternative to bitcoind.

As for why they chose the programming language Go, they write that Go has a number of advantages over the C++ used by the only full Bitcoin implementation currently in major use, bitcoind. Particularly importantly for financial software, Go offers “integrated test infrastructure, no active memory management, standard formatting, platform independent code, simpler parallelism, built-in profiling and documentation facilities [and is] virtually crash-proof.” Integrated test infrastructure is particularly important; the easier and more natural it is to write tests, the more tests can be written, and the smaller the chance we will see another serious blockchain fork from an unforeseen edge case. Parallelism, profiling and crash-proofness are also particularly useful attributed for Bitcoin exchanges, so given the performance bottlenecks that all exchanges were revealed to have in the Bitcoin market crash in April new exchange developers may well consider using Go as their language of choice.

This is not the first attempt at reimplementing the Bitcoin protocol; Amir Taaki’s libbitcoin (written in C++) and Mike Hearn’s BitcoinJ (Java) have both accomplished the feat, although the former is being used only in some experimental versions of the Electrum server software and the latter in some Android light clients. Importantly, there are no alternative implementations of Bitcoin that are currently being used to any significant degree by miners.

There are reasons to believe that this is unhealthy for the Bitcoin ecosystem. In theory, Bitcoin is an open source protocol in which anyone can participate, and on most levels that is true. Partial implementations of Bitcoin that can make and verify transactions exist already, and are the backbone of alternative Bitcoin clients like Blockchain and Electrum. In practice, the deeper into the protocol one goes the more it becomes a monoculture, but monocultures are dangerous. If there is only one implementation being widely used, then unforeseen bugs appearing (or even disappearing) in upgrades can cause the entire Bitcoin blockchain to essentially fork into two as the two versions of the protocol disagree on which transactions and blocks are valid and which are not. Because the blockchain builds on itself, even one mistake will have consequences that cascade forever into the future.

This was the cause behind the blockchain fork that took place in March – a previously unknown limitation in the database software used by bitcoind 0.7 caused a fork only after bitcoin 0.8, which used a different database which did not have this limitation, was released. The two versions then disagreed on the validity of a single block, and the fork followed from there, sending the price crashing by 23% before enough major mining pools got together on IRC chat to correct the issue. With three or five re-implementations, on the other hand, any single version disagreeing with the others will be highly inconvenient for users of that version, but will not have a grave effect on the entire network.

Multiple implementations is also good for standardization; in a monoculture, the dominant software becomes the standard, giving the developers of the dominant software a higher degree of control over the future development of the protocol than was originally intended. This does not even require the developers to make a conscious decision to start subverting the design process for personal gain to be harmful. Even if, as is almost certainly the case today, developers are acting with the best of intentions, there is a bias in software development toward increasing complexity and confusion that is mitigated if multiple implementations have to work together on every change. If there is only one implementation, errors get uncovered later rather than sooner, and the result is a sort of Talebian “stability breeding its own instability” that ultimately, as in the case of the March blockchain fork, causes disaster. Additionally, developers have no incentive to even document the protocol as long as it works internally. Currently documentation does exist on the Bitcoin wiki, but with multiple implementations we can be much more certain that the page will be updated, and even improve in quality, in the future.

Whether or not btcd will actually be used by miners is hard to say; it is entirely possible that miners will remain comfortable with the existing bitcoind, and the stability through decentralization that a healthy ecosystem of alternative implementations can bring will never come to pass. But even in such a state, this will still be a step forward for Bitcoin if only because it makes it easier for Go users to interact with the protcol. A Bitcoin implementation in Go has already been written, but it has not been updated in nearly a year, and appears to have only ever had a single developer behind it. btcd is backed by a corporation that is clearly well-versed in security and privacy, inspiring much more confidence in its reliability.

For the near future, Conformal writes that “our goal is to continue releasing packages from btcd as we increase test coverage and bring them up to a professional level. The intent is that each piece can then be publicly reviewed for correctness as they become available while we continue work on the remaining packages.” The company intends to continue releasing packages over the next several weeks, pushing out one at a time so that the community can better review the packages as they come out. Their next release will be btcjson, a package the deals with JSON-RPC messages. You can watch out for more releases on the Conformal blog.