David Kendal: A man, a Plan.

In 2001, Paul Graham wrote Beating the Averages, a treatise on programming language design that turned out to be a precursor to the design essays for Arc.

On the whole, Beating the Averages holds up well 8 years since its last revision. I can’t possibly hope to write anything comparable without repeating it wholesale, so in a sense it has also become a design essay for Plan. This essay will serve as another, explaining where I think Graham is wrong, or at least why Arc hasn’t taken off even with its intended demographic.

Let’s throw out syntax right away. Hackers have long loved Perl, whose syntax bears at least as much complaint-per-user as Lisp’s. A good friend of mine who is new to Lisp (from PHP, with some Ruby and Python) recently commented that he in fact likes how regular its syntax is and how everything is expressed as a function — even the mathematical operators. And if Lisp’s lack of syntax is off-putting, there are always read-macros. (at least in Common Lisp and eventually in Plan) So we’re assuming that syntax is not a big deal to programmers who don’t know Lisp.

How can we find out what really makes a language popular? If we examine the mechanics of popularity, it might give us some clue.

So why is Arc, and Lisp in general, so unpopular? Paul Graham has said before that his goal isn’t popularity — at least not mass popularity. The popularity Graham seeks is with smart people (hackers). However, I think popularity with hackers and popularity with everyone else are not independent factors.

I think it’s important to attract the programmers who Graham says are “the masses” or non-hackers. Without them, the real hackers won’t come to your language en masse either.

Why?

The first reason is exposure and awareness. Though hackers pay great attention to new developments in programming languages, there is so much development happening that it’s impossible to be aware of every new thing. Even if a hacker heard of a programming language, he probably still won’t learn it. Because programming languages are so sticky, a hacker may just see the name of your language and skip over it, so there’s no increase in your language’s mindshare. He might find about just enough about it to see what influenced it and if it has interesting unique features — at least he’s more aware of it, but still isn’t programming in it daily.

It’s a big job to convince people to program in your language — hackers most of all. Hackers are skeptical, and because they’re so smart, they can respond inquisitively immediately to any argument favourable to you. But the one thing hackers can’t do is alter their subconscious. If people (even dumb people) keep talking about a programming language, it’s going to interest a hacker in it, simply because its name will be imprinted in his mind.

This doesn’t mean you can attract smart hackers to your language just by running an advertising campaign though. Sun tried that with Java, but a: their marketing was too explicit, and it just seemed like one company trying to push its own technology, and b: Java itself is a detestable language. So, in addition, it seems like language quality is not something you can make up for in marketing.

So the formula thus far is: exposure to people who’ll talk about it and use it + being a good technology = smart people working with your language.

Another reason it’s important to appeal to regular programmers as well as hackers is that hackers will write libraries and make your platform.

Hackers are quietly egotistical. They’re thrilled by people using their own code. If your nobody using your language needs libraries because they can all write their own code, nobody will release libraries. When they are released, only a small number of people will use them.

In other words, if you have people who need libraries, hackers will write them. When you have more libraries, your language looks more attractive to people who need them, so hackers write more of them, attracted by the desire to learn and solve interesting problems (even ones that have been solved before in other languages). You end up in a slow, but virtuous, cycle in which your platform looks increasingly attractive.

This is a simplistic view, of course. In reality, hackers use libraries, and non-hackers do publish code. But in practice this only speeds up the cycle, because it means hackers are also attracted by the existing code on your platform, and non-hackers are attracted by the ability to have their egos stimulated by seeing other people using things they made. The result: lots of smart people are using your language. (You also have a bunch of less-smart people, but they are learning all the time to program well and become smart.)

Another problem for Lisp is that the rest of the world doesn’t think or work like Lisp. It’s the same problem that plagued the Macintosh in the mid-90s: it had too many proprietary, Mac-specific modi operandi which ignored the industry standards. Networking is a good example: Ethernet on the Mac was never great before the iMac, because they wanted you to use LocalTalk. Even the Ethernet ports provided were not standard RJ-45 sockets, but rather a proprietary socket which required an adapter. (To be fair to Apple, there was no RJ-45 when they added Ethernet. But there was no reason to keep using the proprietary jack when the standard arrived a few years later.)

Lisp has a similar problem. For example: the way to write web-apps in Lisp, pioneered by Paul Graham, is to write them in continuation-passing style. You store closures on the server, giving them unique IDs in a hash table. When a user performs an action, the closure for that event is called. The problem: because closures can’t be stored in databases, you really have to use a hash table on your web daemon. So you can’t scale to more than one server because a user performing an action might end up on a different server to the one the set of closures generated by their last action is on, so the server goes looking for a nonexistent closure and the user gets an error. There are ways to cheat around this, but it’s a fundamental problem with the architecture.

So the conventional, stateless-plus-cookies way of building websites turns out to have an important, desirable property: it’s far easier to scale, because it was designed to be easy to scale. (Plus it’s a whole lot easier to have readable URLs when you’re generating random closure IDs to jam in the path when you load a page.)

There are similar examples where Lisp just wants to bend the real world to its own strengths, and the real world just won’t budge.

The fact is, the rest of the world often has a pretty good reason for doing things the way it does. Lisp people just want things to be easy for them, so they have their own idiosyncratic ways of working around the existing standards. It’s the equivalent of Apple letting AppleTalk run over Ethernet; it still cuts out the rest of the world.

It’s mostly by historical accident that Lisp doesn’t fit so well into the world of Unix. Scheme was designed in the mid-1970s, when Unix was not so well-known and still looked like a research operating system, and Common Lisp in the 1980s when Unix had still not won, so to fit its mission it had to be as system-agnostic as possible.

But it’s now the 2010s. It’s not hard to realise that people expect integer->char to be called chr, and to be able to call fork and have a new operating-system–level process appear.

(And, as if to complete my metaphor, Lisp’s networking remains woefully inadequate without implementation-specific functions.)

Just like the Mac was saved by OS X, which planted the classic Mac on top of a Unix core, and encouraging Mac developers to think like Unix hackers, Lisp needs to be saved by planting it on top of a load of Unix APIs, and encouraging Lisp hackers to drop some of their habits that work around (rather than with) the standards they have to follow. (This doesn’t mean we have to drop everything )

This also means Lisps should have an object system. I think a basic Lisp on its own provides enough functionality to implement a good-enough object system, in much the same way that Perl’s bless gave way to frameworks like Moose. With lexical closures, a native symbol kind, and cons pairs, we can do everything any other language’s object system can do.

Regular expressions, the standard Unix way to deal with text, have also been subject to ignorance and well-meaning Lisp-ification— so it’s important to have regular expressions as a first-class object in the language, with its own syntax. While storing regexps in strings is okay, it’s far cleaner to have a separation within the language.

a variety of people using it + being good and powerful + active library development + following the Unix way = success

Almost every programming language (at least those that are popular with hackers) today has all four elements going for it. Although there’s long way to go, I think we might be on the verge of a Lisp renaissance — but only if more languages can gain traction.