How to parse Ruby—programming is terrible

If you want to parse Ruby, I wish you luck—there isn’t any documentation or a formal grammar that fully describes the language. Instead, you have to look at the source code to truly understand Ruby’s syntax. This is where the nightmare starts.

Take a look a Ruby’s parser, defined in parse.y. I’ll wait for your eyes to stop bleeding, and then we’ll try to understand how it got that way.

Parsing code is traditionally a two stage process, first transform the stream of characters into a stream of tokens (lexing), then build trees out of these tokens (parsing)—i.e first you turn “1 + 2” into num(1), op(+), num(2) and then build a tree, i.e op(+, [num(1), num(2]).

A little more trickery is involved in parsing ruby, because of ruby’s (locally) ambiguous syntax. For example, x could be a variable, or a method call, and so x +1 could be ‘add 1 to x’ or it could be ‘call x, with the argument +1’. To parse ruby unambiguously , you must know which variables are in scope.

Parsing Ruby successfully with this two stage process requires dealing with the ambiguity in the parser, or in the lexer—use ambiguous tokens and let the tree builder handle working out the right meaning, or produce distinct tokens so the tree building is easy and straight forward..The former option is slow and memory hungry, so Ruby went for the latter option.

To parse the earlier example, x+1, Ruby’s parser tracks the local variables in scope. After reading x, different flags are set if x is a local variable. On reading +, Ruby checks these flags, and emits an infix operator if it’s after a local variable, and a unary one otherwise.

Similarly, distinguishing puts ([1]).map {|x| x+1} from puts([1]).map {|x| x+1} is handled by setting flags after skipping whitespace, and later checking these flags to emit an LPAREN or an LPAREN_ARG token. As well as semantic whitespace, flags are used for blocks, class definitions and method names too.

It might take a few days of digging, but once you understand the interplay between the flags, the tree builder, and the tokenizer—you can finally start to extract the rules of ruby syntax. Perhaps you can write a cleaner parser this time, but in the end it is just easier to copy parse.y than try to re-implement it. So far, porting parse.y is what every successful Ruby implementation has done

Aside: In MRI’s parse.y, JRuby’s Ruby19Parser.y and IronRuby’s Parser.y you can find the lexer state ‘EXPR_MID’, and as far as I can tell, it doesn’t do anything.

Although I’ve talked about Ruby parsers, I haven’t really explained why anyone sane would want to write one. By luck and various cabal like connections, I stumbled across a rather interesting project to write a new Ruby Interpreter using PyPy. I wanted to help, and started looking at fixing the parser.

In the end, all I could offer was the advice I’ve written here. There are two ways to parse ruby—Do exactly what Matz did, or slowly lose your sanity, gazing into the abyss that is parse.y