jordan orelli

Most developers that are familiar with dynamic scripting language like Ruby, JavaScript, or Python have been exposed to higher-order functions. Coming from a scripting background, it can be hard to translate that knowledge into Go, since the type system seems to get in the way. On the other hand, coming from a statically typed, primarily object-oriented language like C++, C#, or Java, the problem may be the opposite: static type systems are less of a stumbling block, but the usage of higher-order functions may be less intuitive. For programmers with functional experience, most of this knowledge may come across as very pedestrian, but hopefully this article will at least demonstrate how to use Go’s type system with respect to functions.

In this article, we’ll look at a few situations where function types in Go can be very useful. The reader is not assumed to be an experienced Go programmer, although a cursory knowledge of Go will certainly be helpful in digesting the material.

Anonymous Function Closures

Off the bat, Go supports anonymous functions and closures. To me, Go’s support for anonymous functions is most reminiscent of the anonymous function support in JavaScript; it is not the flimsy kind of anonymous function support you get with Python’s lambdas (one statement only) or Ruby’s situation where there are multiple ways to create closures (I’m not a rubyist, so the whole proc/block/lambda/Method thing seems very convoluted to me). Anyway, declaring anonymous functions in Go is very lightweight:

func() {
 fmt.Println("hello")
}

As an expression, that’s fairly useless. Generally you want to save it to a variable, stick it in a data structure, or pass it to another function. It’s fairly common to see this type of thing hanging out in Go code:

fn := func() {
 fmt.Println("hello")
}

We now have a variable fn that is a function; it is of type func(). It can be invoked like any other function by saying f(), or reassigned to any other func() that you might be interested in. Of course, since we have support for closures, we can also reference some data that was declared in the function’s containing scope:

x := 5
fn := func() {
 fmt.Println("x is", x)
}
fn()
x++
fn()

The output of this chunk of code would be:

x is 5
x is 6

Fairly standard. You can run that example here. So far, this is mostly the same as how it looks in JavaScript, except statically typed.

Function Collections

Of course, we can use functions in all the same places that we can use regular data types. We can, for example, create a slice of functions, chose a function at random, and then execute it. We define the type “binFunc”, which is a binary function; it takes two integers, and returns one integer. This isn’t strictly necessary for this example, but typing binFunc over and over again is much more convenient than typing func(int, int) int everywhere you see it.

type binFunc func(int, int) int
func main() {
 // seed your random number generator. In the play.golang.org example,
 // we can't use time.Now() because it isn't available in the Go PlayGround.
 rand.Seed(time.Now().Unix())
 // create a slice of functions
 fns := []binFunc{
 func(x, y int) int { return x + y },
 func(x, y int) int { return x - y },
 func(x, y int) int { return x * y },
 func(x, y int) int { return x / y },
 func(x, y int) int { return x % y },
 }
 // pick one of those functions at random
 fn := fns[rand.Intn(len(fns))]
 // and execute it
 x, y := 5, 12
 fmt.Println(fn(x, y))
}

(run this example)

Of course, this specific application is obscenely contrived, but it illustrates a core concept in Go: functions are first class.

Functions as Fields

Similarly, we might be interested in using a function type for a struct field. Doing so would allow us to attach additional information to a function, such as a label that we can access at runtime:

type op struct {
 name string
 fn func(int, int) int
}
func main() {
 // seed your random number generator
 rand.Seed(5)
 // create a slice of ops
 ops := []op{
 {"add", func(x, y int) int { return x + y }},
 {"sub", func(x, y int) int { return x - y }},
 {"mul", func(x, y int) int { return x * y }},
 {"div", func(x, y int) int { return x / y }},
 {"mod", func(x, y int) int { return x % y }},
 }
 // pick one of those ops at random
 o := ops[rand.Intn(len(ops))]
 x, y := 5, 12
 fmt.Println(o.name, x, y)
 fmt.Println(o.fn(x, y))
}

Functions can also be stored in maps, so you could do something similar with map[string]binFunc.

Recursive Function Types

Another interesting aspect of function types is that they allow us to define recursive function types, which are function types that operate on themselves. That is, a function type that either takes it’s own type as a parameter, or users its own type as a return value. You can use this technique to to lots of things if you contrive hard enough, so I’ve contrived a way of using recursive functions to perform a random walk, just for the sake of illustration. Yes, this is an utterly contrived example. To start, we might say that a walk function is a function that takes a pointer to an integer, and returns a walk function:

type walkFn func(*int) walkFn

It is expected to increment the integer that is being pointed to, and then return a function describing the next step that should be taken. An example of a valid walkFn would be the following walkEqual function:

func walkEqual(i *int) walkFn {
 *i += rand.Intn(7) - 3
 return walkEqual
}

The effect of running this function would be that the integer that i points to would be randomly modified by +-3, and then the function would return itself. We could then perform a random walk in the following fashion:

fn, progress := walkEqual, 0
for i := 0; i < 20; i++ {
 fn = fn(&progress)
 fmt.Println(progress)
}

Of course, this isn’t very fun. Let’s involve two more functions, walkForward and walkBackward, which will guarantee that we walk in a “forward” (i.e., positive) direction, and “backward” (i.e., negative) direction. Each function will execute itself, and then return one of the other two functions, randomly. We’ll start with a little utility function named pickRandom that will randomly choose between its input functions, and return that function:

func pickRandom(fns ...walkFn) walkFn {
 return fns[rand.Intn(len(fns))]
}

Then we define our three “walk” functions as follows:

func walkEqual(i *int) walkFn {
 *i += rand.Intn(7) - 3
 return pickRandom(walkForward, walkBackward)
}
func walkForward(i *int) walkFn {
 *i += rand.Intn(6)
 return pickRandom(walkEqual, walkBackward)
}
func walkBackward(i *int) walkFn {
 *i += -rand.Intn(6)
 return pickRandom(walkEqual, walkForward)
}

Executing the walk is exactly as it would be before, but now notice that walkEqual can return either walkForward or walkBackward. You can run this example on the Go Playground to see how it works as a full program.

This particular example is deliberately contrived, so as to keep the focus on the mechanics of recursive functions. In the standard library, this technique is used to make a lexer in the text/template package (view in stdlib source here). A full run-down on how that works is covered in Rob Pike’s talk Lexical Scanning in Go.

Function Types as Interface Values

Function types in Go can also have methods. It’s hard to see, at first, why this would be useful. There are two side effects to the fact that function types can have methods in Go: firstly, since any type that can have methods can satisfy an interface, function types in Go can also be boxed into valid interface types. Secondly, since methods in Go can have either pointer or value receivers, we can use methods on function pointers to switch the function being pointed to in the method’s calling context. That’s a bit heady at first, so let’s go through those two things in order.

First, the most obvious way to make a function type also satisfy an interface is to pick an interface with only one method. Keeping it simple, let’s define an add function like we did above, but we’ll give it an Error() string method. In Go, any type that has an Error() string method is a valid error type, so our function can serve as both a function and an error.

type binFunc func(int, int) int
func add(x, y int) int {
 return x + y
}
func (f binFunc) Error() string {
 return "binFunc error"
}
func main() {
 var err error
 err = binFunc(add)
 fmt.Println(err)
}

You can run this totally useless example on the Go Playground.

It’s worth noting that in this case, we had to perform a type conversion to convert the function add, which is of type func(int, int) int to type binFunc. This may seem like an onerous misfeature of the Go runtime, but what if there was another valid conversion for func(int, int) int that would implement error:

type loudBinFunc func(int, int) int
func (f loudBinFunc) Error() string {
 return "THIS ERROR IS A LOT LOUDER"
}

If both binFunc and loudBinFunc are defined, the runtime wouldn’t know how to convert our add function. Even if there is only one valid implementation of an interface for a given type, the runtime will not perform the conversion for us automatically. It seems very annoying at first, but it makes it easier to write bug-free code.

In the standard library, we can see an example of using a function type to satisfy an interface in the net/http package. The http.Handler type is the interface used by the standard library to register http handlers:

type Handler interface {
 ServeHTTP(ResponseWriter, *Request)
}

There’s also an http.HandlerFunc type, which is just a wrapper around func(http.ResponseWriter, *http.Request) that satisfies http.Handler. This allows us to use either a function or a struct as a web handler. The end result is that we get both the http.Handle and http.HandleFunc functions, which makes it a bit more convenient to write web handlers.

Now that we’ve established how methods on functions allow us to satisfy an interface, I’ll show a slightly more esoteric usage that turns out to be remarkably useful: let’s stick some functions in a json document.

The encoding/json package in the standard library is the normal way to encode and decode json documents. This package is interface-based; it defines an interface, json.Unmarshaler, that is to be used in the unmarshaling of json documents:

type Unmarshaler interface {
 UnmarshalJSON([]byte) error
}

That is, if you define this method, the encoding/json package will use this method when unpacking json data; otherwise, it attempts to figure out what to do on its own (and generally gets it correct). Let’s stick with the binFunc example, but this time, we’ll stick some binary functions in a json document. We’ll start with the tired old binFunc definition:

type binFunc func(int, int) int

Since functions themselves do not have any persistent data, we need somewhere to register names. We can use a map[string]binFunc to store our functions with their associated names:

var fnRegistry = map[string]binFunc{
 "add": func(x, y int) int { return x + y },
 "sub": func(x, y int) int { return x - y },
 "mul": func(x, y int) int { return x * y },
 "div": func(x, y int) int { return x / y },
 "mod": func(x, y int) int { return x % y },
}

In Go, if we attempt to retrieve a value from a map with a nonexistent key, the retrieval operation will return the zero value for the given type (if you’re asking “but what if we legitimately put nil into that map?” then see here). For function types, the zero value is always nil, so we can just test against nil to see if we’ve retrieved a value. Now we need to actually implement the UnmarshalJSON method on our binFunc type. Since we want to make changes that are seen by the caller of the function, we want to implement UnmarshalJSON with a pointer receiver, and since we know we want a string, we just piggy-back of the existing string unmarshaling capabilities of the standard json package.

func (fn *binFunc) UnmarshalJSON(b []byte) error {
 // start by unmarshaling the name of the function that we want
 var name string
 if err := json.Unmarshal(b, &name); err != nil {
 return err
 }
 // get the function out of our function registry
 found := getBinFunc(name)
 if found == nil {
 // return a descriptive error if we can't find the function
 return fmt.Errorf("unknown function in (*binFunc)UnmarshalJSON: %s", name)
 }
 // dereference the pointer receiver, so that the changes are visible to the caller
 *fn = found
 return nil
}

With that defined, we can now unmarshal a binFunc from a json document. See here for the full, runnable example that unpacks just one function, and here’s an example of using this to unpack a list of functions. Of course, this is just the tip of the iceberg, but it shows what types of things can be done with this technique.

Channels of Functions

There’s one last pattern that is very unique to Go, and that’s the result of what happens when you combine function types with channels. A full description of Go’s concurrency model is out of scope for this post, but for the unfamiliar looking to get their toes wet with Go’s concurrency model, I recommend reading the concurrency section in Effective Go, or if you have the time for it, watch the very excellent Google I/O talk Go Concurrency Patterns. From this point forward, some knowledge of Go is assumed.

Since channels are primitives in Go, and we can make channels of any other type, we can even make channels of functions. Since functions can be anonymous, and functions are closures, we can combine these three properties to make channels of anonymous function closures. The definition for this channel type is obvious once you realize that it is possible: chan func(). Keeping in the theme of this post, let’s start by making a slice of functions, but we’ll make each one a closure:

x := 10
fns := []func(){
 func() { x += 1 },
 func() { x -= 1 },
 func() { x *= 2 },
 func() { x /= 2 },
 func() { x *= x },
}

We’ll also nab another piece of code we looked at earlier, and device a mechanism for picking one of those functions at random:

func pickFunc(fns ...func()) func() {
 return fns[rand.Intn(len(fns))]
}

Next, we make a channel of func() values:

c := make(chan func())

And we define a function that takes as its parameters a function channel, a number describing how many functions it should write into this channel, and then a set of candidate functions:

func produce(c chan func(), n int, fns ...func()) {
 defer close(c)
 for i := 0; i < n; i++ {
 c <- pickFunc(fns...)
 }
}

At the start of this function, we use defer close(c) to ensure that our channel gets closed when we’re done producing values. From there, we just use a regular for loop, pick a function at random using pickFunc, and then write it into the channel that we’ve been given. On the receiving end, all we need to do is read values from the channel using a range clause, then immediately execute these functions upon being read. By adding a sleep after each read operation, we’re able to implement a rate-limiting device for arbitrary closures.

for fn := range c {
 fn()
 fmt.Println(x)
 time.Sleep(delay)
}

Putting this all together, we arrive at the following program:

package main
import (
 "fmt"
 "math/rand"
 "time"
)
var delay = 200 * time.Millisecond
func pickFunc(fns ...func()) func() {
 return fns[rand.Intn(len(fns))]
}
func produce(c chan func(), n int, fns ...func()) {
 defer close(c)
 for i := 0; i < n; i++ {
 c <- pickFunc(fns...)
 }
}
func main() {
 // time is frozen on Playground, so this is always the same.
 rand.Seed(time.Now().Unix())
 x := 10
 fns := []func(){
 func() { x += 1 },
 func() { x -= 1 },
 func() { x *= 2 },
 func() { x /= 2 },
 func() { x *= x },
 }
 c := make(chan func())
 go produce(c, 10, fns...)
 for fn := range c {
 fn()
 fmt.Println(x)
 time.Sleep(delay)
 }
}

Run this example on the Go playground.

Conclusion

Functions are great!

2:53 pm  •  5 February 2013  •  8 notes