The Shape of Everything: We Need A Standard Layered Image Format

Apr 30, 2013

Last summer, Adobe killed their image exchange format "FXG". The idea was to have a publicly defined XML based image format which could handle vectors, bitmaps, and layers, and could be read and written by any app that wanted to support it (as Acorn did for a while).

I can't say I'm sorry to see it go. It was a horrible format. The goal is worthy, but the implementation of it was an incredibly bad idea. When you want to send someone an image you want to pass them a single file, not an XML file with a folder of assets. While there are technical benefits to this, it's an incredible burden on the customer.

There is of course PSD which is the native format for Photoshop, and over the years it has become the de facto standard for layered images. PSD is a crazy format and implementing a reader and writer for PSD files is non-trivial and nobody but Adobe actually supports it correctly. It's crazy hard (and I'm not blaming Adobe or PS engineers for this- extending a file format for 25 years isn't exactly an easy thing to do).

So what would be better?

Quote from Appropriate Uses For SQLite.

There's my vote. Acorn has been using SQLite as its native file format since version 2.0, and it has been wonderful. When writing out and reading in an image I don't have to think about byte offsets, I mix bitmap and vector layers together in the same file, and debugging a troubled file is as simple as opening it up in Base or your preferred SQLite tool. This sure beats opening a PSD file in a hex editor to figure out what's going on.

Using SQLite for an image format has a ton of advantages:

• SQLite is easy to use and easy to embed. It's also public domain so there's no licensing to worry about.
• Just about every scripting language worth its salt already has binding for SQLite, or even ships with it installed. Anyone can write a simple Python script to look inside a sqlite file and dump out or update the contents.
• Scalability. A SQLite database can grow in size to 2 terabytes.
• It's Atomic. With transaction support you aren't going to be writing to an image and have partial saves when the power goes out.
• Updates. Changing a single layer of a 2000 layer image doesn't need to update the whole file. You simply update the row associated with the layer, and go on your merry way.

A simple schema for the database will also go a long way. Here's the three tables Acorn uses:

create table layers (id text, parent_id text, sequence integer, uti text, name text, data blob);

This is where your layers are stored. The id of the layer is a unique identifier (UUID). The parent_id is the layer's group UUID; if this value is null, then it's a top level layer. The sequence is the order of the layers in the group, the UTI is the type of the data the layer holds, and the name field is UTF8 that is shown to the user in the layers list. The final field is data for the layer.

Acorn uses 'public.png' for its bitmap layers, 'com.flyingmeat.acorn.shapelayer' for its vector layers, and 'com.acorn.group' for the group layers. Acorn can also load other bitmap types and in a future release will use tiff for bitmap layer data.

Even if you app doesn't know how to read certain layer types, you can open up and preserve the existing data while being able to modify the layers that you do understand.

The second table is defined like this:

create table layer_attributes (id text, name text, value blob);

The id is the UUID of the layer, the name field is a key for the attribute, and the value is … the value. This is where things like the frame of the layer are stored, bend mode, opacity, whether it is visible or not, etc. You get the idea- it's a key value store for the layer.

And finally, the third and last table is defined as:

create table image_attributes (name text, value blob);

This is the key value store for the whole document. The image's color space, rulers, is the grid showing or are the guides locked? Etc. There's even a composite of the image in here so if an application just wants to show a preview of the image it's a single query to pull it out (this idea is stolen from PS, which puts a composite of the image at the end of a PSD file for the same reasons).

This format is flexible, it's easy to implement, and Adobe, if you are listening you should really give SQLite a serious look. And if there are any other companies out there wanting to work on this- please get in touch with me (gus@flyingmeat.com). I'm not married to Acorn's format or table names, but I think it would be a great start.

It's 2013. Wouldn't it be awesome if you could hand off a single layered file format to multiple image editors, and it would just work?