virtual lane graphics for swimming with css3 - xyk

If you watched any of the Olympic swimming events last year on television, you probably saw something like this:

Pretty cool stuff. These 'virtual graphics' for sports always intrigue me because there is practically zero time to edit it into the shot. Everything has to be done in real-time. Graphics like this also appear in many other high-profile sporting events like the Super Bowl and the World Series, where the ball is tracked three-dimensionally live. 

Now, my school just happened to be hosting a swimming meet between several other schools. We were going to broadcast and record it anyway, so I thought I might as well try to recreate the graphics. In the end, what I came up with was pretty much visually the same as the one you see in the Olympics.

olympic technology

In a nutshell, the process works by taking the clean feed from the camera and tracking the lane dividers separating the lanes. Graphics are then inserted between, which is what you see up top. The graphics also adapt to the shot, so if the operator pans or zooms, the graphics stay where they are. I don't actually know how they manage to track camera motion in the Olympics, but in the NFL, there are special mechanical rigs attached to the camera that helps the graphics software. 

three dimensional projection

Doing camera tracking would make it a hundred times more complicated, so I didn't bother trying to figure that out. In hindsight, I think it would have been relatively easy to do as long as I had rotary encoders on the tripod feeding data to the renderer. 

I spent many hours trying to pick something to render all the graphics (including the static ones). What renders graphics fast and can update quickly? A browser, of course. The Chrome Skia and V8 engine are heavily optimized for graphics rendering, and HTML5 has matured to a point where you can do advanced graphics like this.

CSS3 has an amazing property called perspective that can transform this:

Into what you see above. 

Imagine a cube with all faces transparent except for the bottom, which composed of the image to the left. Normally, the image would look just like that: looked down from the top face in a birds-eye view. Now imagine the cube rotating on all three axes. This shifts the perspective, and with some educated guesstimation, the image can be projected virtually onto anything. 

The red lines are the edges of the imaginary prism surrounding each row. The water surface of the pool can be thought of as the bottom face of the prism.

The graphic overlaid on the school's pool. This is a test image; scroll down for screen captures from an actual race. Play with it in your browser here.

keying

For actual keying, the background (meaning everything except the graphic itself) is set to rgb(0,255,0). The production mixer my school has is the Sony Anycast, which takes a VGA signal directly from my laptop. This is keyed over the finish-line camera shot I set up on the balcony above the pool. 

practicality and limitations

The use of a subtractive keying system (removing the green) poses several limitations on the graphics. The most obvious would be when the swimmer dives, the graphics float above the swimmer. Ideally, the swimmer's body would literally 'fly' over the graphics into the water. Another limitation would be dynamic opacity. The Anycast only accepts a single opacity level for each key, so I had to balance it between other static graphics.

Compared to the Olympics, the camera placement is not ideal. It is a lot closer to the pool, so I had to use a wide-angle lens, giving the picture corner distortion. It's not obvious, but each lane had small angular adjustments to account for distortion. 

experiment

Experiment and play with the overlay here. Drag the sliders to see how perspective in CSS works. Does not work in IE.

gallery

Camera placement relative to the pool. The camera on the left side of the balcony is the primary tracking camera. On the pool deck next to the flag pole is the robotic camera I built. 

Screen capture from an actual race. 

Some of the production equipment. On the far left is the dedicated graphics laptop. From right to left: Sony Anycast mixer, computer for streaming and recording, sound mixer, and program monitor.