Are video codecs written in JavaScript really the future? | Ars Technica

Describing it as "the future," Mozilla has been showing off ORBX.js, a video codec roughly comparable to the industry-standard H.264 that can be decoded entirely in JavaScript.

ORBX.js was developed by a company called OTOY. OTOY's major product is Octane Render, a 3D renderer that works exclusively on NVIDIA cards using CUDA. Working with customers including Autodesk, OTOY has developed technology to allow applications such as Autodesk's 3ds Max modelling software to be accessed over the Web without using plugins.

Central to this is the ORBX codec. The codec allows efficient, real-time encoding on GPUs, and can be decoded in JavaScript. The JavaScript decoder works in all modern browsers, both desktop and mobile, though with differences depending on the browser features available.

Like most video codecs, ORBX distinguishes between video frames that are encoded in their entirety—"I-frames"—and those encoded as deltas relative to another frame—"P-frames" and "B-frames". I-frames can be decoded independently of one another, but P- and B-frames can only be decoded when their surrounding frames have also been decoded. ORBX, again like most other video codecs, can be used to create video streams made up entirely of I-frames, or streams that use a mix of I-frames and P-frames. I-frame-only modes don't compress as efficiently, but are simpler to decode.

For browsers including Internet Explorer 10 and Safari on iOS, ORBX is used in I-frame-only mode. For other browsers, including Firefox and Chrome, it uses a more conventional mixed mode. That's because the mixed mode depends on WebGL for part of its decoding. I-frames can be encoded entirely in JavaScript, but P-frames require the use of shader programs due to their greater complexity. Internet Explorer 10 and Safari on iOS don't support WebGL, and so can't be used to run shader programs. As a result, they use about twice as much bandwidth for the same level of video quality.

OTOY plans to develop ORBX to make it suitable for a wider range of applications than existing codecs like H.264. The company wants to include features such as support for floating point data, to make it suitable for both high dynamic range imagery and video, and to make it usable for storing 3D data such as depth buffers.

Its encoding efficiency is broadly comparable to H.264—the company says that it's 25 percent "better" than H.264 in some tests, but in others H.264 pulls ahead. The new HEVC/H.265 codec has better encoding efficiency than both H.264 and ORBX (it produces smaller files for a given level of quality), but requires far more computational power to encode and decode.

Being designed for streaming live apps, the ORBX encoder boasts low latency. To prove this point, the company has demonstrated the game Left 4 Dead 2 being streamed to a browser.

Possibilities for ORBX

OTOY describes a number of future ideas for ORBX. One plan is for more powerful remote application support, not merely streaming a video of an application (similar to the way VNC provides remote application support), but streaming the graphical API calls that an application makes, using client-side WebGL to actually perform the drawing. This reduces the load on the server end, and can reduce bandwidth requirements substantially.

Another idea is to use ORBX for DRM-free streaming video. OTOY believes that ORBX's efficient on-GPU encoding makes it practical to embed within each video stream a per-user watermark. Watermarked video wouldn't be DRM-protected, and so would be relatively easy to record and redistribute, but the watermarking would make it possible to determine who released a given video, in turn opening the door to pursuing the original pirates.

Mozilla's interest in ORBX is obvious. Mozilla is aggressively promoting the Web as the platform to replace all other platforms. This is the entire point of its Firefox OS smartphone platform: that rather than developing apps tied to Android or iOS or Windows Phone or BlackBerry, developers should write apps for the Web, using JavaScript, HTML, and related technologies.

Demonstrations of video decoding in JavaScript serve to bolster Mozilla's argument that JavaScript performance is good enough to make it usable for almost every application—even those traditionally regarded as "compute-intensive." Mozilla itself has experimented with JavaScript-based video decoding with its Broadway decoder. This was fast enough to decode H.264 at 30 frames per second in JavaScript alone.

The alternative to DRM is also attractive to an organization that promotes the open Web. DRM capabilities are set to be included in browsers, a decision that isn't universally popular among open Web proponents. The use of watermarking instead of DRM has some precedent—iTunes Music Store, for example, made a successful transition from DRM to watermarks—and could be pushed as a demonstration of the lack of necessity for DRM in browsers.

But is this really "the future" as Brendan Eich, Mozilla CTO, greeted it? That's harder to see.

Challenges remain

ORBX looks very cool for certain applications, but as a demonstration of the power of "openness" it's currently a little lacking. Organizations such as Mozilla may not like H.264's royalty-bearing status, but it's nonetheless an open, documented, published standard. ORBX, however, is none of these things. It's a proprietary algorithm created by a commercial organization. We spoke to OTOY, and the company told us it has no immediate, specific plans to publish it openly. That's not to say that the company is necessarily opposed to doing so; rather, it's not sure how to do without undermining its commercial products.

Even apart from this detail, using JavaScript for video decompression is problematic. After many years of refusal, Mozilla decided last year that it would start supporting H.264 compression in the HTML5 <video> tag.

The company was ideologically opposed to this, due to H.264's patents and royalties, but came to realize that it was a practical necessity, especially for Firefox OS. H.264 video is abundant, and is unambiguously "the winner" of all the current major video codecs. Consequentially, H.264 is widely supported in hardware, enabling battery-efficient hardware-accelerated playback of H.264 video. For Firefox OS to be viable, it had to support this codec and this hardware.

H.264's abundance and hardware support are no less important this year than they were last. Video codecs written in JavaScript may be technically impressive, but they mean eschewing that hardware support. The use of WebGL and shader programs may improve the power efficiency somewhat, but they are still inferior to the dedicated motion video hardware found in all modern GPUs. It's unlikely that mobile users or video distributors are going to be willing to give up this power advantage any time soon.

The viability of watermarking as an alternative to DRM is also speculative conjecture rather than identification of a genuine industry trend. In spite of the changes in the audio market, the video market has remained firmly in favor of DRM. The audio and video markets have significant differences in terms of both production and usage model, and it's not clear that what works in one market will necessarily work in another.

Nonetheless, aspects of the OTOY demonstration probably are indications of what the future will hold. Sony bought cloud gaming company Gaikai last year, and there's speculation that it'll use Gaikai-powered cloud gaming as its backwards compatibility solution for the PlayStation 4. Cloud gaming pioneer OnLive almost went out of business last year, after failing to attract sufficient customers, but a comparable service that's built-in to an existing games platform could fare rather better—though latency problems are likely to be a permanent fixture.

Similarly, streaming applications are likely to continue to remain important to certain niche users, but it's much less likely that they'll ever be the solution to mainstream application demands. Again, latency is one of the insurmountable difficulties.

The ORBX demo is clever technology, and it shows just how far JavaScript performance has come since browser developers started making it a priority. But as a vision of the future, it's less compelling. Using JavaScript for video is cool, but for most users, not using JavaScript for video will be better.

Listing image by Lee Maguire