The HTML5 Video Tag’s Fatal Flaw

Back in the day there was no standard way to publish video on the web. You could put any kind of video file you wanted on the server, but there was no guarantee that your readers would have the correct plugin required to view it. Everyone had to have a bunch of plugins installed to have any hope of viewing the majority of video files.

Flash video solve this problem. Flash was installed on nearly every computer anyway, so once they added a video decoder it seemed obvious to provide video content in Flash, even if it was in many ways not as good as the older plugins. Flash video uses massive amounts of processor time and slows down everything else on your computer. On the other hand, websites like Flash because it is easy to skin the player to fit in with the look of the site, and it makes downloading the raw video file (slightly) more difficult.

The <video> tag is supposed to replace Flash by linking to video files in the same way that the <img> tag links to images. In practice this is more complicated than it sounds because videos typically require the ability to skip and rewind content. This means that the browser must be prepared to download different parts of the file and cache things carefully to maintain performance. But these problems have long been solved.

I have been waiting for the big sites to make announcements, and today seems to be <video> day all over the internet. Both YouTube and Dailymotion have demo pages up showing <video> content:

YouTube’s HTML5 Page is designed to show how the <video> tag can replicate the functionality of their famous Flash-based player exactly. Unless you looked at the source (or your OS’s process monitor) you would never know you were using a different player.

The Dailymotion HTML5 Demo is even more impressive, using the <video> tag in combination with fancy Javascript to post-process the video and extract frames.

All this is very cool, but the two demos reveal the video tag’s fatal flaw : codecs. When the video tag was proposed, all browsers were supposed to support an unencumbered decoder named Ogg Theora (no seriously, that’s its name). There were just three problems:

  • Ogg Theora’s quality was not as good as other codecs
  • Although Ogg Theora was designed to be free of patent issues, it was felt that it may be a bit of a lightning rod for litigation.
  • Certain companies may have a vested interest in seeing their own codecs used.

So the requirement to support Ogg Theora was dropped. This means that although all HTML5 browsers will support <video>, there is no guarantee that they will be able to play and particular file. Firefox (at least the 3.5 beta) plays Ogg Theora, but Safari plays H.264 (a superior but expensive to license codec) but not vice versa. For instance, one of the demos above plays in Firefox, the other plays in Safari. This puts us in the farcical situation of having no standard way to publish video, exactly where we started.

There is also the small point that the most widely used browser (IE) does not support the video tag, and probably won’t for years. I predict that Flash video will be around for a while yet, and I am not happy about it.


36 thoughts on “The HTML5 Video Tag’s Fatal Flaw

  1. Edouard Poor

    I remember seeing the video tag stuff go by a year or more ago on Robert O’Callahan’s blog. I personally blame Mozilla’s Open Source political grandstanding for this mess. Ogg Theora got given to the Open Source movement by a failing company years back, and is basically a slightly crappier version of MPEG-1. It’s never been cited in a patent case because the MPEG-1 patents have all expired, and, probably more importantly, no-one actually cares about such an unused and underperforming codec.

    There were only two possible outcomes from that decision (which Mozilla used it’s weight in the browser market to push through in a manner uncannily similar to Microsofts usual standards committee shenanigans).

    Either everyone would continue to ignore the codec because it makes no economic sense to serve up files that are four times bigger than H.264 and look worse (leaving only the Richard Stallman sex tapes in Ogg Theora format).

    Or, by some unimaginable miracle, Ogg Theora would become the new standard for video, and the people with the patents would suddenly notice and sue anyway. Anyone who’s ever dealt with large companies or patents knows a) patents are written in the broadest, vaguest way possible to cover almost anything, and b) not even having a valid patent claim is zero impediment to suing and tying everything up for 5-8 years anyway.

    I’m hoping that eventually the video decoding just uses the native underlying video architectures on each platform (DirectShow on Windows, Quicktime on Mac OS X, and, uhhh, on Linux), and then you can play anything once the right codec is installed on your OS. And, importantly, the browser will offer to install the missing codec for your system if you try to play a file your system doesn’t currently support. Perian for Mac OS X supports almost everything anyone wants to use (so not Ogg Theora, obviously), and there are plenty of codec packs out there for Windows too.

    And bingo, video on the web would be MPEG-4 (incl. H.264) and DivX AVI. Like all the videos on your computer now.

    Unfortunately the browser people asking the question “What video format should be support?” somehow failed to ask the obvious follow-up question “What video formats do our users have?, which is a shocking indictment on how politics trump user-centric design, even in non-profit organisations.

    Phew – that’s been annoying me for a year or two now. Sorry for using your blog to get that off my chest!

  2. Aza

    I really wish people would all get over themselves and just start using H264 and be done with it. It’s a phenomenal codec – super high quality for the file sizes. Just pick one, use it and let that be the end of it for goodness sake!

  3. Andrew

    There needs to be a mandated standard that everyone can be expected to have installed. So long as licensing fees aren’t ridiculous then h264 (in a standard container, none of this DivX rubbish) seems like the way to go. If clients do not supply a native h264 codec then Flash could make itself useful and play it.

    But you know what I hate more than Flash video?

    Oral surgery for impacted wisdom teeth, that’s what!

    I would expound on web standards more, but I have to go lie down for a while.

  4. Chris Double

    The issue with h.264 is the requirement to pay royalty fees to allowed to distribute software that decodes it. Mozilla distributes the source code to Firefox under a license.

    The license requires Mozilla allow recipients of Firefox to be able to download and distribute Firefox with the same rights as Mozilla have. If Mozilla has a patent license for H.264 they can’t allow downstream users to use the software with the same rights. There is no way Mozilla are able to license H.264 for all possible users of the Firefox source code.

    If you’d like H.264 support in open source browsers using H.264 then the people to be lobbying are the owners of the patents and their representatives, the mpeg-la.

  5. Matthew

    * Ogg Theora’s quality was not as good as other codecs

    The quality is quite good and there was even a recent analysis that showed it was comparable with even H.264. Many implementations, such as in the ffmpeg application, are quite bad (Dailymotion uses this), so people sometimes get a bad impression.

    * Although Ogg Theora was designed to be free of patent issues, it was felt that it may be a bit of a lightning rod for litigation.

    Uh… why? If this codec, which has had legal review and a decade of no litigation, would be a lighting rod, then why would it not be the case for any other codec?

    * Certain companies may have a vested interest in seeing their own codecs used.

    This is absolutely true. There is a MPEG license group that gets royalties fees from the use of their codecs. These companies include Nokia, Apple, and Microsoft, who all have MPEG-licensed technologies they would like to remain in use. And those companies need to either make their codec freely distributable or come up with some other solution that works for everyone. Ogg Theora may not be perfect, but it is the best freely distributable codec right now.

  6. Andrew

    Thanks Chris, a comment from somebody who actually knows what they are talking about!

    I was hoping the issue could eventually be resolved by Mozilla and the h.264 crowd coming to some sort a agreement for a worldwide license. I can see now why that was a bit naive.

    Matthew, companies sue over patents all the time, justified or not. The fact is that Theora could infringe on any number of broad patents, and that both video codecs and the web in general are areas in which a lot of lawsuits get filed. It doesn’t matter who eventually wins or if which side is morally right – a lot of people are going to decide it isn’t worth the bother.

  7. Justin

    Andrew, isn’t the same true (possible patent infringement) of any codec? How does Theora suffer more from this than another codec?

  8. Dillon

    I would like to point out to those that think otherwise: Ogg theora is not close to the quality of h264. They retracted their results that said the quality was similar because there was a bug in the psnr measurement that made h264’s psnr 4-5 dB lower. Then and only then was theora close to comparable. Note that if h264 scored only 3 dB higher than theora that still means it’s 2x better (dB is logarithmic). I don’t even think theora has b-frames which even mpeg-2 has. Additionally the people making theora only THINK that it is free of patent issues but they don’t really know. You should read some of the comments of xiph, one of the lead devs of theora, and then you’d see that even internally there is it’s a mess.

  9. Robert O'Callahan

    “2x” better? What does that even mean to human beings? Theora is just fine, and better than what everyone’s used to on the Web.

    At Mozilla we think that people should be able to serve and play video without paying patent licensing fees or “Internet broadcast” fees (H.264 over the Internet is only free until next year). This is surprisingly controversial.

  10. Andrew

    Holy cow – more people showing up who know what’s what. Personally I have no trouble with Ogg Theora, it may not be the best but it is clearly “good enough” to serve its purpose.

    Sadly, the fact that there is no standard codec that can be expected to work with all browsers means that the video tag is almost useless. If I really wanted to serve Theora files I would consider using Javascript to replace the video tag with the Java applet Theora player at runtime for browsers without support. But that is hardly a great solution.

    Thinking with my fingers here; perhaps the Theora people could package up a proper easy-to-use Theora browser plugin and get it widely distributed (it would be in Mozilla’s interests to help out here). If a few sites started displaying “download Theora to view this video” messages then the plugin would be popular in no time and Firefox would look even more attractive (because it would just work).

  11. Andrew

    Forgot to thank Robert O’Callahan for his interesting blog posts and all his work on Firefox. For those that don’t know, Robert is one of the Firefox hackers working on video (among other things).

  12. Dillon

    I’m all for license-free video, but again the problem is that the claims that theora is free of patents come from the developers and there hasn’t been any expert analysis that I know of. Yes, h.264 has patents on it and will incur licensing fees, but at least it is known what these are. For theora it’s unknown and that’s why it’s a possible litigation lightning rod.
    In regards to quality, I’d like to point to this post on doom9: http://forum.doom9.org/showthread.php?p=1282866#post1282866 which might sound okay but then there’s this: http://forum.doom9.org/showthread.php?p=1283024#post1283024 (MV stands for motion vector)
    Basically, if you want HD theora at the same quality as h.264 would provice, expect to use significantly more bandwidth.

  13. Phil

    One small point: just because IE won’t support it for a while doesn’t mean it can’t be used because of the fallback features.

  14. Chris Double

    Andrew, the annodex group of open source developers did start work on a browser plugin to play Theora. This used liboggplay as the base library and was part of the liboggplay distribution (liboggplay is the same library used to implement Theora playback in Firefox 3.5).

    A release was done but development eventually stalled since Firefox 3.5 would have video support anyway.

    The fallback option to enable Theora playback tends to be Cortado, the Java applet. This is what I use on tinyvid.tv as a fallback and it works ok. Not fantastic but good enough.

    XiphQT is a QuickTime plugin that enables Theora playback. This can be installed to get playback with the video element in the Safari 4 browser.

    So if you count XiphQT this gives Theora playback in Safari, Chrome, Firefox and an experimental Opera build. That’s not bad coverage.

  15. Kenneth Pardue

    The internet broadcast fees for H.264 haven’t been decided yet but will be announced no later than January 2010, no less than one year before the fees take effect.

    See this article for more information: http://www.streamingmedia.com/article.asp?id=10969&page=1&c=3

    The thing is, the MPEG-LA group spent the better part of the last decade (successfully) putting H.264 EVERYWHERE. It is accepted as a standard. Hardware acceleration on video cards? Got it. iPhone? Got it (in fact, H.264 is pretty much all it’ll play.) Bluray? Got it. Google Chrome? Getting it. Apple Safari? Got it. DivX 7? Got it. Windows 7? Getting it. Mac OS X? Got it. Quicktime X? It’s practically the house built upon the foundation of H.264. Even the White House uses H.264 MP4 files.

    There’s a vitality for a standard base codec, but realistically, this should not be the reject from the video wars of the last millennium OVER a vastly superior format that’s EVERYWHERE. There are some big players that have an interest in HTML5 succeeding in the MPEG-LA group. I’m confident adoption won’t really go anywhere at least until folks find out what the license fees will be early next year, and that we’ll see some sort of cooperation that will see H.264 become that codec that’s placed in the standard for the meantime, and something based upon Dirac in the future once it is optimized and hardware catches up.

  16. Dillon

    I would say there’s more popaganda on the side promoting theora, but there’s likely some on both sides. That clip you picked has very little motion (and motion happens to be something that theora is very bad at). Even then, at that bitrate (1.4 Mbps) it’s pretty bad. His suit looks like a blurry mess of blocks and if you look at the right side of his suit (his left) and you can barely distinguish his collar from the rest of the suit. Also there’s all sorts of ringing around the edge of his face. The intro and exit screens, which have even less motion, still look bad. It may be that the source video is bad though.

    I’d also like to point out that it’s been found recently that there are two other companies that own VP3 patents besides On2 and those have not been licensed to Xiph.

  17. Darren

    “Holy cow – more people showing up who know what’s what. Personally I have no trouble with Ogg Theora, it may not be the best but it is clearly “good enough” to serve its purpose.”

    I disagree. This is based on the assumption that even though people are moving to Blu-ray in the home and downloading HD video on the net wherever possible, they’re still going to be satisfied with watching low-quality embedded video in the future. I think with services like Hulu we’re already seeing people’s expectations for video quality rise. There’s also the question of bandwidth. As people watch more and more video on the web, the smaller file size of H.264 video for roughly the same quality as Theora becomes more important.

  18. Andrew

    Services such a Hulu are not going to move to any version of the video tag in a million years, because they absolutely require an extra layer of “security” (read DRM!) This is the one area that I predict Silverlight may actually become popular – it has better features for custom handling video than Flash.

    I think you are confused about what the video tag is going to be used for. It is sites like Wikipedia and archive.org that will see the most benefit – nobody is going to be serving HD fullscreen content using the video tag.

  19. Dillon

    Well it seems like Youtube is considering using the video tag for HD fullscreen content. I for one am glad that they are using h.264. Theora is like a polished version of a 15 year old codec that isn’t as open as people would like to believe.

  20. Paul D.

    I have to say that, looking at that OGG/H.264 comparison page (http://html5.ht.funpic.de/bbb.html), H.264 blows Ogg away.

    It’s time to bit the bullet, Firefox. Build in h.264 support and offer the download from servers in a country without software patents — much like what was done back when browsers with 128-bit encryption were considered illegal in the US.

    Patents are BS to begin with, and it’s time to route around the damage.

  21. Dark Phoenix

    “Build in h.264 support and offer the download from servers in a country without software patents”

    Can’t be done. That’s a license violation, period.

    BTW, wasn’t the supposed bad quality of Theora the whole reason work started on Dirac? It seems to me that the Ogg/Dirac combination could compete with MPEG-4.

    BTW, considering Microsoft isn’t coming to this party, cutting Mozilla out is a VERY BAD IDEA, considering they represent the largest group of Internet users after IE users. If neither IE nor Firefox supports MPEG-4 video in the video tag, no one will use it, Safari support or no.

  22. Michael A. Peters

    There is a solution. Firefox can use gstreamer for multimedia the same way Safari uses QuickTime. GStreamer works well on *nix installs, not sure how the Windows builds do, but GStreamer is open source and has a plugin structure that allows commercial third party plugins (IE the ones sold by fluendo that I believe give it legal H.264 capability) to be installed.

  23. Andrew

    Michael, there is a blog post by Robert O’Callahan explaining why the Firefox team does not want to do that.

    Basically they don’t trust external codecs, and with good reason.

    I don’t think anyone is happy with the way things are now, but the reasons the Firefox doesn’t support H.264 seem to me to be better than the reasons Safari doesn’t support Theora.

  24. Aaron

    I am genuinely surprised that so many people would rather standardise on a codec that is patented and under the control of a group that, by definition, exists to make as much money as possible through the codec’s use.

    The web has been this successful because it started free and open. I firmly believe that Theora would be the only sensible choice for the web, even if it were only comparable to obsolete patented codecs (which it isn’t). The reason I bother to post, however, is because of the glaring inaccuracies in the comments so far. It is also worth remembering that there is a lot to be gained financially by those supporting H.264, if it is accepted as a standard. Also, some companies involved in the debate will already purchase so many licences for H.264 that additional per-unit costs for licences are not as high. See:
    http://www.mpegla.com/avc/AVC_TermsSummary.pdf
    As pointed out, these terms could completely change in the future, especially if the web comes to rely on H.264.

    The fairest comparison page that I have found is this:
    http://people.xiph.org/~greg/video/ytcompare/comparison.html
    (also discussed here: http://people.xiph.org/~maikmerten/youtube/ ).

    As somebody has already pointed out, a lot of recent work (for example Thusnelda) has gone into improving Theora quality, so older encoders are no longer representative. There is also huge room for improvement, as I understand that a key design point of the codec was that huge advances could be made in the encoding without changing the decoder (ideal for embedded devices). A very honest (and now somewhat out of date, with the recent improvements to Theora) critique of Theora was provided here:
    http://web.mit.edu/xiphmont/Public/theora/demo.html
    which concluded:
    “This document […] says the current encoder is lacking compared to the very very best. It certainly is. Yet, the Theora format is entirely capable of closing with H.264 and MPEG-4 in terms of R-D while still requiring a fraction of the CPU time.”

    Reduced playback demands seem important to me, given the increasing trend to browse the web and explore rich content on small devices.

    Please stay strong on this, Mozilla!

  25. Dillon

    Here’s the thing, xiph’s comparison is only fair if Youtube is all that you are looking at. Youtube does not use good settings for their h.264 encodes. Additionally, as I’ve stated before, xiph uses poor settings for x264 in his other comparisons with theora and that makes it look as though theora is more competitive.

    Also any expert in video codecs will tell that theora simply cannot match h.264 in quality, or even probably vc1. This based on the simple fact of what avoiding patents allows. It’s not what it seems like at the current state but what is technically possible with the technologies implemented in theora. It simply cannot match h.264 in any objective sense. What I’ve heard is that maybe it can get up to the quality of a ASP encoded videos.

    As for CPU decode time, that is a major issue that the Mozilla team and developers of other browsers are going to have to work on. As is, it’s more efficient to decode h.264 through flash than theora through the video tag. These developers are leaping into the world of video playback and display – something they probably don’t know much about. It’s hard enough to get video to display properly in a dedicated player. It’s almost akin to how the ogg developers are coming from the world of audio encoding and are now dealing with video encoding.

  26. Petter

    Paul D, they can’t where would that be? Iran would pretty much be the only one. Software patents aren’t just enforced in the USA and Japan but Europe too. Where many of the holders are. And every country bound by international treaties. This is true for AAC, H264 and pretty much the rest. This is not about export regulations like encryption was. That wasn’t a patent problem. There’s nowhere to hide except the few free formats such as Theora, DIRAC and Schrödinger. The patent licenses just aren’t compatible with free software or Firefox. They could use MS Media Foundation/Directshow on Windows to provide playback though. But that would essentially be Silverlight so anyone developing such a site would probably just go that route then. And that has other drawbacks too.

Comments are closed.