web » Life of Andrew

The HTML5 audio tag

Mar 302010

I have been mucking around with the audio tag as part of my quest to understand where HTML5 is going. The <video> tag gets all the press but I think there are many more opportunities to use audio in web apps. HTML5 is closing the gap between plugin-based apps (Flash, Silverlight, Java, etc) and sound support is an important part of that goal.

(Those of you who don’t care how it works should go directly to the TV Themes demo puzzle. It works best in Firefox3.6 and the latest version of Safari, although most browsers should function to some degree.)

The audio tag is pretty flexible, able to handle both long form audio (songs and spoken passages – the theme medley on the demo page for example) and short snippets of background audio (alerts, and confirmations – the demo plays one of two short tones when you type an answer. Video game sound effects are another example.) Optionally, the audio tag can provide a user interface for starting and stopping the audio, useful for playing long streams of audio. Different browsers have different ideas about how this should look, but they all function much the same way.

In theory, the audio tag is as easy as embedding an image into HTML:

<audio controls>
	<source src="music.mp3">
	You can put HTML here that will be displayed if the browser does not understand the audio tag
</audio>

However, the devil is in the details. There are two problems with the audio tag that complicate matters. The first is that only the very latest browsers support the audio tag at all. This means that if you want to provide audio that everyone can use, you are going to have a fall-back method available. Before the audio tag, people used to use Flash for this purpose and it still works. A number of sites provide simple Flash-based audio players that you can embed – I ended up using the player provided by Google.

<object codebase="http://fpdownload.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=7,0,0,0" height="27" width="400" align="middle" classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000">
	<param name="_cx" value="10583"><param name="_cy" value="714"><param name="FlashVars" value="">
	<param name="Movie" value="http://www.google.com/reader/ui/3247397568-audio-player.swf?audioUrl=http://full/path/to/music.mp3">
	<param name="Src" value="http://www.google.com/reader/ui/3247397568-audio-player.swf?audioUrl=http://full/path/to/music.mp3">
	<param name="WMode" value="Window"><param name="Play" value="0">
 
	<param name="Loop" value="-1">
	<param name="Quality" value="High">
	<param name="SAlign" value="LT">
	<param name="Menu" value="-1">
	<param name="Base" value="">
	<param name="AllowScriptAccess" value="never">
	<param name="Scale" value="NoScale">
	<param name="DeviceFont" value="0">
	<param name="EmbedMovie" value="0">
 
	<param name="BGColor" value="">
	<param name="SWRemote" value="">
	<param name="MovieData" value="">
	<param name="SeamlessTabbing" value="1">
	<param name="Profile" value="0">
	<param name="ProfileAddress" value="">
	<param name="ProfilePort" value="0">
	<param name="AllowNetworking" value="all">
	<param name="AllowFullScreen" value="false">
 
	<embed type="application/x-shockwave-flash" src="http://www.google.com/reader/ui/3247397568-audio-player.swf?audioUrl=http://full/path/to/music.mp3" allowscriptaccess="never" quality="best" bgcolor="#ffffff" wmode="window" flashvars="playerMode=embedded" pluginspage="http://www.macromedia.com/go/getflashplayer" height="27" width="400" />
</object>

Not exactly elegant, is it? Apart from being uuuuug-ly, the full URI of the sound file must be used (the audio tag can use relative paths). Also, the Flash players are not scriptable in the same way as inbuilt audio tag is, which can make doing tricky stuff like animating other content in response to the audio more difficult.

The second problem with the audio tag is the same codec problem I talked about in a previous rant (The HTML5 Video Tag’s Fatal Flaw) For legal reasons, different browsers play different formats of audio – most notably Firefox will not play mp3s while Safari will not play ogg. There is no single format that will play in all browsers except for uncompressed wavs, which are too fat to be useful except for very short snippets.

To get around this problem the audio tag allows multiple files to be specified. The first file that the browser thinks it can play will be used, but it does mean you have to encode and store multiple versions of each audio file.

<!-- Only one of these files will be downloaded -->
<audio controls>
	<source src="music.ogg" type="audio/ogg">
	<source src="music.mp3" type="audio/mpeg">
</audio>

The demo page also uses the audio tag to play sound effects in the background, using audio elements that do not have a user interface. For simplicity I used wav files (download from this awesome source of free effects.) Since they have no user interface, Javascript must be used to play them:

<audio id="clicksound" preload="auto">
	<source src="click.wav" type="audio/wav">
</audio>
 
<script type="text/javascript">
function playSound( )
{
	var a = document.getElementById( "clicksound" );
	if ( !a ) return;
	if ( !a.play ) return; // will exit if the browser does not understand the audio tag
 
	a.play();
}
</script>

It is all pretty simple but as always there are problems. I did not find a good way of replicating this using Flash, so browsers that do not understand the audio tag do not play these background noises. Also, Google Chrome (which has otherwise excellent support) contains a weird bug that prevents it playing the first couple of seconds of an audio file, making it useless for short sounds. Apparently Firefox3.5 had the same problem, but it works perfectly in 3.6.

I created the demo to see if the audio tag could replicate the functionality of Flash-based applications for both long-form audio and background sound effects. It does seem to be possible provided you are targeting a modern browser and are prepared to work around certain annoyances. Hopefully the next few years will see an improvement in support for audio, I can see many uses for it especially if the iPad (which does not support Flash) takes off.

Python and The Very Slow Server

Computer, Uncategorized 1 Response »

Nov 282009

I don’t usually do a lot of Python programming, but I always enjoy it when the opportunity arises. Python is in no way a “clean” language, it has all sorts of warts and limitations that mean that it tends to not get used for big projects. Despite this (or maybe because of it), Python remains my go-to language for Getting Small Things Done Quickly. It is impossible to overstate the utility of just being able to start coding a function by bashing away at the python console – nothing else has given me the same sense of instant gratification since I started programming in BASIC back in the 80s.

The other big advantage of Python is the useful utility libraries that come with it as standard. Want to send twenty thousand emails? Just import smtplib. Want to generate code based on data from a spreadsheet? Import csv and away you go. Need a file that is exactly 32Mb is size? No problem. These are real examples from my job where Python has saved me many hours.

The most recent use I have put Python to is a slow server. For various murky and uninteresting reasons I need a rate-limiting HTTP server, one that I can easily control the speed at which it sends data. Enter Python’s very handy BaseHTTPServer module, which allows you to create custom HTTP servers with only a few lines of code by subclassing a request handler. Although the BaseHTTPServer is fairly useless for serving real files, it is perfect for this type of thing since it does all the boring work of parsing headers and returning status codes.

I don’t care about the contents of the data, just its size and how long it takes to serve. Since I will be varying these parameters a lot, I decided to make them part of each request so that each request could take a different amount of time – this means I don’t have to restart the server between each test run. Modifying the code to serve actual file data would be very simple.

I enjoyed writing this server so much that I regret that it didn’t take longer. Now I actually have to use it for its intended purpose, which I can assure you is not going to be as pleasant.

Here is the complete Python source:

# A very simple HTTP server designed to for testing situations where the data returned
# is not important but the rate at which it comes down is. This server can be started
# using the command: python delayserver.py
#
# Once started, it will listen for requests on port 8000
# Requests should be of the form http://<address>:8000/size=<bytes>,duration=<seconds>
# where: <bytes> is the size of the response data
# and    <seconds> is how long you want it to take (at minimum, it may take longer)
#
# Notes:
# * The timing is pretty inaccurate for small byte sizes, this isn't a problem for
#   what I need it for
# * Press ctrl-c to stop serving
 
 
import time
import BaseHTTPServer
 
class MyHTTPRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
 
	def do_GET(self):
		request = self.path.strip("/")
		duration = 1
		size = 1024
 
        	validRequest = False
		params = request.split(",")
		for p in params:
                    temp = p.partition("=")
                    if (temp[0] == "size"):
                        size = int(temp[2])
                        validRequest = True
                    elif (temp[0] == "duration"):
                        duration = int(temp[2])
                        validRequest = True
 
                if (validRequest == False):
                   self.send_error(404)
                   return
 
                self.send_response( 200 )
                self.send_header( "Content-Length", str(size) )
                self.send_header( "Pragma", "no-cache" )
                self.end_headers()
                self.slowWrite( self.wfile, size, duration )
 
 
        def slowWrite(self, output, size, duration):
                bytesWritten = 0
                startTime = time.time()
                while ( bytesWritten < size ):
                        now = time.time()
                        if (duration != 0):
                                desiredBytes = ( (now - startTime) / duration ) * size
                        else:
                                desiredBytes = size
                        desiredBytes = min( size, desiredBytes )
                        if (desiredBytes < bytesWritten ):
                                time.sleep(0.2)
                        else:
                                while (bytesWritten < desiredBytes):
                                        output.write('A')
                                        bytesWritten = bytesWritten + 1
                                output.flush()
                now = time.time()
                self.log_message( "Request took %f seconds",   now - startTime  )	
 
if __name__ == "__main__":
    http = BaseHTTPServer.HTTPServer( ('', 8000), MyHTTPRequestHandler )
    print "Listening on 8000 - press ctrl-c to stop"
    http.serve_forever()

I should point out that I am by no means an expert at Python, so take this code with a pinch of salt.

WordPress Upgrade Time

Computer 2 Responses »

Jul 302009

I have just upgraded to the latest version of WordPress. Usually this is a simple procedure, but this time something went wrong – attempting to view the blog threw up a blank page!

Luckily I could still get to the admin pages, so I knew my database was still OK (of course I hadn’t bothered with a backup, so I was worried for a minute.) A couple of other people had the same problem on the WordPress forums, but no solution was forthcoming there.

It turns out that part of the upgrade process had overwritten my index.php file with an almost empty file that contained only comments. Whatever caused this problem (db upgrade? Who knows?), it went away when I copied the file afresh. I am starting to see why most people use a dedicated blog hosting service…

I think everything is working now – let me know if you see anything weird.

Safari 4 is Pretty Good

Computer 1 Response »

Jun 132009

Safari 4 has been out for a couple of days now, and I must say I am enjoying using it. On the Mac, Safari has always had a great overall browser experience but Firefox always managed to stay my weapon of choice for viewing the Internet. This may change, Safari 4 is a very nice piece of software.

Apple is clearly wanting to make Safari an integral part of the Mac experience – Safari is very well integrated into the Mac OS (you can use Spotlight to search your history, passwords are stored in the keychain, etc), and the UI has all the polish you would expect from Apple. I particularly like the graphical Top Sites view that Safari presents when you first start it.

One thing I have always liked about Safari is the way it renders text and graphics. It always seemed to be just that little bit more polished than other browsers – correctly anti-aliasing fonts and respecting the embedded colour profiles of images. In my opinion Safari is still the best looking browser.

Apple are making a big song-and-dance about how Safari 4 is much faster than other browsers. What they mean is that they have included a very good Javascript JIT compiler which speeds up script-heavy sites by a large margin. This is excellent news, but Safari is hardly alone in focusing on Javascript performance and very recent versions of other browsers have very similar performance.

On Windows, Safari is something of an oddity. It works just as well and still renders sites better than most other Windows browsers, but its awkward neither-Mac-fish-or-Windows-fowl UI doesn’t help. The Windows version also lacks the smooth GUI animation that makes the Mac version so pleasant to use. Apple are doing their best to push Safari onto iTunes users, but I can’t see it taking off except among web developers, Safari’s built in web development tools are very cool.

There are still a few problem areas. Safari doesn’t seem to enjoy displaying animated GIFs, which often stutter before they are fully loaded. It very occasionally beachballs for a second on some pages, not all the time but enough to be annoying.

Finally, another rant about the HTML video tag (since the last one got quite a bit of attention): Safari supports the video tag but farms out the video to Quicktime. I suppose this is better than nothing, but you can tell that it is not well integrated. Videos do not show up in the list of page assets and it is clear that Quicktime is downloading the video itself, bypassing the browser’s cache. On top of this, performance is quite poor when the Javascript controls are used. The video quality is top-notch, but the experience is disappointing.

It sounds like I am dumping on Safari, but really they are minor niggles in a sea of greatness. I still think Firefox has the edge (particular Firefox3.5, which is shaping up nicely) but you could do worse.

The HTML5 Video Tag’s Fatal Flaw

Computer 36 Responses »

May 282009

Back in the day there was no standard way to publish video on the web. You could put any kind of video file you wanted on the server, but there was no guarantee that your readers would have the correct plugin required to view it. Everyone had to have a bunch of plugins installed to have any hope of viewing the majority of video files.

Flash video solve this problem. Flash was installed on nearly every computer anyway, so once they added a video decoder it seemed obvious to provide video content in Flash, even if it was in many ways not as good as the older plugins. Flash video uses massive amounts of processor time and slows down everything else on your computer. On the other hand, websites like Flash because it is easy to skin the player to fit in with the look of the site, and it makes downloading the raw video file (slightly) more difficult.

The <video> tag is supposed to replace Flash by linking to video files in the same way that the <img> tag links to images. In practice this is more complicated than it sounds because videos typically require the ability to skip and rewind content. This means that the browser must be prepared to download different parts of the file and cache things carefully to maintain performance. But these problems have long been solved.

I have been waiting for the big sites to make announcements, and today seems to be <video> day all over the internet. Both YouTube and Dailymotion have demo pages up showing <video> content:

YouTube’s HTML5 Page is designed to show how the <video> tag can replicate the functionality of their famous Flash-based player exactly. Unless you looked at the source (or your OS’s process monitor) you would never know you were using a different player.

The Dailymotion HTML5 Demo is even more impressive, using the <video> tag in combination with fancy Javascript to post-process the video and extract frames.

All this is very cool, but the two demos reveal the video tag’s fatal flaw : codecs. When the video tag was proposed, all browsers were supposed to support an unencumbered decoder named Ogg Theora (no seriously, that’s its name). There were just three problems:

Ogg Theora’s quality was not as good as other codecs
Although Ogg Theora was designed to be free of patent issues, it was felt that it may be a bit of a lightning rod for litigation.
Certain companies may have a vested interest in seeing their own codecs used.

So the requirement to support Ogg Theora was dropped. This means that although all HTML5 browsers will support <video>, there is no guarantee that they will be able to play and particular file. Firefox (at least the 3.5 beta) plays Ogg Theora, but Safari plays H.264 (a superior but expensive to license codec) but not vice versa. For instance, one of the demos above plays in Firefox, the other plays in Safari. This puts us in the farcical situation of having no standard way to publish video, exactly where we started.

There is also the small point that the most widely used browser (IE) does not support the video tag, and probably won’t for years. I predict that Flash video will be around for a while yet, and I am not happy about it.

New Zealand Public Holidays

Culture No Responses »

May 252009

Queen’s Birthday has snuck up on me again, as it does every year.

I can never seem to keep my public holidays straight, particularly the floating ones, but that is not going to be a problem anymore now that I have found that the Department of Labour publishes all of New Zealand’s public holidays in iCalendar format.

Now that I am subscribed to their feed I should never forget an upcoming day off again. It’s such a pity that after this weekend we don’t get anymore holidays until October.

Sketch This Page! Image Processing with the Canvas Tag

Computer, my projects No Responses »

Mar 262009

I have been quietly impressed with the progress web browsers have been making in recent years towards the goal of supporting a wide variety of applications. The promise of web-apps rivaling traditional desktop applications seems within reach after nigh-15 years of ballyhoo. Most recent browsers have extremely fast Javascript support and highly optimized DOMs, allowing a good level of interaction.

Of course, there a times when moving <div>s around just doesn’t cut the mustard. The canvas tag is not well used but is nothing less than a surface you can draw on using Javascript. It supports all the normal primitives (lines, arcs, fills, etc) and allows (indirect) access to the pixel data. While some people (eg: Project Bespin) are using the canvas tag in to offer extended functionality, I am busy fooling around.

Check out the fruit of my labour – Sketch This Page!.

It’s a Javascript bookmarklet that replaces each <img> element in a page with a same sized <canvas>. It works by copying the image to a temporary canvas, extracting the pixel data for some hacky post-processing, and then blatting the pixel-data onto the final canvas. The Javascript could certainly be better, but it works well enough and I am impressed with the speed.

The biggest flaw is that browsers will not allow Javascript to access the pixel data of images that are loaded from a different domain than the main page. This is a great idea from a security standpoint, but it does limit the usefulness of the bookmarklet. It would be great if Sketch This Page! worked on sites like Flickr, but sadly it is not to be.

For those interested, here is the function that actually dithers the image. I was going for a hatched look with the diagonal lines.

function generateBWDitherImage( src, dst )
{
var srcContext = src.getContext(“2d”);
var dstContext = dst.getContext(“2d”);

dstContext.putImageData( dstImageData, 0, 0 );
}

The New Zealand Web Harvest 2008 Harvests Too Much

Computer, Culture, my projects, WordMap 4 Responses »

Oct 142008

I was looking through the server logs for this site tonight and I noticed a user-agent that I hadn’t seen before. It turns out that the National Library of NZ is trying to take a snapshot of all New Zealand sites to archive as part of their mandate to preserve cultural history.
Continue reading »

Javascript and Saved Passwords

Computer 1 Response »

Oct 112008

First some facts…

Fact 1:
None of us can remember the passwords for the dozens of web sites we re all registered on. That is why web browsers all optionally store logon information and automatically fill out logon pages when we revisit a site.

Fact 2:
Web browsers do all support a special type of input control type specifically for passwords. Nothing entered into a password field is displayed, any characters are all displayed as asterisks. This prevents the password from being observed, either when it is first typed nor when it is automatically entered on subsequent visits to the page.

Fact 3:
Most web browsers allow you to type Javascript code into the address bar. This code is run in the context of the currently displayed document and has access to the object model.

Hmmmmmmmm…
Continue reading »

Chrome

Computer 2 Responses »

Sep 032008

So Google is going where angels fear to tread, and has released a browser of their own: Chrome. This is a very interesting move; I can think of a few reasons why Google might think a custom browser might be a good idea:

although everyone thinks of Google as a search engine they are really in the cryto-marketing field. Firefox plugins that remove ads must be a worrying development for them.
anything that helps people view more web pages is a win for Google, since more pages viewed equals more ad impressions.
having their own browser gets them a seat at the table when new web-standards are being created.
Google has a vested interest in promoting internet commerce, releasing a secure browser with safety features built in supports that goal.
most browsers have a search box that redirects to Google or another search engine. Chrome has a search/location bar that only redirects to Google – AdWords ahoy!
perhaps even sneakier, the search bar widget uses Google Suggest, so it is sending back information to Google as you type. I am sure Google can think of useful things to do with this information.
the search/location widget is a very interesting idea in its own right, effectively minimising the utility of domain names (which can be spoofed or camped.) Expect to see lawsuits fly when people work that their expensive .com domain name is not worth as much as they thought if this idea catches on.

I used Chrome for a couple of hours today at work (luckily I am in a line of work were this is not considered goofing off.) The interface is very clean and slick, even better than Safari. Having the location bar as part of the tab rather than above it make a lot of sense, and Chrome does it much better than IE, which I have always found visually confusing. I also like the way that Chrome uses the window title bar when full screen, giving you an extra few pixels of vertical height.

Chrome uses the infamous WebKit HTML layout engine, as seen in Safari and various Linux browsers. It is very fast at complex pages and supports all the fancy -webkit extensions to CSS. Sadly it does not include the excellent graphics renderer that comes with Safari, so fonts and images still look jaggy. Also, no support for @font-face – come on!

The network code seems pretty tight. Browsing through proxies worked well (a lot better than Safari) although it never seemed to do automatic NTLM authentication like IE.

There are a few omissions. Weirdly, Java applets do not seem to be supported. I am not sure if that is deliberate, or just something they haven’t gotten around to yet. No Mac version as yet, although it is apparently on the way.

Finally I must observe that commissioning a comic book for announce a product launch is one of the weirdly cool things I have ever seen.

A new font

Computer 2 Responses »

Jul 232008

To celebrate my blog reaching 100 published posts, I decided to buy it a new font. After researching font creation for a couple of days, I decided that it was all too complicated and sought professional help: Fontifier is a web service that takes the pain (and believe me it is a pain) out of making a font. For the princely sum of 9 US dollars and 10 minutes effort you too can have a reasonable looking typeface.

The font I got back after uploading my sample sheet was OK, but I wasn’t happy with some of the letterform and the kerning was not very good. Enter FontForge, an open-source font editor. Although a little buggy and a lot hideous, FontForge let me fix up the problems without too much trouble.

And now for a rant about embedded fonts: I see today that Microsoft is trying to resurrect their EOT font embedding technology. This was the next big thing 10 years ago – a way for the browsers to download custom fonts for display while supposedly protecting the font from being pirated. What it is in practice is a non-standard font format that is a pain in the neck for legitimate users and no hinderance to the fiendish font-pirates at all. Everybody saw this 10 years ago, which is why you have never seen a page with embedded EOT fonts even though the technology has been around for a decade – nobody can be bothered.

The upshot is that IE8 will not support standard OpenType fonts in stylesheets. This is terrible news – if you are viewing this page using Safari on the Mac then you are seeing text rendered in my handwriting. This is purely decorative (it is so, shut up!) but I can think of several more practical reasons why a page may want to embed a custom font – think mathematical equations or hieroglyphs. With two completely non-compatible font formats, few authors are going to make the effort, and everybody loses.

New WordPress Theme

Computer 1 Response »

May 292008

I updated to the lastest version of WordPress last week and I thought it was time for a new look. Unfortunately I have noticed that the RSS feeds don’t work in quite the same way as they did before, and although I like the general look of this theme there are some things that bug me a little. I will be dusting off my php coding skills and making some modifications over the next few weeks.

Tax Cut Calculator

my projects No Responses »

May 252008

Another budget has come and gone, this one more discussed than most for it actually contains the much promised but little seen tax cuts (read the Minister’s Executive Summary for details).

Here is a little project I whipped up to calculate the amount you are going to better off when the tax cuts come into force. All calculations are done in dollars, not the blocks of cheese that seem to have become currency over the last week. The current exchange rate: 1kg of tasty cheese = $NZ16.15.
Continue reading »

Safari 3.1

Computer No Responses »

Mar 262008

The browser wars are starting to hot up again. Apple is making a late play for cross platform browsing by releasing Safari 3.1 onto the world. Safari is MacOS X’s bundled browser – on the Mac it has always been pretty good, but the recent Windows versions have been terrible. Safari 3.1 is actually decently stable, and introduces Windows users to some nice features.

For a start, even on Windows Safari uses the Mac’s font rendering technology. Although opinions differ, I much prefer the Mac’s method of rendering fonts, to my eyes it is easier to read and more attractive. By comparison, standard GDI fonts in other Windows applications look spidery and harsh.

Safari’s other claim to fame is diligent adherence to up to date web standards. Features such as the canvas tag and SVG images are built in. Even better, advanced CSS properties such as animation and transforms allow for some cool effects – and some lame ones.

The Windows version of Safari does have some problems. A long standing issue is Safari’s reluctance to work with HTTP proxies. 3.1 is better than previous versions but still displays a distressing preference for crashing in a heap. Connecting directly to the internet works fine however.

It is hard to see exactly what Apple is trying to achieve with the Windows version of Safari. It seems unlikely that Safari will gain much traction against IE and Firefox (especially with Firefox 3 on the horizon). I suspect that Windows Safari is mainly intended for web developers – the more people who have access to Safari the more likely websites will be developed that work properly with Apple’s browser – including the all important iPhone. In any case, having another option, especially such a capable one, is not a bad thing.