Christian Heilmann

Posts Tagged ‘mozilla’

Lecturing at MIT about HTML5 Video – video, slides and (lots of) notes

Sunday, January 16th, 2011

I am currently in Boston and went to MIT to give a guest lecture on HTML5 multimedia as part of a lecture series by Mozilla leading up to a game competition. Here are the slides and some notes on what I talked about.

You can see “Multimedia on the web” on any HTML5 enabled device here (courtesy of vid.ly) or embedded here:

See the slides on Slideshare

Here are the notes of what I talked about:

Quick history of Multimedia on the web

It is sometimes a good idea to think back about what we did before we continue to move on. That gives us a chance to avoid repeating mistakes and to remember solutions to issues we face once more.

In the beginning, we had images as the only multimedia elements in browsers. As the connection speeds were bad we had to be very careful in what formats to use:

  • JPEG images have millions of colours but are lossy when you make them smaller in file size. You can see those “artifacts” when people save text screenshots as JPEG - the text looks fuzzy and is hard to read.
  • GIF images had a fixed number of 256 colours, had transparent pixels and could be animated images. Saving photos as GIF resulted in dithering to simulate more colours and larger files.
  • PNG was the first open format and a good middle ground – no animation though and IE doesn’t get the transparency right
  • WBMP was a special 2 colour format for mobile devices in WML pages

As the connection speeds were bad we came up with some very clever image formats and browser attributes:

  • Progressive JPG shows a highly artifacted and small (in filesize) version of the image first and increases its quality during download
  • Interlaced GIF loaded the GIF line by line – first lines 1, 3, 5, 7 and so and then 2, 4, 6 and so on – that way you saw what the image was about whilst it was loading
  • Netscape had a lowsrc attribute that allowed you to define a smaller image to show first which gradually got covered up by the full fat image

We animated with GIFs and later on with JavaScript animations – the latter giving us control over the animation but also requiring JS, which was not commonly used or turned on.

Audio on web sites was mostly MIDI background sounds – and let’s thank our lucky stars that this is over.

Next we used Java Applets for high fidelity animations and they gave a lot of web sites the charm of things found in the bargain bin at Hallmark.

Then RealPlayer came around. Now not on the “to have” list of anyone any more RealPlayer was awesome for its time: SMIL support, great streaming ability, supported by all platforms and very good compression. A shame that both the player and the encoders were closed and you had to pay for everything.

Other players that allowed video embedding were Quicktime, Windows Media Player, Shockwave and even Adobe Acrobat had some image conversion options. Other plugins like iPix allowed for zooming into images and VRML was the idea of having 3D worlds inside the browser.

All of them had the same problems though – they were plugins and thus alien to the browser. You needed to constantly upgrade them, they made the browser less stable and there was no interaction with the rest of the page.

Another issue is security – a lot of security exploits of browsers actually work by attacking plugins as they have deeper access into the operating systems than JavaScript itself.

After a while, Flash came out as the main plugin people would use for multimedia on the web. It came with great abilities, a good set of editors and libraries and allowed you to show and – more importantly – protect your videos from being downloaded. Right now DRM will be one of the the main reasons to use Flash movies instead of going native with HTML5.

Annoyances with Flash

The first thing I found lately with Flash is bad performance. On my MacBook air watching a few YouTube videos triggers the fan of the processor – something HTML5 Video doesn’t do. The reason is that Flash is a one size fits all solution for audio, video, animation and lately also 3D animation.

Whilst Air and Flex allows you to build movies with external editors, you mostly still use Flash builder to create your files instead of the editor you use to build the rest of the page.

No matter how cool Flash is, it is still an alien, black box in the browser and not available to manipulate from the outside. This means first of all bad accessibility:

  • You can’t reach Flash videos with your keyboard on browsers othe than IE - regardless of keyboard controls inside the video
  • Keyboard access in the movies needs to be added by the developers.
  • Whilst audio and video can be powerful tools to aid people with learning difficulties it is tough to make them available to them because of the two points above

The way to make Flash talk to the rest of the page is to provide APIs to your Flash controls. YouTube for example has an API that allows you to write your own player controls that are keyboard accessible. I’ve used this API in the past to build Easy YouTube. Some things I could not do though – like providing subtitles and captions in an accessible manner.

HTML5 audio and video

And this is where HTML5 comes to the rescue – it comes with native audio and video elements that are not in a black box but part of the browser and as accessible for you to manipulate like a block of text, an element or an image is. There are many upsides to native multimedia controls:

  • Better accessibility
  • Better performance – the elements do one thing well rather than being a catch-all solution
  • Much simpler API
  • Allows for styling and overlays – covering Flash movies with other elements in Internet Explorer was always a pain
  • View-source “hackable” – changes can be done in plain HTML rather than needing changes in another editor and compilation

Take Oprah’s new web site for example – hovering over any of the big pictures reveals a video. This is HTML5 video and thus easy to maintain in a CMS without the whole page being a Flash movie. It is also much higher quality and smaller file size than animated GIFs.

Painful stuff – codecs and conversion

Of course not all is sunshine, bunnies and daisies in HTML5 video and audio land and the reason is copyright and intellectual property. An open system using closed video and audio encodings cannot work. Video and audio gets converted to smaller, stream-able formats before it goes on the web. There is no such things as an “AVI file” or “MOV file” – these are container formats with audio and video streams all using different codecs to make them small. The issue is that these systems are all copyrighted and you need to pay money to create files in these formats. Which is why we needed open formats for HTML5.

We now have H.264 or MP4 files that are open but we might need to pay for them later, we have Ogg Theora files that are open and Google released VP8 or WebM to have a new, high quality and small filesize format for the web. Our job now is to convert proprietary format videos as created by our digital cameras into these open formats. There are a lot of tools for that:

  • Audacity is a full-fledged audio editor that allows you to save OGG audio files.
  • WebM tools allow you to create files in WebM format.
  • Evom is another encoder.
  • VLC is not only an awesome video player but also allows recording and encoding.

  • Ogg convert, Firefogg and TinyOgg are all OGG converters.

  • For Mac there is Miro Video Converter which is free and open source to easily convert Videos to WebM.
  • MPEG StreamClip converts videos to MP4 for Mac and Windows.
  • ffmpeg is a command line tool that most of the above solutions use in one way or another.

A very simple way to convert a video to all the necessary formats is to host it on Archive.org. If your video is licensed with Creative Commons archive.org does not only offer you full-length hosting (YouTube only allows 15-30 minutes) but also converts the video to OGG and MP4 automatically for you.

Embedding

Embedding HTML5 video into your document is incredibly easy:


Anything inside the audio or video that will be displayed when the browser doesn’t support them – in this case a not very helpful message but an important one. Normally you’d use this to offer a link to download the audio or video file or a Flash fallback (if you really must).

Right now, that doesn’t do anything though. If you want to display a video that the user can control (jumping to another time, changing the volume, playing our pausing the video) all you need to do is add a controls attribute:


With that you have a video that can be controlled via mouse or keyboard – it is that easy. The controls look different from browser to browser and some have features others don’t. Safari for example is the only browser that allows the video to be full screen. Other browser makers decided against that option to prevent people from using video for phishing purposes.

In order to support all the capable browsers you need to offer the video in at least two formats:

Use MP4 as the first format to ensure that iOS devices play it. Make sure to add a type attribute to each source as otherwise browsers will load a small part of each file to determine the type.

Additionally you can use the media attribute to serve different quality movies to different devices. The following example sends a high quality video to devices that are wider than 800 pixels and a lesser quality version to others (when using MP4):

Other attributes you can use in the video and audio tag are:

  • poster – define a picture to show before loading.
  • height/width – dimensions of the video
  • loop – automatically restart the video or audio
  • preload (auto/none/metadata) – when set to auto the browser preloads the video/audio when it is possible (most mobile browsers will not to save bandwidth, though). When set to metadata only the necessary data gets loaded to show the length of the media and set the right dimensions

Controlling

As there are differences in the controls across browsers you might want to create your own player controls. This is pretty easy. You have a few methods to use:

  • load() – load a new media.
  • canPlayType(type) – returns probably, maybe and “” (really!)
  • play() – play the movie
  • pause() – pause the movie.
  • addTrack(label,kind,language) -for subtitles

You also have a few properties to control and read the state of different parts of the video/audio file:

  • Video details
    • width
    • height
    • videoWidth
    • videoHeight
    • poster
  • Controls
    • controls
    • volume
    • muted
  • Tracks
    • tracks
  • Network state
    • src
    • currentSrc
    • networkState
    • preload
    • buffered
  • Ready state
    • readyState
    • seeking
  • Playback state
    • currentTime
    • startTime
    • duration
    • paused
    • defaultPlayBackRate
    • playbackRate
    • played
    • seekable
    • ended
    • autoplay
    • loop

This is a lot to play with. For example to create a simple play button for an audio file you might have a button element in the page with the class button. To make it functional all you need to do is this:

var audio = document.getElementsByTagName(‘audio’)[0];
var play = document.getElementsByClassName(‘play’)[0];
play.addEventListener(‘click’,function(e){
var t = e.target;
if(audio.paused){
audio.play();
t.innerHTML = ‘pause’;
} else {
audio.pause();
t.innerHTML = ‘play’;
}

e.preventDefault();
});

You check if audio.paused is true and call play() when it is, otherwise you call pause(). Change the innerHTML of the button at the same time and you are doen.

However, simply checking the state of an object is not safe. It makes much more sense to actually listen to events fired by the audio/video object. And there are a lot of interesting events to listen to:

  • loadstart
  • progress
  • suspend
  • abort
  • error
  • emptied
  • stalled
  • play
  • pause
  • loadedmetadata
  • loadeddate
  • waiting
  • playing
  • canplay
  • canplaythrough
  • seeking
  • seeked
  • timeupdate
  • ended
  • ratechange

Using these events, the player button code changes slightly:

video.addEventListener(‘play’, playEvent, false);
video.addEventListener(‘pause’, pausedEvent, false);
video.addEventListener(‘ended’, function () {
this.pause();
}, false);
function playEvent() {
play.innerHTML = ‘pause’;
}

function pausedEvent() {
play.innerHTML = ‘play’;
}

play.onclick = function () {
if (video.ended) { video.currentTime = 0; }
if (video.paused){
video.play();
}else{
video.pause();
}

};

No magic there.

If you want to see all the events and properties in action (and more importantly see if a browser supports them or not) check out the Media events demo page at the W3C web site.

One very good trick for syncing video and other effects is using the currentTime property. One example of that is the Spirit of Indiana demo where we sync a Google Maps animation with a video. The main gotcha is that the timeupdate event fires a lot and not necessarily every second. Therefore you need to throttle the changes to full seconds by using parseInt(). You can see this in action at the video demo:




window.addEventListener(‘load’,
function() {
var stage = document.getElementById(‘stage’);
var log = document.getElementById(‘logger’);
var seqlog = document.getElementById(‘loggersequences’);
var v = document.getElementsByTagName(‘video’)[0];
but = document.createElement(‘button’);
but.innerHTML = ‘Click to see Lindbergh’s flight’;
stage.appendChild(but);
but.addEventListener(‘click’,function(e) {
v.play();
e.preventDefault();
},false);
var now = 0;
v.addEventListener(‘timeupdate’,function(o){
log.innerHTML = v.currentTime;
var full = parseInt(v.currentTime);
if(full >= now) {
seqlog.innerHTML = now;
now = now + 1;
}

},false);
},
false);

Transforming

The real power of HTML5 and native video is when you can see how easy it is to transform the video element in the page. Here are some demos:

  • Transforming Video shows how simple it is to create a player that allows you to zoom inside a video and rotate it using CSS transformations
  • Paul Rouget’s round video demo shows how to create a play/pause button in SVG, greyscale a video and rotate it using CSS transformations
  • Paul’s Video Mashup demo shows how to apply SVG filters like Gaussian Blur, masking and how to apply CSS transforms to a running video

Simply check the source codes on these demos to see how they are done – the power of an open web.

Realtime changes

Adding filters and transforming the video container with CSS is one thing. Using Canvas and copying frames from the video into it you can do realtime changes to the video itself, which is really nice.

  • Remy Sharp’s round video demo shows how to create a rotating, bouncing circle that contains a video
  • Paul Rouget’s Green demo shows how to simulate a green screen effect with open web technologies. You copy each frame over to a canvas and read out the RGB value of each pixel. When it is green, just set its opacity to allow the picture behind the video to shine through.
  • The dynamic content injection by Paul shows how to analyse a video and find white parts to use as corner points for an embedded canvas. You can then inject other videos, canvas animations or images into the other video.
  • The Tracker demo shows how to detect human shapes and movement in a video
  • The edge detection demo shows how to detect edges in a live video using JavaScript

Awesome audio stuff

As you can see there is a lot of cool realtime things you can do with video. Audio is less sexy somehow as you can play and maybe play backwards (only works in Safari at the moment). Wouldn’t it be cool to read out the audio data or dynamically generate sounds though?

There is a Firefox4 API out there that allows you to do that. Read the introduction to the API or check the Audio API Wiki to get to know it.

One cool demo using this API is the HTML5 Guitar Tab generator and there is already a library out there called sfxr that allows you to create audio on the fly.

Interviewed Bruce Lawson for Mozilla

Thursday, January 6th, 2011

Over on the Mozilla Blog I just started a new series of blog posts – interviews introducing “People of HTML5”. My first victim was Bruce Lawson of Opera:

Bruce Lawson presents The HTML 5 experiments

Go and read the interview with Bruce Lawson and check and download the video for, err, private moments.

Banging the drum for the open web in Barcelona – Drumbeat Festival 2010

Monday, October 18th, 2010

If you care about the open web and you want to be part of sustaining it by teaching the amazing web people of the future, get your butt down to Barcelona from the 3rd to the 5th of November for the Mozilla Drumbeat festival.

Mozilla Drumbeat

One of the cool things of the web is all the free content you can find. Creative Commons for video, audio and texts and Open Source for code have shown the closed commercial world for years now that you can reach more people, create better products and create more good-will than any of your marketing campaigns could ever do to boot.

coder vs. salesman

Especially in education we have to be aware that there is a painful divide between what schools and universities teach and what is needed in the market. Outreach by companies in the form of University Hack Days and free talks is good but there should be a more formalised way to get kids to learn the new stuff rather than things that impress a professor but give you deer-in-headlight looks from interviewers.

Products like the Opera Web Standards Curriculum and WaSP interact already battle this and Mozilla wants to join the good fight with the P2PU School of Webcraft.

As an educator and trainer and someone who publishes a lot of learning material for free on the web I thought it very much down my alley to go to the conference, and hey – I will be there to take part, present and answer questions.

Should be a blast – see you there!

Promote better JavaScript documentation with PromoteJS

Sunday, September 26th, 2010

As I heard during my visit in the valley last week and just now from the goings-on at the JSConf in Berlin, Mozilla is working on ramping up the already amazing MDC documentation for JavaScript and you can help by providing some guerrilla SEO - simply add the following image with the right code available at http://www.promotejs.com/ to any article that talks about JavaScript:

JavaScript JS Documentation: JS Array splice, JavaScript Array splice, JS Array .splice, JavaScript Array .splice

The reason is obvious when you search for the term “JavaScript” – the things you find are wiki articles or even products that tell you not to care about learning JavaScript when you use them. As the term JavaScript was a hot SEO term during the DHTML days and with the resurrection now a lot of sites washed out the experience of finding something good when you try to use the web to find it. When I wrote my JavaScript book Beginning JavaScript with DOM Scripting and Ajax I had a chapter that told you where to find good information about JavaScript and other than the MDC and SelfHTML (if you are German or Spanish) I drew quite a blank. Lately Sitepoint started to put together some good docs but then all we had was W3Schools.

Let’s shine a Flashlight on the elephant in the room: W3Schools. This site is the number one for almost every technical search term in CSS, JavaScript and HTML. There are a few reasons for this:

  • Its SEO is spot on – heading structure, amount of links and deep level content (long tail)
  • It leaves you with a good feeling – “oh this is how simple it is! Cool, no need to read more, just copy and paste the example and I am an expert”
  • It is fast and small

The “good feeling” part is also what makes it dangerous. I ranted in several talks about the Siren Song of W3schools – it gives you the what but not the why. You can use it as a reference when you already know your stuff but it is dangerous for people to learn a subject matter. Programming is not about syntax alone – it is also about the environment you apply it in. W3schools is like bringing up your kid on fast food – quick, fast and cheap. This is not the fault of the maintainers of W3Schools – it is the fault of people sending new learners there for information.

The PHP docs are a good compromise of fast information with insights from implementers in the field – I learnt more from comments on the docs in PHP than from the entries themselves.

In the write great code examples chapter of my Developer Evangelism handbook I described a blueprint how to write a code example that explains and educates. Sadly this takes much more work than just showing the syntax. And this is why the docs sites that simply tell the “how” are winning – they update much faster with less overhead.

It is very cool to see Chris WillisonChris Williams(DOH!) and others take on the challenge and Mozilla to me is the perfect host for this kind of effort – they are open, independent when it comes to JS and have a track record of providing good docs.

Interestingly enough about five few years ago I tried a similar approach for all docs:

Replacing old tutorials with new ones - Obsoletely Famous by photo

Other than getting it hacked once I found no support for it though so I shut it down a year ago. Now with the leading names of the market talking about a similar thing at the leading conference of the language we have a better chance. Help us!

Mozilla Add-on workshop and Firefox4 with HTML5, CSS3 and SVG

Thursday, July 1st, 2010

Yesterday night Mozilla invited developers to the Hub in King’s Cross, London to learn about Add-ons and the future of Firefox.

Nick Nguyen on Add-ons

Nick Nguyen The evening started with Nick Nguyen with Mozilla giving us an overview of the reach and numbers of extensions that are currently available for Firefox and Thunderbird. Some of the numbers were:

  • To date there were 2 billion downloads
  • only 40% of the add-ons were created in the US (followed by Germany) – this is the same in the Yahoo Developer Network, the US is the main market but developers are elsewhere
  • 40% of Firefox users have add-ons (8 is the most common amount of installs)
  • People created 588 Themes
  • 1054 Search Engines were added to Firefox
  • Overall the community created 4441 Extensions
  • The surprising number was that people created 156,147 Personas (skins for Firefox)

This, according to Nick, makes the Mozilla stack the “richest extensions platform”.

The move to Jetpack

Nick then went on to explain the changes in add-on architecture, especially the move from XUL based add-ons to Jetpack. Whilst the old school XUL path still yields the most powerful results, Jetpack is the preferred way to build for the future, because of a few reasons:

  • user don’t need to restart the browser to install them
  • updates are a lot easier
  • you only need to know JS, CSS and HTML
  • there’s a user friendly web IDE - the addon builder.

Promoting your add-ons

Nick then explained some of the options for promoting your add-ons. Promotion is possible for targeted markets and promoted add-ons normally tend to get 5-10k downloads/day. Firefox 4 will have add-ons in a tab – full screen and featured instead of the currently somewhat cumbersome add-on manager.

The Rock your Firefox blog describes the story of add-ons and how it improves people’s lives. It is an editorial servers for developers. Featured add-ons get 2-3 days exposure and ten-thousands of downloads.

The Mozilla contributions program allows people to donate money for developers. So far 500 add-ons (12% overall) received money from the programme. It has run for 10 months and so far raised $90,000.

The next step for Mozilla now will be an add-on Marketplace – which is a store for add-ons. They want to try it out and see how it will work out. In a survey conducted amongst the add-on developer community 3 out of 4 developers like the idea of reaching 100 million active add-on users and sell their add-ons for 1 to 5 dollars. Mozilla is very outspoken about never to consider a DRM model and consider fair pricing the better way to fight piracy. The store will accept Paypal and all major credit cards. One of the key requirements is support for non US developers and the store encourages free add-ons that feature more options in the paid versions – much like the Android market does now.

Overall Mozilla like to see their planned app store not like a mall but like an organic farmers market.

Justin Scott – on making add-ons people will love

Icon Brushup challenge by photo Justin Scott talked us through some of the steps how to build a really good add-on and explained some of the pitfalls to avoid. His talk was structured in three stages of add-on deployment: building, marketing and listening.

Building your add-on

One of the biggest feedback items Mozilla users gave in a survey about add-on happiness is that they are expecting a clean first run of the add-on. things people didn’t like at all were:

  • Add-ons that require registration
  • modal dialogs and wizards
  • asking to change the user’s settings
  • pages that pop up after updating and changelogs

If you for example click on a link in an email you don’t want your Firefox to open a different pages as an add-on needed upgrading. This was one of the main reasons why the new add-on manager in Firefox 4 will not pop up any longer but will be much more discreet.

A few tips for good deployment:

  • Put your dialogs on the first run page or have it user triggered instead of forcing users to change setting when the add-on gets upgraded
  • Don’t clutter the UI - instead try to integrate your extension with the UI as if they are a part of Firefox.
  • Stick to one way of showing the extension and allow the user to choose where they want to see it – as an icon in the toolbar, on the bottom right, as a right-click extension and so on.

One bad example Justin showed was the Delicious extension which adds 7 logos to the Firefox chrome in different areas.

The next point Justin brought up is that it is very important to earn and keep the user’s trust. He explained some of the main mistakes that add-on developers do:

  • change the browser’s search provider for ad sales
  • replace ads on websites with own ads
  • install other add-ons with the one you agreed to install

This lead to the “no surprises policy” of Mozilla:

  • Every add-on needs to have a full disclosure of what it does (including the changes mentioned above)
  • Users have to opt-in to these changes
  • Changes have to revert on uninstall
  • All changes have to be included from the start and not on the second or third run

Some tips to avoid annoyances are:

  • Choose a core functionality and stick with it – don’t change the functionality on updates
  • Have a clear privacy policy – install the proper way – not with .exe.
  • Stay current: update compatibility before the next release, avoid conflicts with other add-ons, leverage new features like restartless add-ons, larger icons and mobile support
  • Build fast add-ons. Your add-on should not delay the start up of Firefox. Mozilla are working on tools to measure performance and tell developers how to change them.

Marketing for your add-on

Next up were some of the ideas how to market your add-on. Pretty basic advice but surveys in the community showed that these things constantly get done the wrong way.

  • The name of your add-on should mirror the functionality – don’t get too creative.
  • Have a nice icon (half the extensions don’t have an icon – even 40% of the approved ones – to battle this Mozilla is expanding the default “jigsaw icon” to more descriptive ones. They also offer a design help forum and run a icon redesign contest soon)
  • Provide helpful screenshots and keep them updated – video screenshots are in the planning
  • Avoid EULAs – Firefox now has different install buttons for EULA and non-EULA add-ons. Surveys showed that 44% of the plugins without a EULA got installed, but only 5% of those with a EULA. Another 40% of the installers dropped out after reading the EULA which means that only 3% got installed.

Listening to your audience

Justin’s last point was to listen to your users:

  • Have a good support avenue (email,website,get satisfaction)
  • Monitor user reviews – developer can reply to them
  • Use tools – statistics (how fast do people upgrade), recent activity feed as RSS, compatibility reporter

Myk Melez on Jetpack

Live coding of an extension by photo Next up was Myk Melez with a deep dive on Jetpack. He started with a history and description of Jetpack which came from the Mozilla Labs. The reasons for Jetpack were that it was too hard to build add-ons, that browser updates broke add-ons, that restarting your browser is annoying and that add-ons have system access but probably don’t need it (which of course is a security issue aswell).

The goals of Jetpack subsequently are to simplify add-on development with CSS and JavaScript, to work with upgrades, to limit the power of plugins and to avoid restarts.

Jetpack came around in Spring 2009 and had these features but still allows for full system access. People liked it a lot, especially that you could build add-ons inside the browser which is great for casual developers.
There were however some issues with bootstrapping into Firefox and compatibility with updates and there was no way yet to harness third-party APIs in your extensions with Twitter being the most requested.

In Fall 2009 there was a reboot of Jetpack which included a web application for casual developers, an SDK for developers, XPI bundles + API bundled with add-in to solve bootstrapping and CommonJS Modules for code sharing with third parties.

In 2010 the webapp and SDK came out as a incubation programme of Mozilla instead of labs level and after five SDK releases, Jetpack is now restartless on 4 and betas, has a unit testing harness and documentation tool. The web app allows you to create add-ons and libraries and instant testing via a helper add-on.

Myk then did a live demo building a new extension using the of the add-Ons Builder (which is Bespin inside the browser allowing you to build extensions).

The future of Jetpack will have the limiting of system access, new APIs for window and sidebar, debugging options (console and memory management), compatibility across updates and the web app will get addons.mozilla.org integration.

You can read all about Jetpack at jetpack.mozillalabs.com and try out the builder yourself at builder.mozillalabs.com.

Lightning Talks

After a quick break with awesome food and drinks (really) we headed for some lightning talks.

Artzilla by photoArtzilla's China simulator by photomobile firefox add-ons by codepo8

First up was Evgeny Shadchnev showing his Invisible hand price comparison Add-On. Specifically he explained how they moved from XPI development to Jetpack and how the test driven design allowed them simple development in a way no other browser extension platform does.

Next was Tobias Leingruber from Artzilla.org who talked about the lighter side of add-on development and hacking the web to be a funnier and more interesting place. I loved the talk as I myself have this mean streak for my add-ons (check Touretter as an example). Tobias talked about things artzilla had built, like a Dislike button for Facebook, the China channel firefox add-on that simulates the internet censorship in China and an add-on that finally removes Justin Bieber from the internet – thanks guys!.

Brian King talked about building add-ons for Firefox Mobile and pointed out the slow performance on handsets, the need for a different UI and how you can build cooler add-ons as you can tap into orientation, gps and camera. He also explained that you don’t need a phone to develop as there are nightly builds of desktop Fennec and a Maemo virtual image for Maemo apps (simulating a Nokia N900). One big tip was the Addon collector extension which allows for bunch syncing on extensions from desktop to mobile.

Paul Rouget and Tristan Nitot on Firefox 4 and HTML5

Paul and Tristan from the Paris office were next and blew our minds with some of the cool stuff coming up in Firefox 4
Mozilla Add-on evening by photoPaul Rouget by photoHTML5 embedded inside SVG by photo

A reminder of what XUL is about

Paul started somewhat on a tangent by showing 3 tips for your first add-on. His main point was that you should understand XUL, CSS and JavaScript to build great add-ons (which somewhat contradicted the Jetpack talk but eh, he is French :) SCNR).

Paul explained that the Firefox UI is a big web page. Firefox itself is XUL and you can open it in Firefox itself with chrome://content/browser/browser.xul. He then showed how to build a browser in XUL using the XUL explorer in 20 lines of XML. Once you understood that the whole browser is a XUL document, the next step is to understand Overlay which means that you add to the browser’s chrome. You can find out where to inject your code using the DOM inspector.

All you need to do then is to write a manifest file and an install.rdf and package up your extension. All of this is described in detail at https://addons.mozilla.org/en-US/developers.

New features in Firefox

Firefox is getting a bad reputation right now for being slow and unstable. A lot of these issues are actually not the fault of the main engine but add-ons and – especially on a mac – plugins. Whenever my Firefox dies it is because of some Flash movie. This is why from 3.6.4 onwards Firefox will have a crash protection which means that if a Flash, Quicktime or Silverlight plugin crashes, the browser stays up and just tells you to reload the current page as
the plugin caused an issue.

Tristan re-iterated Mozilla’s cause and idea of the web as an open and free platform and that they are working hard to make it easy for developers to build with free tools what has been done with closed source platforms up to now.

Tristan and Paul then went through some Firefox 4 and Firefox Mobile 2 demos – some of which I had seen before, and others that impressed me a lot. Things Paul showed were:

  • HTML5 Video display
  • Painting with Canvas
  • Image manipulation with Canvas – pixel testing, face detection with opencivitas
  • Green screen technologies in images and video by detecting pixel colours.
  • HTML5 embedded inside SVG (yes!)
  • SVG as an IMG
  • SVG as a CSS background
  • SVG filter/mask/clip
  • SVG animations
  • Inline SVG inside HTML5
  • CSS3 (selectors, @font-face, 2D Transforms, Transitions, Shadow, Gradients ,calculations – calc(2em-10px) )
  • APIs: Geolocation, Offline (IndexDB, localStorage, AppCache, FileAPI – binary content of a file input, file drag and drop, web workers, websockets)
  • Websockets controller running the presentation from the mobile.
  • WebGL

If you want to see some of the cool things they showed, here are some videos:

CSS3 filters and SVG masking on HTML5 Video:

Highly interactive video interface with SVG masking and transitions:

WebGL in Firefox 4 and on Android:

After this, Tristan covered some of the other features of Firefox 4, especially the upcoming speed improvements:

  • TraceMonkey (a new Javascript engine)
  • Lazy Frame Construction
  • Reducing I/O from the main thread
  • Startup Time
  • Hardware accelleration
  • GPU text/graphics/video rendering
  • Using GPU for text rendering
  • JavaScript JIT (JaegerMonkey)
  • HTML5 parser running own thread
  • slicker interface
  • no more modal dialogs
  • no startup interuptions
  • updates in the background

Other features of the upcoming interface are:

  • switch to tab (awesomebar storing open tabs)
  • dedicated application tabs (icon tab instead of text)
  • Permission control per web site – all the different options by domain
  • extensions as own page

The plan is to ship RC1 in October – let’s see if that works out :)

Summary

I had a great time at the event and I am convinced again that Mozilla are doing a great job in building the infrastructure for the Open Web. If you think only webkit is an HTML5 browser, check out some of this. One of my favourite parts of the whole demo at the end was that all of the examples are progressively enhanced, so if you watch them in a browser that is not Firefox you still get the features supported in that browser – something that the Apple showcase of HTML5 for example doesn’t bother to do.