Christian Heilmann

Posts Tagged ‘video’

FUD again – Flash vs. HTML(5) – yes, open things are easy to retrieve

Wednesday, January 26th, 2011

I am very disappointed with a post released today by Serge Jespers, Adobe Evangelist entitled Stealing content was never easier than with HTML5.

I am mostly disappointed by the headline which I hope is there for link-baiting and Google juice. In other words, really cheap and lame propaganda.

The post has a good intention: there is a problem with open media standards in HTML5 that there is no way to protect premium content from being downloaded. There is no DRM, there is no encryption or watermarking. If we want premium content publishers on the bandwagon, then we need to think about that. Posts like this with this headline make it hard for us to even reach people to talk about the options. It is FUD and Adobe as a company that claims to support HTML5 should know better.

Right now, I do tell people who are paranoid about their content to use Flash as it has a certain degree of protection against simple downloading. If Adobe’s official spokespeople keep spouting messages like this, I will move to tell them to use Silverlight.

The argument that it is easy to download video in HTML5 as you can see it in the source is like saying that it is easy to steal newspapers in train stations as they are easy to reach. It is invalid – that you can steal it is not the issue here, the issue is what you do with the paper. Do you take it from the stand and go and pay it without looking at it? Do you read it on the spot and then put it back? Do you leave with it without paying? Or do you check the headlines and when you are intrigued you go and pay for it? Do you take the paper, go to a copier, copy it and then try to sell the copies? Fact is – it is dead easy to get the paper, the same way it is dead easy to get a video online.

I find myself many a time downloading YouTube videos as I am on the go a lot. Being on the go (and considering data plans and roaming) means I have no connection, or I have a flaky connection that downloads the movie to minute 3 and then stalls while the fan of my laptop spins. Instead, at home when I got my fat connection I download some talks and screencasts and watch them offline using VLC. Then I delete them – or I blog about them first linking to YouTube and advertise them that way.

To download these extremely well protected Flash movies I use one of the dozens of services, browser extensions or apps out there.

Allowing people to download a movie means you get a bigger audience. If you make people jump through hoops to watch a movie or make them watch a 5 minute ad for a 10 second film you shouldn’t be surprised if they use P2P filesharing or Hotfile, Rapidshare, Fileserve, Megaupload and dozens of other services to download the movie in 10 seconds and then watch it at their leisure.

Recognise something? The mere fact that there are thousands of downloaders for YouTube and paid for hosting services that offer easy to download pirated copies means that there is a demand for that feature. A feature that would make me damn happy to have on YouTube and I would pay for it.

So instead of demonising HTML5 as the backdoor that will allow evil doers to steal your goodies maybe it is time for premium content providers to open up to the needs of web users, and find a way to publish previews of content and the full content for subscribed users. There are ways to make money and share your products – but not if you spend most of your time and money on things that seemingly give you protection but in reality are just a glass shield.

Lecturing at MIT about HTML5 Video – video, slides and (lots of) notes

Sunday, January 16th, 2011

I am currently in Boston and went to MIT to give a guest lecture on HTML5 multimedia as part of a lecture series by Mozilla leading up to a game competition. Here are the slides and some notes on what I talked about.

You can see “Multimedia on the web” on any HTML5 enabled device here (courtesy of vid.ly) or embedded here:

See the slides on Slideshare

Here are the notes of what I talked about:

Quick history of Multimedia on the web

It is sometimes a good idea to think back about what we did before we continue to move on. That gives us a chance to avoid repeating mistakes and to remember solutions to issues we face once more.

In the beginning, we had images as the only multimedia elements in browsers. As the connection speeds were bad we had to be very careful in what formats to use:

  • JPEG images have millions of colours but are lossy when you make them smaller in file size. You can see those “artifacts” when people save text screenshots as JPEG - the text looks fuzzy and is hard to read.
  • GIF images had a fixed number of 256 colours, had transparent pixels and could be animated images. Saving photos as GIF resulted in dithering to simulate more colours and larger files.
  • PNG was the first open format and a good middle ground – no animation though and IE doesn’t get the transparency right
  • WBMP was a special 2 colour format for mobile devices in WML pages

As the connection speeds were bad we came up with some very clever image formats and browser attributes:

  • Progressive JPG shows a highly artifacted and small (in filesize) version of the image first and increases its quality during download
  • Interlaced GIF loaded the GIF line by line – first lines 1, 3, 5, 7 and so and then 2, 4, 6 and so on – that way you saw what the image was about whilst it was loading
  • Netscape had a lowsrc attribute that allowed you to define a smaller image to show first which gradually got covered up by the full fat image

We animated with GIFs and later on with JavaScript animations – the latter giving us control over the animation but also requiring JS, which was not commonly used or turned on.

Audio on web sites was mostly MIDI background sounds – and let’s thank our lucky stars that this is over.

Next we used Java Applets for high fidelity animations and they gave a lot of web sites the charm of things found in the bargain bin at Hallmark.

Then RealPlayer came around. Now not on the “to have” list of anyone any more RealPlayer was awesome for its time: SMIL support, great streaming ability, supported by all platforms and very good compression. A shame that both the player and the encoders were closed and you had to pay for everything.

Other players that allowed video embedding were Quicktime, Windows Media Player, Shockwave and even Adobe Acrobat had some image conversion options. Other plugins like iPix allowed for zooming into images and VRML was the idea of having 3D worlds inside the browser.

All of them had the same problems though – they were plugins and thus alien to the browser. You needed to constantly upgrade them, they made the browser less stable and there was no interaction with the rest of the page.

Another issue is security – a lot of security exploits of browsers actually work by attacking plugins as they have deeper access into the operating systems than JavaScript itself.

After a while, Flash came out as the main plugin people would use for multimedia on the web. It came with great abilities, a good set of editors and libraries and allowed you to show and – more importantly – protect your videos from being downloaded. Right now DRM will be one of the the main reasons to use Flash movies instead of going native with HTML5.

Annoyances with Flash

The first thing I found lately with Flash is bad performance. On my MacBook air watching a few YouTube videos triggers the fan of the processor – something HTML5 Video doesn’t do. The reason is that Flash is a one size fits all solution for audio, video, animation and lately also 3D animation.

Whilst Air and Flex allows you to build movies with external editors, you mostly still use Flash builder to create your files instead of the editor you use to build the rest of the page.

No matter how cool Flash is, it is still an alien, black box in the browser and not available to manipulate from the outside. This means first of all bad accessibility:

  • You can’t reach Flash videos with your keyboard on browsers othe than IE - regardless of keyboard controls inside the video
  • Keyboard access in the movies needs to be added by the developers.
  • Whilst audio and video can be powerful tools to aid people with learning difficulties it is tough to make them available to them because of the two points above

The way to make Flash talk to the rest of the page is to provide APIs to your Flash controls. YouTube for example has an API that allows you to write your own player controls that are keyboard accessible. I’ve used this API in the past to build Easy YouTube. Some things I could not do though – like providing subtitles and captions in an accessible manner.

HTML5 audio and video

And this is where HTML5 comes to the rescue – it comes with native audio and video elements that are not in a black box but part of the browser and as accessible for you to manipulate like a block of text, an element or an image is. There are many upsides to native multimedia controls:

  • Better accessibility
  • Better performance – the elements do one thing well rather than being a catch-all solution
  • Much simpler API
  • Allows for styling and overlays – covering Flash movies with other elements in Internet Explorer was always a pain
  • View-source “hackable” – changes can be done in plain HTML rather than needing changes in another editor and compilation

Take Oprah’s new web site for example – hovering over any of the big pictures reveals a video. This is HTML5 video and thus easy to maintain in a CMS without the whole page being a Flash movie. It is also much higher quality and smaller file size than animated GIFs.

Painful stuff – codecs and conversion

Of course not all is sunshine, bunnies and daisies in HTML5 video and audio land and the reason is copyright and intellectual property. An open system using closed video and audio encodings cannot work. Video and audio gets converted to smaller, stream-able formats before it goes on the web. There is no such things as an “AVI file” or “MOV file” – these are container formats with audio and video streams all using different codecs to make them small. The issue is that these systems are all copyrighted and you need to pay money to create files in these formats. Which is why we needed open formats for HTML5.

We now have H.264 or MP4 files that are open but we might need to pay for them later, we have Ogg Theora files that are open and Google released VP8 or WebM to have a new, high quality and small filesize format for the web. Our job now is to convert proprietary format videos as created by our digital cameras into these open formats. There are a lot of tools for that:

  • Audacity is a full-fledged audio editor that allows you to save OGG audio files.
  • WebM tools allow you to create files in WebM format.
  • Evom is another encoder.
  • VLC is not only an awesome video player but also allows recording and encoding.

  • Ogg convert, Firefogg and TinyOgg are all OGG converters.

  • For Mac there is Miro Video Converter which is free and open source to easily convert Videos to WebM.
  • MPEG StreamClip converts videos to MP4 for Mac and Windows.
  • ffmpeg is a command line tool that most of the above solutions use in one way or another.

A very simple way to convert a video to all the necessary formats is to host it on Archive.org. If your video is licensed with Creative Commons archive.org does not only offer you full-length hosting (YouTube only allows 15-30 minutes) but also converts the video to OGG and MP4 automatically for you.

Embedding

Embedding HTML5 video into your document is incredibly easy:


Anything inside the audio or video that will be displayed when the browser doesn’t support them – in this case a not very helpful message but an important one. Normally you’d use this to offer a link to download the audio or video file or a Flash fallback (if you really must).

Right now, that doesn’t do anything though. If you want to display a video that the user can control (jumping to another time, changing the volume, playing our pausing the video) all you need to do is add a controls attribute:


With that you have a video that can be controlled via mouse or keyboard – it is that easy. The controls look different from browser to browser and some have features others don’t. Safari for example is the only browser that allows the video to be full screen. Other browser makers decided against that option to prevent people from using video for phishing purposes.

In order to support all the capable browsers you need to offer the video in at least two formats:

Use MP4 as the first format to ensure that iOS devices play it. Make sure to add a type attribute to each source as otherwise browsers will load a small part of each file to determine the type.

Additionally you can use the media attribute to serve different quality movies to different devices. The following example sends a high quality video to devices that are wider than 800 pixels and a lesser quality version to others (when using MP4):

Other attributes you can use in the video and audio tag are:

  • poster – define a picture to show before loading.
  • height/width – dimensions of the video
  • loop – automatically restart the video or audio
  • preload (auto/none/metadata) – when set to auto the browser preloads the video/audio when it is possible (most mobile browsers will not to save bandwidth, though). When set to metadata only the necessary data gets loaded to show the length of the media and set the right dimensions

Controlling

As there are differences in the controls across browsers you might want to create your own player controls. This is pretty easy. You have a few methods to use:

  • load() – load a new media.
  • canPlayType(type) – returns probably, maybe and “” (really!)
  • play() – play the movie
  • pause() – pause the movie.
  • addTrack(label,kind,language) -for subtitles

You also have a few properties to control and read the state of different parts of the video/audio file:

  • Video details
    • width
    • height
    • videoWidth
    • videoHeight
    • poster
  • Controls
    • controls
    • volume
    • muted
  • Tracks
    • tracks
  • Network state
    • src
    • currentSrc
    • networkState
    • preload
    • buffered
  • Ready state
    • readyState
    • seeking
  • Playback state
    • currentTime
    • startTime
    • duration
    • paused
    • defaultPlayBackRate
    • playbackRate
    • played
    • seekable
    • ended
    • autoplay
    • loop

This is a lot to play with. For example to create a simple play button for an audio file you might have a button element in the page with the class button. To make it functional all you need to do is this:

var audio = document.getElementsByTagName(‘audio’)[0];
var play = document.getElementsByClassName(‘play’)[0];
play.addEventListener(‘click’,function(e){
var t = e.target;
if(audio.paused){
audio.play();
t.innerHTML = ‘pause’;
} else {
audio.pause();
t.innerHTML = ‘play’;
}

e.preventDefault();
});

You check if audio.paused is true and call play() when it is, otherwise you call pause(). Change the innerHTML of the button at the same time and you are doen.

However, simply checking the state of an object is not safe. It makes much more sense to actually listen to events fired by the audio/video object. And there are a lot of interesting events to listen to:

  • loadstart
  • progress
  • suspend
  • abort
  • error
  • emptied
  • stalled
  • play
  • pause
  • loadedmetadata
  • loadeddate
  • waiting
  • playing
  • canplay
  • canplaythrough
  • seeking
  • seeked
  • timeupdate
  • ended
  • ratechange

Using these events, the player button code changes slightly:

video.addEventListener(‘play’, playEvent, false);
video.addEventListener(‘pause’, pausedEvent, false);
video.addEventListener(‘ended’, function () {
this.pause();
}, false);
function playEvent() {
play.innerHTML = ‘pause’;
}

function pausedEvent() {
play.innerHTML = ‘play’;
}

play.onclick = function () {
if (video.ended) { video.currentTime = 0; }
if (video.paused){
video.play();
}else{
video.pause();
}

};

No magic there.

If you want to see all the events and properties in action (and more importantly see if a browser supports them or not) check out the Media events demo page at the W3C web site.

One very good trick for syncing video and other effects is using the currentTime property. One example of that is the Spirit of Indiana demo where we sync a Google Maps animation with a video. The main gotcha is that the timeupdate event fires a lot and not necessarily every second. Therefore you need to throttle the changes to full seconds by using parseInt(). You can see this in action at the video demo:




window.addEventListener(‘load’,
function() {
var stage = document.getElementById(‘stage’);
var log = document.getElementById(‘logger’);
var seqlog = document.getElementById(‘loggersequences’);
var v = document.getElementsByTagName(‘video’)[0];
but = document.createElement(‘button’);
but.innerHTML = ‘Click to see Lindbergh’s flight’;
stage.appendChild(but);
but.addEventListener(‘click’,function(e) {
v.play();
e.preventDefault();
},false);
var now = 0;
v.addEventListener(‘timeupdate’,function(o){
log.innerHTML = v.currentTime;
var full = parseInt(v.currentTime);
if(full >= now) {
seqlog.innerHTML = now;
now = now + 1;
}

},false);
},
false);

Transforming

The real power of HTML5 and native video is when you can see how easy it is to transform the video element in the page. Here are some demos:

  • Transforming Video shows how simple it is to create a player that allows you to zoom inside a video and rotate it using CSS transformations
  • Paul Rouget’s round video demo shows how to create a play/pause button in SVG, greyscale a video and rotate it using CSS transformations
  • Paul’s Video Mashup demo shows how to apply SVG filters like Gaussian Blur, masking and how to apply CSS transforms to a running video

Simply check the source codes on these demos to see how they are done – the power of an open web.

Realtime changes

Adding filters and transforming the video container with CSS is one thing. Using Canvas and copying frames from the video into it you can do realtime changes to the video itself, which is really nice.

  • Remy Sharp’s round video demo shows how to create a rotating, bouncing circle that contains a video
  • Paul Rouget’s Green demo shows how to simulate a green screen effect with open web technologies. You copy each frame over to a canvas and read out the RGB value of each pixel. When it is green, just set its opacity to allow the picture behind the video to shine through.
  • The dynamic content injection by Paul shows how to analyse a video and find white parts to use as corner points for an embedded canvas. You can then inject other videos, canvas animations or images into the other video.
  • The Tracker demo shows how to detect human shapes and movement in a video
  • The edge detection demo shows how to detect edges in a live video using JavaScript

Awesome audio stuff

As you can see there is a lot of cool realtime things you can do with video. Audio is less sexy somehow as you can play and maybe play backwards (only works in Safari at the moment). Wouldn’t it be cool to read out the audio data or dynamically generate sounds though?

There is a Firefox4 API out there that allows you to do that. Read the introduction to the API or check the Audio API Wiki to get to know it.

One cool demo using this API is the HTML5 Guitar Tab generator and there is already a library out there called sfxr that allows you to create audio on the fly.

Recording Interviews on Skype and converting to HTML5 friendly formats the easy way

Wednesday, January 5th, 2011

Tomorrow I will start a series of interviews on the Mozilla hacks blog about “People of HTML5”. I thought it is a good idea to introduce some people and ask them a few questions and thus give a face to the awesome you meet on IRC or see on GitHub.

I found myself with a few issues to crack:

  • How do I record a Skype call with good quality?
  • How do I convert the video then to a HTML5 friendly format (as “People of HTML5” with Flash movies would be a bit ironic)

The fun thing I found out is that with a good connection and some web trickery this is dead easy. Check the following screencast for the 150 second version:

The first thing I needed was to record the skype call. I tried using ishowu but that didn’t quite do it.

So I thought I bite the bullet and spend some money. Call recorder for Skype was $20 for my mac and does the job swimmingly – it adds itself to Skype and gives you a record button. It stores the recordings as MOV with an AAC and H.264 encoding.

Once I recorded the videos I wanted to convert them for HTML5 embedding. The easiest thing I found was Miro Video Converter – simply drag your file onto it, select WebM/V8 and it converts and saves the file for you (kids, this is why you use Macs!).

All well and good, but not all HTML5 browsers get WebM yet, right? You also need an MP4 version and a OGG Theora one. Also, I needed a space to put the videos where they get the right MIME type and I don’t pay.

Enter Archive.org. If you release your videos and audio recordings as Creative Commons, you can store them there. They provide an uploader on the page or an FTP access (which strangely enough is only one upload at a time). The really cool thing about Archive.org is though that if you wait for a bit, they will have converted your video to MP4 and OGG for you!

Check the video page on Archive.org of the screencast – I only uploaded the WebM video, the other formats and the animated GIF gets created for me automatically.

The only thing is that it doesn’t give you any information that it is converting so all you can do is wait, go back to the page and see when the magic happened.

Once the Archive.org server pixies did their work, you can use a video tag to show your video, using all the sources from your video page:

Notice that as a fallback I point to a site with the Flash movie (either YouTube or in this case Screenr.com which I used to record this screencast).

And that’s it! A $20 app, and free web services and we have hosting and conversion. We can also add subtitling, if we just add Universal Subtitles. I love the web!

TTMMHTM:How web sites work, how many people see them, open data and lots of accessibility and “girly” stuff

Sunday, January 24th, 2010

I am right now sitting at Heathrow Terminal 5 in London on my way outbound to a two week stint in the Silicon Valley (Sunnyvale/Mountain View) to meet with the US team. And here are the things that made me happy this morning:

TTMMHTM: Mario Bento, YUI3, Accessibility Interview, wearable computing pulp fiction typography and live carbon of the UK

Tuesday, July 21st, 2009