During some interesting discussions on Twitter yesterday I found that there is now more than ever a confusion about JavaScript dependence in web applications and web sites. This is a never ending story but it seems to me to flare up ever time our browsing technology leaps forward.

I encountered this for the first time back in the days of DHTML. We pushed browsers to their limits with our lovely animated menus and 3D logos (something we of course learned not to do again, right?) and we were grumpy when people told us that there are environments out there where JavaScript isn’t available.

Who turns off JavaScript?

The first question we need to ask about this is what these environments are. There are a few options for that:

• Security systems like noscript or corporate proxies that filter out JavaScript
• Feature phones like old Blackberries (I remember switching to Opera Mini on mine to have at least a bearable surfing experience)
• Mobile environments where carriers proxy images and scripts and sometimes break them
• People on traffic-limited or very slow connections
• People who turn off JavaScript for their own reasons
• People sick of modal pop-ups and other aggressive advertising

As you can see some of them are done to our end users (proxying my companies or mobile provider), some are probably temporary (feature phones) and some are simply their own choice. So there is no way to say that only people who want to mess with our cool web stuff are affected.

Why do they turn off JavaScript?

As listed above, there are many reasons. When it comes to deliberately turning off JavaScript, I’d wager to guess that the main three are security concerns, advertising fatigue and slow connectivity.

Security is actually very understandable. Almost every attack on a client machine happens using JavaScript (in most cases in conjunction with plugin vulnerabilities). Java of course is the biggest security hole at the moment but there is a lot of evil you can do with JavaScript via a vulnerable web site and unprotected or outdated browser and OS.

Slow connectivity is a very interesting one. Quite ironic – if you think about it – as most of what we use JavaScript for is to speed up the experience of our end users. One of the first use cases for JS was client side validation of forms to avoid unnecessary server roundtrips.

Now when you are on a very flaky connection (say a free wireless or bad 3G connectivity or at any web development conference) and you try to use for example Google Reader or Gmail you’ll end up with half broken interfaces. If the flakiness gets caught during first load you actually get offered a “HTML only low version” that is very likely to work better.

The best of both worlds

This is totally fine – it tries to give an end user the best experience depending on environment and connectivity. And this is what progressive enhancement is about, really. And there is nothing evangelical about that – it is plain and pure pragmatism.

It seems just not a good plan under any circumstances to give people an interface that doesn’t work. So to avoid this, let’s generate the interface with the technologies that it is dependent on.

With techniques like event delegation this is incredibly simple. You add click handlers to the parent elements and write out your HTML using innerHTML or other, newer and faster techniques.

So why is this such a problem?

Frankly, I really don’t know. Maybe it is because I am old school and like my localhost. Maybe it is because I have been disappointed by browsers and environments over and over again and like to play it safe. I just really don’t get why someone would go for a JS-only solution when the JS is really only needed to provide the enhanced experience on top of something that can work without it.

The mythical edge case application

A big thing that people keep coming up with are the “applications that need JavaScript”. If we are really honest with ourselves, then these are very rare. If pushed, I could only think of something like photoshop in the browser, or any other editor (video, IDE in the browser, synth) that would be dependent on JavaScript. All the others can fall back to a solution that requires a reload and server-side component.

And let’s face it – in the times of Node.js the server side solution can be done in JavaScript, too. Dav Glass of Yahoo 2 years ago showed that if a widget library is written to be independent of its environment, you can re-use the same rich widget client and server side.

The real reasons for the “App that needs JavaScript” seems to be a different, non-technical ones.

The real reasons for “Apps that need JavaScript”

Much like there are reasons for not having JavaScript there are reasons for apps that need JavaScript and deliver broken experiences.

• You only know JS and think people should upgrade their browsers and stop being pussies. This is fine, but doesn’t make you the visionary you think you are as it is actually a limited view. We called that DHTML and it failed once – it can fail again
• You are building an app with a team without server side skills and want to get it out cheaply. This can work, but sounds to me like apps that “add accessibility later”, thus quadrupling the time and money needed to make that happen. Plan for that and all is good.
• You want to get the app out quickly and you know you’ll have to re-write it later. This is actually a pretty common thing, especially when you get highly successful or bought by someone else. Good luck to you, just don’t give people the impression that you are there to stay.
• Your app will run in a pure JS environment. Of course this means there is no need to make it work without JS. One example of this would be Air applications. Just make sure you bet on tech and environments that will stay on the radar of the company selling it.
• Your app really needs JS to work. If that is the case, just don’t offer it to people without it. Explain in a nice fashion the whys and hows (and avoid telling people they need to turn it on as they may not be able to and all you do is frustrate even more) and redirect with JS to your app.

In summary – sort of

All in all, the question of JavaScript dependence reaches much further than just the technical issues. It questions old best practices and has quite an impact on maintainability (I will write about this soon).

Let’s just say that our discussions about it would be much more fruitful if we started asking the “what do we need JS for” question rather than the “why do people have no JS”. There is no point in blaming people to hold back the web when our techniques are very adaptive to different needs.

There is also no point in showing people you can break their stuff by turning things in your browser on and off. That is not a representation of what happens when a normal visitor gets stuck in our apps.

Maybe all of this will be moot when node.js matures and becomes as ubiquitous as the LAMP stack is now. I’d like to see that.

I am lucky enough to have a vid.ly pro account to convert videos. Lucky because lately the free service started limiting the amount of times you can watch a video in a month (as they were hammered by a lot of traffic from Asia abusing the service). In case you still haven’t heard about vid.ly – it is a service that converts a video into a few dozen formats for HTML5 embedding and gives you a single URL to redirect devices to the correct format of the video.

Now, to make it easier for my colleagues to convert and embed videos in HTML5, I built a simple interface for converting and embedding a video on our blogs. For this I am using the API, but I wanted to avoid having to give my key out for colleagues to use.

The interface to convert videos is pretty easy:

<header><h1>Vid.ly conversion and embed</h1></header>
<section>
  <?php echo $message; ?>
 
  <p>Simply add the URL of the video to convert below and you get the embed code. 
An email will inform you about the successful conversion. 
Conversion could take up to an hour.</p>
 
  <form method="post">
    <div><label for="email">Email:</label><input type="text" id="email" name="email"></div>
    <div><label for="url">URL:</label><input type="text" id="url" name="url"></div>
    <div><input type="submit" name="send" value="make it so"></div>
  </form>
</section>

One of the cool features of the API is that it allows you to define an email that is not the one connected with the key to be the one that gets notified both of the conversion start, errors and success email. That made my job a lot easier. All I needed to do was assemble the correct XML and send it to the API. As the result is XML, too, I needed to check what came back and give feedback in the form:

<?php
$key = '{add your key here}';
$message = '';
if(isset($_POST['send'])){
 
  if($_POST['email'] !== '' && $_POST['url'] !== '') {
    $query =  '<?xml version="1.0"?>'.
              '<query><action>AddMedia</action><userid>481</userid>'.
              '<userkey>'.$key.'</userkey>'.
              '<notify>'.$_POST['email'].'</notify>'.
              '<Source><SourceFile>'.$_POST['url'].'</SourceFile>'.
              '<CDN>AWS</CDN></Source></query>';
    $url = 'http://m.vid.ly/api/';
    $ch = curl_init();
    curl_setopt($ch,CURLOPT_URL,$url);
    curl_setopt($ch,CURLOPT_POST,1);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 
    curl_setopt($ch,CURLOPT_POSTFIELDS,'xml='.urlencode($query));
    $result = curl_exec($ch);
    curl_close($ch);
 
    $xml = simplexml_load_string($result);
 
    if($xml->Success) {
      $vid = $xml->Success->MediaShortLink->ShortLink;
      $video = '<video controls width="100%" controls preload="none"'.
               ' poster="http://cf.cdn.vid.ly/'.$vid.'/poster.jpg">'.
               '<source src="http://cf.cdn.vid.ly/'.$vid.'/mp4.mp4" '.
               'type="video/mp4">'.
               '<source src="http://cf.cdn.vid.ly/'.$vid.'/webm.webm" '.
               'type="video/webm">'.
               '<source src="http://cf.cdn.vid.ly/'.$vid.'/ogv.ogv" '.
               'type="video/ogg">'.
               '<a target="_blank" href="http://vid.ly/'.$vid.'">'.
               '<img src="http://cf.cdn.vid.ly/'.$vid.'/poster.jpg" '.
               'width="500"></a>'.
               '</video>';
      $message = '<div class="success"><h1>Conversion started</h1>'.
                 '<p>The video conversion is under way. '.
                 'You should get an email telling you so and an email when '.
                 'the video URL is ready. The code to copy & paste into '.
                 'the blog is:</p>'.
                 '<textarea>'.htmlspecialchars($video).' </textarea>';
    } else {
        $message = '<div class="error"><h1>Error</h1>'.
                   '<p>Something went wrong in the conversion,'.
                   'please try again.</p></div>';
    }
 
  } else {
    $message = '<div class="error"><h1>Error</h1>'.
               '<p>Please provide a video URL and email</p></div>';
  }
}
?>

Pretty simple, isn’t it. Now my colleagues can add their email, give the form a URL where the video to convert is on the web and will get a copy and paste HTML for the video, for example:

<video controls preload="none" style="width:100%;height:300px;" 
poster="http://cf.cdn.vid.ly/1l5i5m/poster.jpg">
<source src="http://cf.cdn.vid.ly/1l5i5m/mp4.mp4" type="video/mp4">
<source src="http://cf.cdn.vid.ly/1l5i5m/webm.webm" type="video/webm">
<source src="http://cf.cdn.vid.ly/1l5i5m/ogv.ogv" type="video/ogg">
<a target='_blank' href='http://vid.ly/1l5i5m'>
<img   src='http://cf.cdn.vid.ly/1l5i5m/poster.jpg' width="500"></a>
</video>

Which results in:

Giving HTML5 video to the browsers who support it and a link to vid.ly for those who don’t :) The code is on GitHub as a Gist:

Note: the server is defunct at the moment but you can get the source code of the calendar on GitHub: code and zip

In Yesterday’s tutorial I went up to making the advent calendar use Ajax to load the different content, now for extra effect I made the entries move to where you clicked the calendar. There is not much more to it and you can see the whole thing in action here.

Step 4: Moving overlays

Now, first of all, instead of showing the entry above the calendar we now cover it, which means we should give the end users a way to get rid of them. this means adding a close link to the entry. As the entry is assembled in the API, this needs to happen there:

<?php
  $data = '';
  $day = 0;
  $day = +$_GET['day'];
  if (+date('m') < 12 || $day > +date('d')) {
    $day = 0;
  }
  if ($day > 24) { $day = 24; }
 
  if ($day) {
    $data .= '<h1><a href="#">Title '.$day.'</a></h1>'.
             '<p>Description</p>'.
             '<p>See it in action <a href="#">here</a></p>';
  }
 
  $data .= '<p><a href="index.php" id="close">close</a></p>';
 
  if (isset($_GET['ajax'])) { echo $data; }
?>

The link also points back to the page, so if there is no JavaScript available, the page just reloads.

Now, in order to hide and show the entry, I positioned it absolutely and gave it a higher z-index:

article {
  z-index: 10;
  position: absolute;
  top: -400px;
  width: 700px;
  background: #d3caff;
}

This, of course needs to be un-done when there is a day selected. In the case of a no JavaScript version, this needs to be done in the PHP itself. To override the original, I defined a class:

article.show {
  top: 200px;
}

And in the PHP I write out this class when a valid day was sent in:

<article <?php if($day) {echo ' class="show"';} ?>>
   <?php if($day) {echo $data;} ?>
</article>

In order to move the entry to the right height, you need to capture the mouse position when the user clicks one of the links. For this, I modified the event delegation handler to read the y position of the mouse and send it through to the ajax loader:

list.addEventListener('click', function(ev) {
  var t = ev.target;
  if (t.tagName ==='A') {
    var y = ev.clientY + document.body.scrollTop +
            document.documentElement.scrollTop;
    load(+t.innerHTML, y);
  }
  ev.preventDefault();
}, false);

The loading function then does not only change the content of the article element but it also adds the class of “show” to it and modifies its top position to the y parameter it received.

function load(day, y) {
  var httpstats = /200|304/;
  var request = new XMLHttpRequest();
  request.onreadystatechange = function() {
    if (request.readyState == 4) {
      if (request.status && httpstats.test(request.status)) {
        output.innerHTML = request.responseText;
        output.className = 'show';
        output.style.top = y + 'px';
      } else {
        document.location = 'index.php?day=' + day;
      }
    }
  };
  request.open('get', 'simpleapi-closing.php?ajax=1&day='+day, true);
  request.setRequestHeader('If-Modified-Since',
                           'Wed, 05 Apr 2006 00:00:00 GMT');
  request.send(null);  
}

To make this smooth, all we need to do is add a CSS transition to the element – isn’t it wonderful to work with browsers that don’t suck?

article {
  z-index: 10;
  position: absolute;
  top: -400px;
  width: 700px;
  background: #d3caff;
  -moz-transition: top 1s;
  -webkit-transition: top 1s;
  -ms-transition: top 1s;
  -o-transition: top 1s;
  transition: top 1s;
}

To hide the element again we need to re-set the position and remove the “show” class. This happens in two cases: when the user clicks the close link (we gave it an ID of “close”) and when the user hits the ESC key on the keyboard. For the former we use event delegation on the article element (as its content keeps getting re-written), for the latter a keyboard handler on the document:

output.addEventListener('click',function(ev){
  var t = ev.target;
  if(t.tagName === 'A' && t.id === 'close') {
    output.style.top = '-400px';
    output.className = '';
    ev.preventDefault();
  }
},false);
 
document.addEventListener( 'keydown', function(key) {
   if ( key.keyCode === 27 ) {
     if( output.className === 'show' ) {
       output.style.top = '-400px';
       output.className = '';
     }
   }
 }, false );

That’s it! Smooth entries moving up and down when you click different links.

Step 5: Adding the real content

The final change to make this work was to find a way for people to send me links to show. For this, all I needed to do was create a Google spreadsheet and send it around.

You can publish your spreadsheets on the web using File > Publish to the web > Start Publishing. I chose CSV as the format and then changed the API accordingly.

I load the document in using cURL and then split it into an associative array using the csv_to_array() method by M. Bunge found on the PHP docs for str_getcsv().

I then check if there is an entry for the day and if there isn’t I just give back a “not yet” string. Otherwise I get the data from the associative array and assemble the real string.

<?php
$now = '';
$day = 0;
$data = '';
 
if (isset($_GET['day'])) {
  $day = +($_GET['day']);
  if ($day > 24) { $day = 24; }
  if (+date('m') < 12 || $day > +date('d')) {
    $day = 0;
  }
 
  $url = 'https://docs.google.com/spreadsheet/pub?'.
         'key=0AhphLklK1Ve4dEp5X2tBNHFPM0hQSHpZQnBjYl9NLUE&output=csv';
  $ch = curl_init(); 
  curl_setopt($ch, CURLOPT_URL, $url); 
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 
  curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
  curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
  $output = curl_exec($ch); 
  curl_close($ch);
 
  $csvdata = csv_to_array($output);
  $now = $csvdata[($day-1)];
  if ($now) {
    $data .= '<h1><a href="'.$now[0].'">'.$now[1].'</a></h1>'.
             '<p>'.$now[2].'</p>';
    if ($now[3] !== '') {
      $data .= '<p>You can also <a href="'.$now[3].
               '">see it in action here</a>.</p>';
    }
  } else {
    $data .= '<h1><a href="http://developer.mozilla.com">Not yet!</a></h1>'.
             '<p>You have to wait, like all the others.</p>';
  }
}
 
$data .= '<a id="close" href="index.php">x</a>';
 
if (isset($_GET['ajax'])) { echo $data; }
 
function csv_to_array($input, $delimiter=',') {
  $header = null;
  $data = array();
  $csvData = str_getcsv($input, "\n");
  foreach($csvData as $csvLine) {
    if (is_null($header)) {
      $header = explode($delimiter, $csvLine); 
    } else {
      $items = explode($delimiter, $csvLine);
      for($n = 0, $m = count($header); $n < $m; $n++){
        $prepareData[$n] = $items[$n];
      }
      $data[] = $prepareData;
    }
  }
  return $data;
}
?>

And that is that – a simple advent calendar in PHP progressively enhanced with JavaScript and CSS.

See the final functional demo in action at:
http://isithackday.com/calendar-tutorial/realcontent.php

Final tweaks

In the final version I will of course add some caching instead of hitting Google Docs live and I added a lot more CSS to make it look the way I wanted to. But this should get you going.

The Final Build folder has a cleaned up version, with external CSS and JavaScript and a nice (function(){}()); around the JS to stop those global variables.

You can also get the whole project on GitHub

Note: the server is defunct at the moment but you can get the source code of the calendar on GitHub: code and zip

Yesterday morning I got inspired to have a go at building an advent calendar for the Mozilla Developer Network where we are going to have a daily link for you. Here’s my work log on how I did it. Maybe it is inspirational for you, maybe not. Let’s just have a go.

Step 1: Distributing the 24 “doors” on a massive logo

This was simple. I put the logo as a background into an element with the right dimensions and positioned that one relative. I then wrote a wee script to position 24 list items with the correct links randomly on the “canvas”:

<?php 
echo '<ol>';
$width = 700;
$height = 800;
for ($i=0;$i<24;$i++) {
  $x = rand(10,$width-10);
  $y = rand(10,$width-10);
  echo '<li style="left:'.$x.'px;top:'.$y.'px">'.
          '<a href="index.php?day='.($i+1).'">'.($i+1).'</a>'.
       '</li>';
}
echo '</ol>';
?>

I gave the list items a size and positioned them absolutely and then looked at the results. I reloaded until not many overlapped, moved those by hand, grabbed the innerHTML of the list in Firebug and I had my awesome calendar layout. :)

See it here: http://isithackday.com/calendar-tutorial/random.php

Step 2: Build the API

The next step was to build the API to display the when you click on one of the links. This I build as a dummy, too. It is pretty simple, really.

<?php
  $data = '';
  $day = 0;
  $day = +$_GET['day'];
  if ($day > 24) { $day = 24; }
 
  if ($day) {
    $data .= '<h1><a href="#">Title '.$day.'</a></h1>'.
             '<p>Description</p>'.
             '<p>See it in action <a href="#">here</a></p>';
  }
?>

I cast the day URL parameter to an integer and make sure that it can’t be more than 24. If there was a valid number sent in, I assemble a string of HTML with the output data to show for the day. As $day is preset as 0 and will be 0 when the data sent in is not a valid number, $data will be empty.

In the page I then create an element and conditionally write the content out:

All I need to make this work as an advent calendar later on is add a date validation in there.

<?php 
if (+date('m') < 12 || $day > +date('d')) {
  $day = 0;
}
?>

For now, I commented that out, though.

See it in action here: http://isithackday.com/calendar-tutorial/dummyapi.php

Step 3: Ajaxify

This all works now, but to make it slicker, let’s load the content via Ajax when possible. The first step there was to define the article element as the output container and use event delegation to find out which list element was clicked:

var list = document.querySelector('ol'),
    output = document.querySelector('article');
 
list.addEventListener('click', function(ev) {
  var t = ev.target;
  if (t.tagName ==='A') {
    load(+t.innerHTML);
  }
  ev.preventDefault();
}, false);
 
function load(day) {
  alert(day);
}

I then moved the dummy api out to an own file (simpleapi) and added a check for another parameter:

<?php
  $data = '';
  $day = 0;
  $day = +$_GET['day'];
  if ($day > +date('d')) {
    $day = 0;
  }
  if ($day > 24) { $day = 24; }
 
  if ($day) {
    $data .= '<h1><a href="#">Title '.$day.'</a></h1>'.
             '<p>Description</p>'.
             '<p>See it in action <a href="#">here</a></p>';
  }
  if (isset($_GET['ajax'])) { echo $data; }
?>

That way I only had to do a classic Ajax call adding a new parameter to the url to trigger the echo in the PHP:

function load(day) {
  var httpstats = /200|304/;
  var request = new XMLHttpRequest();
  request.onreadystatechange = function() {
    if (request.readyState == 4) {
      if (request.status && httpstats.test(request.status)) {
        output.innerHTML = request.responseText;
      } else {
        document.location = 'index.php?day=' + day;
      }
    }
  };
  request.open('get', 'simpleapi.php?ajax=1&day='+day, true);
  request.setRequestHeader('If-Modified-Since',
                           'Wed, 05 Apr 2006 00:00:00 GMT');
  request.send(null);  
}

If you wonder about the date in the request header, yes I did indeed rip my own code from my 2006 Ajax book. :)

http://isithackday.com/calendar-tutorial/ajax.php

That’s all for today

That is the main functionality, really. In part 2 I will then show you how to pull the real data off the web and make this look much sexier, as shown here:

You may not know it, but I am a huge Calvin and Hobbes fan. Not as big as this guy, but big:

The reason to me is that Calvin and Hobbes is full of features that I consider the ingredients of great quality work, and something we should strive to emulate:

• Simplicity – none of the comics are very complex, there is no depth in the paintings, no 3D shadings with metal effects. A few black and white lines arranged in the right way make the effect
• Staying in the boundaries – the comic never deviated from the normal format of a few panels with the exception of the Sunday specials which were in colour and full-page or some Christmas specials. This is OK and I never expected it to do anything like that. Staying within standardised boundaries and being beautiful at the same time is harder than breaking the mould all the time.
• Beautiful details – what makes C&H lovable is the little details – Calvin’s shoes that look like bread rolls, the jumping TV set, Hobbes’ fuzzy belly. All of which are a few lines that are not like you expect them but make you see the effort that went in
• Delivery vs. money focus – there was never any official Calvin and Hobbes merchandise other than the books. Bill Watterson wanted to deliver a great product on paper and that is it. No animated series, no real actor movie, just books. I sincerely hope it stays that way.
• Syndication brings fame – Calvin and Hobbes is printed in newspapers, it uses an existing channel to reach as many people as possible. This is a tough place to get to, but you reach everybody and not just a chosen few.
• Good messages – beyond the funny pictures, Calvin and Hobbes covers a lot of important topics. From environmental issues to philosophy, you find good messages without any religious overtones found for example in Peanuts. Through the mouth of a very excited kid bad things come across in a very powerful way when he is bummed out about something.
• Rules can be bent or broken – Calvinball has no rules, but seems to be a lot of fun. Science is fascinating but can be bent if you want to have a transmogrifier to turn you into a pterodactyl if you want to.
• Relationships keep things interesting – Calvin and Susie Derkins or Calvin and his dad remind me a lot of times about the tensions and relationships I have with people I work with in a less destructive way that Dilbert does. Yes, there is animosity, yes there are misunderstandings, but all in all we care about each other.
• The source matter is not important if your imagination is good – Calvin building his snowmen of horror or assembling junk dug out from the ground into a dinosaur skeleton are great examples of that.
• Summon your alter egos – when Calvin escapes his dilemma of having to do homework by becoming “stupendous man” or “spaceman spiff” I think of how I look at my work from the angle of an end user or person with a disability. Going out of your normal frame of mind can be immensely powerful.
• Be passionate and up for it – Calvin is always 110% (it might be the sugar in the breakfast cereal) and so should we. Passion is the most important part that drives creativity. Passion can lead to controversy and misunderstandings. This is good, and it is sad to see that we live in a society were people with strong opinions are labeled destructive and negative. They may not be, they are just up for it and want to change things and experience new things all the time. If we stop that we deliver mediocrity
• Be warm and don’t fear to show your emotions – the most wonderful moments are in Calvin and Hobbes when emotion comes through, beyond all logic. When Calvin forgets Hobbes in the garden and his parents seek him in the dark (in his stuffed toy form) and Calvin’s mom catches herself shouting his name, we know that the characters care for each other a lot. When Calvin cuddles up with his parents every time something bugs him you want to be part of it. Emotions are good, we should not hide behind a mask of awesome or professionalism

I might be rambling here, but I think it is important that we give our best when we create. It is a magical world out there and we all have some little magic to bring into it. When Calvin gets moody about autumn arriving and the summer being over and Hobbes explains that the leaves with their different colours look like fireworks of mother nature, I knew that even bleak moments can become magical when we allow them to.

And if you can not deliver this every time you do something, that is not a problem either. Some days are bad, so bad that even your lucky rocketship underpants don’t help. All we can do is deal with them and move on.