Christian Heilmann

Posts Tagged ‘ajax’

Maintainable JavaScript at the ThinkVitamin JavaScript online conference

Monday, September 13th, 2010

Today I spoke at the ThinkVitamin JavaScript online conference alongside Simon Willison, Stuart Landgridge and Drew McLellan.

Topic and slides

My topic was “Maintainable JavaScript” and I managed to pack 150 slides into the 50 minutes of my talk. The Slides are available on Slideshare.

Maintainable Javascript carsonified

View more presentations from Christian Heilmann.

What I covered

I went through a few of the tricks you should apply when you want to write maintainable JavaScript:

  • Using, not abusing libraries
  • Separation of concerns
  • Building for extensibility
  • Documenting your work
  • Planning for performance
  • Avoiding double maintenance
  • Live code vs. development code

A few of the topics got repeated by the other speakers but frankly I am always fascinated just how many people do not know about the X-requested-with header JavaScript libraries add to Ajax requests and how you can use it to render different content for Ajax requests and for include() requests. This allows you to maintain your whole content and HTML on the server and have all the Ajax goodness rendering out a simple HTML string.

The code example for the validation is available on GitHub.

Another thing that is missing a lot in people’s approach to JS is a proper build process. That’s why a lot of people try to optimise for performance during development and leave hard to maintain, specialist code behind.

In essence, I showed that by building logical backend code you can save yourself a lot of JavaScript and you build things that work instead of things you hope will work.

The conference experience

Speaking at pure online conferences is weird. For starters we had to change the order as Simon Willison is on his honeymoon world-tour and currently in Marrakesh, which meant his connection was flaky. The audio during his talk was not the best and he didn’t even bother with video.

It is a strange experience to sit there with a headset and talking to your desktop as you cannot see the audience. Talks get much better when you can interact with the audience, ask questions, see them falling asleep or seething with rage – that kind of thing. In an online conference this is missing and you cannot read the chat whilst you present as this is too distracting. Drew was suffering the most from this lack of interaction and I had my audio drop out once which meant I had to repeat some slides.

All in all I find these things work better when there is a discussion going on – like a moderator interviewing a speaker over Skype and people asking questions. The other option of course would be to record the talks beforehand and play them instead of hoping all works out fine live – it never does. Recording the talks and then having a pure Q&A session with the speaker on skype afterwards seems to me a better way to work with these issues.

The benefits of these conferences are quite cool. People do not have to travel, you can pay one ticket and let a whole office watch, speakers don’t need to wear trousers and the liver of the speakers doesn’t need to suffer as much as it does at normal conferences. I am not quite convinced about the “green” argument of these events as they do use up energy, too but hey, at least it helps people thinking “green”.

Online conferences are a good idea as an add-on but I don’t think they’ll replace conferences as the networking and “meeting in the flesh” aspect is lost and that makes them feel a bit clinical and cold. That said, Carsonified are doing a good job with these right now and are lovely people when you interact with them as a speaker.

Loading external content with Ajax using jQuery and YQL

Sunday, January 10th, 2010

Let’s solve the problem of loading external content (on other domains) with Ajax in jQuery. All the code you see here is available on GitHub and can be seen on this demo page so no need to copy and paste!

OK, Ajax with jQuery is very easy to do – like most solutions it is a few lines:

$(document).ready(function(){
  $('.ajaxtrigger').click(function(){
    $('#target').load('ajaxcontent.html');
  });
});

Check out this simple and obtrusive Ajax demo to see what it does.

This will turn all elements with the class of ajaxtrigger into triggers to load “ajaxcontent.html” and display its contents in the element with the ID target.

This is terrible, as it most of the time means that people will use pointless links like “click me” with # as the href, but this is not the problem for today. I am working on a larger article with all the goodies about Ajax usability and accessibility.

However, to make this more re-usable we could do the following:

$(document).ready(function(){
  $('.ajaxtrigger').click(function(){
    $('#target').load($(this).attr('href'));
    return false;
  });
});

You can then use load some content to load the content and you make the whole thing re-usable.

Check out this more reusable Ajax demo to see what it does.

The issue I wanted to find a nice solution for is the one that happens when you click on the second link in the demo: loading external files fails as Ajax doesn’t allow for cross-domain loading of content. This means that see my portfolio will fail to load the Ajax content and fail silently at that. You can click the link until you are blue in the face but nothing happens. A dirty hack to avoid this is just allowing the browser to load the document if somebody really tries to load an external link.

Check out this allowing external links to be followed to see what it does.

$(document).ready(function(){
  $('.ajaxtrigger').click(function(){
    var url = $(this).attr('href');
    if(url.match('^http')){
      return true;
    } else {
      $('#target').load(url);
      return false;
    }
  });
});

Proxying with PHP

If you look around the web you will find the solution in most of the cases to be PHP proxy scripts (or any other language). Something using cURL could be for example proxy.php:

<?php
$url = $_GET['url'];
$ch = curl_init(); 
curl_setopt($ch, CURLOPT_URL, $url); 
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 
$output = curl_exec($ch); 
curl_close($ch);
echo $content;
?>

People then could use this with a slightly changed script (using a proxy):

$(document).ready(function(){
  $('.ajaxtrigger').click(function(){
    var url = $(this).attr('href');
    if(url.match('^http')){
      url = 'proxy.php?url=' + url;
    }
    $('#target').load(url);
    return false;
  });
});

It is also a spectacularly stupid idea to have a proxy script like that. The reason is that without filtering people can use this to load any document of your server and display it in the page (simply use firebug to rename the link to show anything on your server), they can use it to inject a mass-mailer script into your document or simply use this to redirect to any other web resource and make it look like your server was the one that sent it. It is spammer’s heaven.

Use a white-listing and filtering proxy!

So if you want to use a proxy, make sure to white-list the allowed URIs. Furthermore it is a good plan to get rid of everything but the body of the other HTML document. Another good idea is to filter out scripts. This prevents display glitches and scripts you don’t want executed on your site to get executed.

Something like this:

<?php
$url = $_GET['url'];
$allowedurls = array(
  'http://developer.yahoo.com',
  'http://icant.co.uk'
);
if(in_array($url,$allowedurls)){
  $ch = curl_init(); 
  curl_setopt($ch, CURLOPT_URL, $url); 
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 
  $output = curl_exec($ch); 
  curl_close($ch);
  $content = preg_replace('/.*<body[^>]*>/msi','',$output);
  $content = preg_replace('/<\/body>.*/msi','',$content);
  $content = preg_replace('/<?\/body[^>]*>/msi','',$content);
  $content = preg_replace('/[\r|\n]+/msi','',$content);
  $content = preg_replace('/<--[\S\s]*?-->/msi','',$content);
  $content = preg_replace('/<noscript[^>]*>[\S\s]*?'.
                          '<\/noscript>/msi',
                          '',$content);
  $content = preg_replace('/<script[^>]*>[\S\s]*?<\/script>/msi',
                          '',$content);
  $content = preg_replace('/<script.*\/>/msi','',$content);
  echo $content;
} else {
  echo 'Error: URL not allowed to load here.';
}
?>

Pure JavaScript solution using YQL

But what if you have no server access or you want to stay in JavaScript? Not to worry – it can be done. YQL allows you to load any HTML document and get it back in JSON. As jQuery has a nice interface to load JSON, this can be used together to achieve what we want to.

Getting HTML from YQL is as easy as using:

select * from html where url="http://icant.co.uk"

YQL does a few things extra for us:

  • It loads the HTML document and sanitizes it
  • It runs the HTML document through HTML Tidy to remove things .NETnasty frameworks considered markup.
  • It caches the HTML for a while
  • It only returns the body content of the HTML - so no styling (other than inline styles) will get through.

As output formats you can choose XML or JSON. If you define a callback parameter for JSON you get JSON-P with all the HTML as a JavaScript Object – not fun to re-assemble:

foo({
  "query":{
  "count":"1",
  "created":"2010-01-10T07:51:43Z",
  "lang":"en-US",
  "updated":"2010-01-10T07:51:43Z",
  "uri":"http://query.yahoo[...whatever...]k%22",
  "results":{
    "body":{
      "div":{
        "id":"doc2",
        "div":[{"id":"hd",
          "h1":"icant.co.uk - everything Christian Heilmann"
        },
        {"id":"bd",
        "div":[
        {"div":[{"h2":"About this and me","
        [... and so on...]
}}}}}}}});

When you define a callback with the XML output you get a function call with the HTML data as string in an Array – much easier:

foo({
  "query":{
  "count":"1",
  "created":"2010-01-10T07:47:40Z",
  "lang":"en-US",
  "updated":"2010-01-10T07:47:40Z",
  "uri":"http://query.y[...who cares...]%22"},
  "results":[
    "<body>\n    <div id=\"doc2\">\n<div id=\"hd\">\n 
     <h1>icant.co.uk - \n
     everything Christian Heilmann<\/h1>\n 
      ... and so on ..."
  ]
});

Using jQuery’s getJSON() method and accessing the YQL endpoint this is easy to implement:

$.getJSON("http://query.yahooapis.com/v1/public/yql?"+
          "q=select%20*%20from%20html%20where%20url%3D%22"+
          encodeURIComponent(url)+
          "%22&format=xml'&callback=?",
  function(data){
    if(data.results[0]){
      var data = filterData(data.results[0]);
      container.html(data);
    } else {
      var errormsg = '<p>Error: can't load the page.</p>';
      container.html(errormsg);
    }
  }
);

Putting it all together you have a cross-domain Ajax solution with jQuery and YQL:

$(document).ready(function(){
  var container = $('#target');
  $('.ajaxtrigger').click(function(){
    doAjax($(this).attr('href'));
    return false;
  });
  function doAjax(url){
    // if it is an external URI
    if(url.match('^http')){
      // call YQL
      $.getJSON("http://query.yahooapis.com/v1/public/yql?"+
                "q=select%20*%20from%20html%20where%20url%3D%22"+
                encodeURIComponent(url)+
                "%22&format=xml'&callback=?",
        // this function gets the data from the successful 
        // JSON-P call
        function(data){
          // if there is data, filter it and render it out
          if(data.results[0]){
            var data = filterData(data.results[0]);
            container.html(data);
          // otherwise tell the world that something went wrong
          } else {
            var errormsg = '<p>Error: can't load the page.</p>';
            container.html(errormsg);
          }
        }
      );
    // if it is not an external URI, use Ajax load()
    } else {
      $('#target').load(url);
    }
  }
  // filter out some nasties
  function filterData(data){
    data = data.replace(/<?\/body[^>]*>/g,'');
    data = data.replace(/[\r|\n]+/g,'');
    data = data.replace(/<--[\S\s]*?-->/g,'');
    data = data.replace(/<noscript[^>]*>[\S\s]*?<\/noscript>/g,'');
    data = data.replace(/<script[^>]*>[\S\s]*?<\/script>/g,'');
    data = data.replace(/<script.*\/>/,'');
    return data;
  }
});

This is rough and ready of course. A real Ajax solution should also consider timeout and not found scenarios. Check out the full version with loading indicators, error handling and yellow fade for inspiration.

Introduction to Yahoo Open Applications

Sunday, October 11th, 2009

Last week I was in Paris for a Yahoo Developer Network evening and Paris Web and one of the talks I gave was an introduction to Yahoo Open Applications. These are applications that you can embed in the Yahoo homepage or My Yahoo and thus allow you to reach millions of users – or extend the Yahoo homepage with your own personal app. Here are the slides and the audio recording of the talk delivered by Sophie Davies-Patrick (aka “my boss”) and me at La Cantine in Paris:

Yahoo Open Applications use the Yahoo Application Platform – YAP. In essence, you write a small app using CSS, JavaScript and HTML and it will get embedded into the Yahoo Homepage.

TweetTrans – translate Twitter updates

The example I showed was a tool that adds little translation links to a Twitter stream:

The code of TweetTrans is available on GitHub and I’ve built it initially as a bog-standard Ajax web application.

The main step afterwards was to convert the app over. This is less hard than it seems upfront but the fact that YAP uses Caja to make the whole application more secure means you need to restrict yourself. Things you cannot use are:

Caja and HTML

  • Custom attributes
  • Custom tags
  • Unclosed tags
  • EMBED
  • IFRAME
  • javascript:void(0)
  • Radio buttons in IE
  • Relative URLs

Caja and CSS

  • star hacks
  • _ hacks
  • IE conditionals
  • Insert-after clear fix
  • expression()
  • @import
  • Background images in IE

Caja and JavaScript

  • eval()
  • new Function()
  • Strings as event handlers (node.onclick = ‘...’;)
  • Names ending with double / triple underscores
  • with function (with (obj) { ... })
  • Implicit global variables (specify var variable)
  • Calling a method as a function
  • document.write
  • window.event
  • OpenSocial gadgets.io.makeRequest return JS

Get the SDK

The easiest way to build Yahoo Open Apps is to use YML which is a markup language that gives you access to the Yahoo social connections and creates Ajax functionality for you without having to write any JavaScript.

If you want to use the full Yahoo social stack the best place to start is to download the SDK which makes the oAuth authentication very easy for you.

Distribution

Once you have your app done you can easily make people install it by sending them a link. This link contains the application ID. In the case of TweetTrans this is:

http://yahoo.com/add?yapid=zKMBH94k

More info

To get more information, check out the YOS section on the Yahoo Developer Network and the forums on YAP.

Again with Ajax Accessibility – my talk at AbilityNet in London

Thursday, July 17th, 2008

Today I went again to help out AbilityNet with one of their workshops talking to a small group of developers and project managers about the problems with Ajax and Accessibility and general Web2.0 concerns. Instead of giving a lot of technical details I tried to point the audience to good resources and get them to find out more for themselves. I hope I succeeded.

[slideshare id=514640&doc=ajaxandaccessibilityredux-1216171321459432-9&w=425]

For the first time I also successfully recorded the talk in mp3 format using the mac powerbook. GarageBand is great for recording, but the editing features suck, good old Audacity came to the rescue. Bear in mind I had 4 hours sleep and a day of presentations behind me when I did this one, and it is more fun to see me explain some of the pauses with gestures :)

Oh look, using Ajax in a stupid way is not a good idea?

Tuesday, April 29th, 2008

It is quite fascinating to me that the newest article on dev.opera.com entitled ‘Stop using Ajax!’ is such a big thing right now. Tweets, shared bookmarks and Google Reader items are pouring in and people seem to consider it an amazingly daring article.

Here’s the truth: James is right. He also was right when he more or less gave the same information as a talk at Highland Fling last year following my presentation on progressive enhancement and JavaScript.

However, there is nothing shocking or daring or new about this. All he says is:

  • Don’t use any technology for the sake of using it
  • Consider the users you want to reach before using a technology that may not be appropriate
  • Make sure your solution is usable and accessible
  • Build your solution on stuff that works, then enhance it.

This is what I consider to be a normal practice when developing any software or web solution.

However, the real question is now why we are at this state – how come that we see this information as daring, shocking or controversial, and how come a lot of comments are still “I don’t care about accessibility because it is not needed for my users”? How come the assumptions and plain accessibility lies are prevailing while the good stuff remains unheard of?

Well, the truth is that we have been preaching far too long to the choir. I’ve been in the web accessibility and standards preaching community for a long time and whenever I asked what about enterprise development and CMS I was told that it is not worth fighting that fight as “We will never reach them”. Well, this is where the money and a lot of jobs are and it is a fact that both accessibility and standards activists in a lot of instances don’t even know the issues that keep the stakeholders in these areas busy. My Digital Web Article ‘10 reasons why clients don’t care about accessibility’ and the follow-up Seven Accessibility Mistakes Part One and Part 2 listed these issues and the wrong ways of how we try to tackle them 3 years ago. My talk at the AbilityNet conference last week Fencing-in the habitat also mentioned this attitude and problems.

Here’s where I am now: I am bored and tired of people fighting the good fight by blaming each other’s mistakes or pointing out problems on systems that are within reach. When people ask for accessibility or Ajax usability advice you’ll get a lot of bashing and “go validate then come back” answers but not much information that can be used immediately or even questions that ask what lead to the state of the product. You’d be surprised what you can find out by asking this simple question.

We have to understand that large systems, frameworks and companies do still run the show, even when we think that bloggers, books on webdesign and mashups push the envelope. They do, but so far they are a minor discomfort for companies that sell Ajax and other out-of-the-box solutions that are inaccessible and to larger parts unusable for humans. When was the last time you used a clever expense or time tracking system in companies that are not a startup or a small web agency? When I was at the AjaxWorld conference in NYC earlier this year I heard a lot about security, ease of deployment and scalability but only a little bit about accessibility (the Dojo talk and the YUI talk, actually). People are a lot more concerned about the cost of software and the speed of release than about the quality or maintainability. It is cheaper to buy a new system every few years than to build one that is properly tested and works for all users. Does your company still have systems or third party solutions that only work on IE/Windows? I am sure there is at least one, ask the HR or finance department.

It doesn’t help to coin another term and call an accessible and usable Ajax solution Hijax, either. As much as I like the idea of it I have to agree with James’ comment – we don’t need another word, we need a reason for people to not just use things out of the box without thinking about them or – even better – offer help to the companies that build the solutions on assumptions in the first place. When I ranted about a system by a large corporation some weeks ago on twitter their marketing manager for EMEA starting following me and I am starting some talks with them.

I have heard numerous times that my ideas about progressive enhancement and accessibility are just a “passing fad” and “that in the real software market you don’t have time for that”. Challenging this attitude is what makes a difference – by proving that by using the technologies we are given in a predictable and secure way does save you time and money. However, there are not many case studies on that…

I cannot change the world when I don’t know what obstacles people have to remove to do the right thing. Deep down every developer wants to do things right, in a clean and maintainable fashion and be proud of what they’ve done. Bad products happen because of rushed projects, bad management and developers getting so frustrated that they are OK with releasing sub-par just to get the money or finally get allocated to a different project.

This is the battle we need to fight – where do these problems come from? Not what technology to avoid. You can use any technology in a good way, you just need to be able to sell it past the hype and the assumption that software is developed as fast as it takes to write a cool press release about it.