After my talk at FOWA in Cambridge yesterday I showed off that http://icant.co.uk is fully driven by YUI and YQL and maintained elsewhere. I’ve recorded a screencast about this earlier which is also available for download as a M4V but hadn’t released the code yet.

So here goes – this is the PHP source of icant.co.uk (without the HTML which is more or less done with YUI grids builder

// get all the feeds to grab the data
$feeds = array(
'http://feeds2.feedburner.com/wait-till-i/gwZf',
'http://feeds.delicious.com/v2/rss/codepo8/myvideos?count=15',
'http://feeds.delicious.com/v2/rss/codepo8/sandbox',
'http://feeds.delicious.com/v2/rss/codepo8/icantarticles',
'http://www.slideshare.net/rss/user/cheilmann'
);

// assemble the YQL statement
$yql = 'select meta.views,content.thumbnail,content.description,title,'.
'link,description from rss where url in ';
$yql .= "('" . join($feeds,"','") . "')";

// assemble the request url
$root = 'http://query.yahooapis.com/v1/public/yql?q=';
$url = $root . urlencode($yql) . '&format=json';

// get the feeds and populate the data to echo out in the HTML
$feeds = renderFeeds($url);
$blog = $feeds['blog'];
$videos = $feeds['videos'];
$articles = $feeds['articles'];
$presentations = $feeds['slides'];

// this function loads all the feeds and turns them into HTML
function renderFeeds($url){

// grab the content from YQL via cURL
$c = getStuff($url);

// as the content comes back as JSON, turn it into PHP objects
$x = json_decode($c);

// reset counter for videos and presentations
$count = 0;
$vidcount = 0;

// start new array to return
$out = array();

// loop over YQL results, if they exist.
if($x->query->results->item){
foreach($x->query->results->item as $i){

// if the link comes from the blog, add to the blog HTML
if(strstr($i->link,'wait-till-i')){
$out['blog'] .= '
' . $i->title .
'
';
}

// if the description contains an embed code, add to videos
// (I use delicious for video bookmarks and add the embed code
// as a description, check here:
//         http://feeds.delicious.com/v2/rss/codepo8/myvideos)
if(strstr($i->description,'link . '">' . $i->title .
'
' . html_entity_decode($i->description) .
'
';
$vidcount++;
}
// for interviews and articles, add to the articles section
if(strstr($i->title,'Interview') ||
strstr($i->title,'Article:')){
$out['articles'].= '
' .$i->title .
'
'.html_entity_decode($i->description).
'
';
}

// for slideshare, add to slides element.
if(strstr($i->link,'slideshare') && $count < 4){
$out['slides'] .= '
' . $i->title .
'
' . $i->content->description->content .
'
';
$count++;
}
}
}
// return back the array :)
return $out;
}

// Grab the conference agenda from my blog
$yql = 'select * from html where url='.
'"http://wait-till-i.com/talks-and-conference-participation/"'.
' and xpath="//ul" limit 1';
$travels = renderHTML($root.urlencode($yql).'&format=xml&diagnostics=false');
// btw, diagnostics=false means YQL doesn't send a diagnostics part

// grab the books from my blog
$yql = 'select * from html where url='.
'"http://wait-till-i.com/books/"'.
' and xpath="//div[@class='entry']"';
$books = renderHTML($root.urlencode($yql).'&format=xml&diagnostics=false');

// this is a quick and dirty solution for the HTML output
function renderHTML($url){
// pull the information from YQL
$c = getStuff($url);
// check that something came back
if(strstr($c,'<')){
// remove all the XML parts
$c = preg_replace("/.*|.*/",'',$c);
$c = preg_replace("//",'',$c);
// remove all comments
$c = preg_replace("//",'',$c);
}
// send it back
return $c;
}

// a simple cURL function to get information
function getstuff($url){
$curl_handle = curl_init();
curl_setopt($curl_handle, CURLOPT_URL, $url);
curl_setopt($curl_handle, CURLOPT_CONNECTTIMEOUT, 2);
curl_setopt($curl_handle, CURLOPT_RETURNTRANSFER, 1);
$buffer = curl_exec($curl_handle);
curl_close($curl_handle);
if (empty($buffer)){
return 'Error retrieving data, please try later.';
} else {
return $buffer;
}
}

My talk at the Future of Web Apps Tour 2009 about remixing the web of data with YQL. I’ll turn this into a slidecast once I am back.

Notes

Evolution

Today I will talk a bit about an evolution that we are all part of, although we might not be aware of it yet.

What is the web?

Whenever people asked me what my job is I told them I am a web developer. This brings the question what I develop, really.

Documents

The web as it stands is made up from documents. The technologies that run it – http for transport and HTML for structure haven’t changed much over the years. Linking documents was a revolution and it made the earth a much smaller place and allowed us to collaborate. However, it got boring quickly.

Things

The web of things has been a running theme for a while. Initially it meant that all kind of devices can be connected to the web (self-ordering fridge and somesuch). It also means that with RESTful web services we can point directly at the thing we want to reach which could be a text but also an image or a video or other embedded rich content in web sites.

Data

In essence, the web is data. Data can be anything that is available on the web or referred to. Data is what we look for, data is what we get. And there is much more than meets the eye.

Connected

By connecting different data sources we even get more information and new data emerges. As humans we all learn differently and having different data sets and various ways of connecting them makes it easy for us to grasp the learnings from the data.

Hunters and Gatherers

The issue is that we overshot the goal. We collect for the sake of collecting and we spend much more time chasing the next big thing to collect than giving the things we already have some love and tag and describe them. As humans we are hard-wired to find things and collect them. It also means that we always want to do everything ourselves and not rely on others. In essence, we collected a solid mass of data and now we don’t know how to plough through it anymore. This is why we try to use technology to clean up the mess for us by injecting landmarks and machine-readable information.

Let’s take this sentence for example. There is much more in there than meets the eye.

My name is Chris. I am a German living in London, England and one of my favourite places to go is Hong Kong. I also enjoyed Brazil a lot.

By using a geolocation service I can analyze the text and add extra information that allows me to make it easy for other systems to understand this sentence. That way I can enrich the information.

My name is Chris. I am a German living in London, England (Name: London,England, GB, Type: Town, Latitude: 51.5063, Longitude: -0.12714) and one of my favourite places to go is Hong Kong. I also enjoyed Brazil (Name: Brazil, Type: Country, Latitude: -14.2429, Longitude: -54.3878) a lot.

This makes data much easier to grasp and gives it a richer experience for us all. The question is how we can do this easily.

APIs

APIs are the web data publisher’s way to give us access to their data. There are hundreds out there and each of them is different. Which leads to another problem.

Language

Each API uses its own language, ways of authenticating, data entry vocabulary and return value. You are lucky to find good documentation and many examples are hard to grasp as they are not available in the programming language you would like to work in.

Documentation can be confusing. And in most cases you don’t really want to have to dig in that deep into the API just get some information.

Simplicity

What we need is a simple way to access all these wonderful APIs and mix and match the content of them.

YQL

The Yahoo Query Language (or short YQL) is a SQL-style language for the data web. Using the YQL console you can easily build most complex queries and get them ready for copy and paste.

You simply enter your statement in the appropriate box and try it out. You can choose to get back XML or JSON and define a JavaScript callback for JSON to use as JSON-P.

Very important is the permalink link. Click this every time you do a complex query as if you reload the page by accident it will still be available to you.

The REST query is a URL ready to copy and paste into a browser or your own script.

The formatted view shows you the XML or JSON; the tree view allows you to drill down into the returned information.

Recent queries are stored, and example queries show you how it is done.

The data tables show all the available data sources. Each table comes with an own description.

What can you do with this?

Say you want to find events in Cambridge. You can query upcoming.org. Sadly enough (and because of people entering bad data) this will not result in anything useful but give you results from London!

select * from upcoming.events where location = "cambridge,uk"

By using the geo.places API you can define Cambridge without a doubt (as a woeid) and then get events.

select * from upcoming.events where woeid in (select woeid from geo.places where text="Cambridge,UK")

The diagnostics part of the resulting data set tells you which URLs where called “under the hood” and how long it took to get them.

The results section has all the events but far too much data for each of them. Say you only want the url, the title, the venue and the description.

You can select only the parts that you want:

select title,url,venue_name,
description from upcoming.events where woeid in (select woeid from geo.places where text= "cambridge,uk")

Which cuts down nicely on the resulting data.

You can get my latest updates from Twitter…

select title from twitter.user.timeline where id="codepo8"

Or only those that I replied to somebody…

select title from twitter.user.timeline where id="codepo8"and title like "%@%"

Or check several accounts!

select title from twitter.user.timeline where id="codepo8" or id="ydn" and title like "%@%"

You could also check my tweets for useful keywords:

select * from search.termextract where context in
(select title from twitter.user.timeline where id="codepo8")

You can scrape the BBC’s news site for links:

select * from html where url="http://news.bbc.co.uk" and xpath="//td[2]//a"

Or get all the alternative text of their news images:

select alt from html where url="http://news.bbc.co.uk" and xpath="//td[2]//a/img[@alt]"

And get better photos from flickr…

select * from flickr.photos.search where text in
(select alt from html where url="http://news.bbc.co.uk" and xpath="//td[2]//a/img[@alt]")

Flexibility

• mix and match APIs
• filter results
• simplify authentication
• use in console or from code
• minimal research of documentation
• caching of results
• proxied on Yahoos servers

YQL gives you a lot of flexibility when it comes to remixing the web and filtering the results. However, there are some things that can not be done with them that are possible with other systems like for example Yahoo Pipes.

Extending

YQL can be extended by Open Tables. This is a simple XML schema that redirects YQL queries to your web service. That way you can be part of the simple YQL interface without needing to change your architecture. The other benefit is that YQL will cache queries, thus hitting your servers less and also limit the access of every user to 1000 calls per hour to YQL.

One of those is for example the real estate search engine nestoria. Using their open table I can look for flats in Cambridge:

use "http://www.datatables.org/nestoria/nestoria.search.xml" as nestoria;
select * from nestoria where
place_name="Cambridge"

Open tables can be added on github to a repository. This will make them available to the YQL community.

Clicking the Show Community Tables link in the console adds all these third party tables to the interface.

Re-Use

Instead of making the language of YQL more complex we also allow for YQL execute tables which can get the data from a query and you can write a JavaScript with full E4X support to convert it to whatever you want before giving it back to the YQL engine.

Last Friday I went to sunny Southwark to go to IPC media and gave a brown-bag presentation. The topic: IPC wanted to have a chat about using libraries and why YUI might be a good choice.

I covered the fact that almost every web site out there is broken, that the reason for that first and foremost is bad communication in development teams and that users, developers and clients are all unhappy about it.

I explained that we develop for an unknown environment and that the development tools we have are just not good enough (albeit getting better every year).

I lamented the lack of documentation and handover procedures and that as developers we are inclined to build small solutions for one problem over and over again rather than contributing to a larger framework of solutions.

I pointed out that we tend to go from visual display to functionality rather than the other way around which is why we produce inaccessible and overly complicated products.

As solutions to these problems I showed how YUI is built and maintained, how its development tools allow you to build in a predictable fashion and that it does very well what every library out there should do for you: making web development easier and less random.

Slides

Here are the slides of the talk:

• PDF without notes – 9.5MB.
• PDF with notes – 9.5MB.

Audio

I also recorded the talk audio and you can download the recording at archive.org. Listening to the audio it sounds a bit of a rant, however it is not, I am just very passionate about the subject of professional web development and making the internet the #1 media :).

Notes

Professional Web Development Tools

Almost every web site is broken.

If you look around the web you will find that almost every site is broken in one way or another. This starts with small display glitches and ends with the sites being inaccessible or not working for users out there.

This is bad.

This is really bad. It hurts the web as a media. We re-invent the web every year as we just cannot seem to get it to work for us.

Unhappy visitors.

Broken web sites lead to unhappy visitors. The real problem there is that unhappy visitors do not complain to the people who could fix the issues. Most visitors either think they’ve done something wrong or just try to find another site that offers the same content and works. Both of these visitors will never come back. Other visitors complain but get stuck in help desks and never get their problem fixed as it is highly unlikely to ever reach the developers who could fix them.

Unhappy developers.

It doesn’t reach the developers as they are too busy with building new functionality and other sites. If we don’t build new things all the time we are neither happy developers nor seen as efficient employers. Fixing things isn’t sexy.

Unhappy clients.

This leads to unhappy clients. If a client realizes something doesn’t work on the site they paid good money for they want it fixed, regardless of how fringe the problem is and if it only shows up on their machine with their (most of the time outdated) setup.

Reasons

There are many reasons for the broken web, and nearly all of them are our own fault or based on misconceptions.

Lack of communication

Probably the biggest problem of web development is that the different parties involved do not talk to each other or know each others tasks. Developers think they know more than designers, designers think developers are not creative enough in using the arsenal at their hands and product managers see the brand more than the media and are oblivious to the technical boundaries and freedoms the internet gives us. Furthermore we all have our deadlines, deliveries and reports to make and write which takes up too much of our time.

Development environment

Web development has the most terrible and undefined environment ever. There are thousands of browser configurations and versions, each of them failing in different ways. There is a lack of good error reporting, difference in server configurations, connection issues… you name it. Our development is hit and miss and we fix more bugs than we write code.

Piecemeal development

As web developers we always try to build small solutions that solve a problem we have right now. We don’t really consider that all things on a site and across sites should work smoothly together. We’ve been disappointed so many times that we don’t really believe in that.

Lack of handover and documentation

The piecemeal development also means we don’t really document or hand something over. As the next developer is most likely as inclined as we are to build something new (as it surely will be much better than the crud we are asked to maintain) there is no point in that.

Interface to functionality

The biggest issue is that we start with the interface and the cool effect and then work our way down to what the user needs to achieve. We tend to forget very fast that not everybody has the same experience or could benefit from the great shiny interface we want to build. There is a skeleton under every web application and if that skeleton is weak it will break no matter how pretty and shiny we make it.

Solutions

There are solutions for all these issues.

Back to Basics

The first thing to think about is going back to basics when it comes to development. How does the web work, what is the most basic way of reaching a certain goal.

http://uk.tv.yahoo.com/#yeug-search – is a search box with several options. It used JavaScript to change the form’s action when any of the links were clicked.

Task: Define type of search, enter search term,submit form.

There was no need for JavaScript – all we needed was a radio button group and doing the forking on the backend. Notice that the fieldset, the options and the search button form a logical sentence. This is very important for accessibility.

TV channel programme

http://uk.tv.yahoo.com/#ytv-listings – the hardest interface to build as a web developer. Looks like a data table but could have shows that are one minute long! This would mean the table has to have 180 columns and use colspan on every table cell.

Analyse what data you display, and find the easiest way to show it.
Then make it look the way you want it to.

The information the data displays is much easier shown as headlines and ordered lists. CSS does the rest.

Build things people want and know how to use.

Here is where Yahoo offers their findings of user testing with real end users – http://developer.yahoo.com/ypatterns. There is nothing that can replace this knowledge and it is normally very expensive to come by. Before you even think about building an own interface to solve a problem users have to solve, give this a whirl.

Using technology for good

Flash video players are to date the best way to show video – http://uk.video.yahoo.com/. However, they have no reliable keyboard control. By providing buttons that work in HTML and control the video via an API you can make it accessible to all.

Aiming for excellence.

This is the new Yahoo currency converter. It is an amazing piece of web development. It works for all users (including screen reader users) and makes it easy to convert currencies.

Here we explained in detail how it works and the approach we took in developing it.

Removing browsers

The biggest step to professional development and keeping our sanity is to get the random element of browsers out of the equation. You cannot support all the browsers in the world and neither should you.

The graded browser support is a framework to define which browsers you test for and get the full experience. Unknown browsers only get what works in them – no JavaScript and even more obscure browsers get no CSS either.

Making browsers behave.

Libraries have one job: make browsers work. Support is the most random thing in our world as web developers therefore it makes a lot of sense to put all the dirty hacking and fixing of wrong browser behaviour into libraries. YUI is what Yahoo built and uses exactly for that purpose.

First issue is that every browser has an internal style sheet that renders HTML. All of them are different which makes it impossible to develop a reliable look and feel across browsers. YUI Reset works around that.

The same applies to typography. By using the YUI fonts CSS you reset the browser typography to allow you to define pixel sizes as percentages, thus having control and allowing users to resize the fonts.

The CSS grids allow you to create multi column layouts that work across all the A-level browsers easily and reliably. Source order independence comes free, too.

If you are lazy, you can also use the grids builder, define your layout, hit the show code button and get a copy + paste HTML document. The CSS will come from our CDN, which means it gets delivered to your customers from a computer near them geographically.

Doing one job at a time.

YUI does what we as developers would love to be able to do: concentrating on one task at a time. Other than “catch-all” libraries, YUI is cut up into several components, each doing one thing. You can mix and match them to your needs.

DOM access.

One of these components is YAHOO.util.Dom which gives you access to everything that happens in the DOM and convenience methods around the more annoying things the W3C DOM API has.

Using this I can write a script that shows the perfect YUI grid for every size of browser.

Predicting issues and fixing them.

One thing you should do as a developer is being paranoid about things breaking. You should be able to see what can go wrong and set traps for it not to happen.

position:fixed is sexy!

Positioning elements fixed can be very cool. Say for example you have a long document but you want to show the navigation next to regardless of how far down the page you scrolled. Another cool use would be a comments field that allows you to copy and paste quotes from the document.

Positioning the navigation as fixed makes it always visible on the page.

However if the browser window is too small there is no way to reach the elements below.

This small script fixes this problem. Using getRegion I can get the size of any element on the page and getViewportHeight() gives me the available space. If there is more space than needed, fixed can be applied.

Once fixed, let’s re-use.

Using the YUI components we build all kind of widgets based on the design patterns.

Using these free widgets you can re-build yahoo mail yourself.

Re-use means the ability to style differently.

All the widgets are style-able using CSS. You don’t need to know JavaScript or change their code to make them look completely different.

Document your work.

The YUI comes with extensive documentation, both created from comments in the code (JavaDoc style) and step-by-step tutorials. The system that generates the docs from the source code is also available as open source.

Learn by example.

YUI comes with over 300 copy and paste examples of how to use the different components and widgets. As this is how most developers work, we realized that this is a very important part of our success.

Allow for extension.

YUI uses custom events for all of this. This allows you to completely separate your own code from the library. Instead of having to call library methods or call your functions from the library all you need to do is to fire or subscribe to events.

Know what is happening.

Not every browser comes with a great debugging suite like FireBug or Opera’s Dragonfly. This is why Yahoo comes with a logging control.

The logger allows you to debug in any browser that the YUI works in. In addition to this all the YUI widgets and components are shipped as debug versions which report everything they do to the logger. This gives you full control over what is happening and when.

Monitor performance.

The YUI profiler allows you to monitor JavaScript performance – even of non-YUI scripts.

Test before you write.

The same applies to the YUI Test suite. Using this you can apply test-driven development methodologies to JavaScript development.

YUI 3

YUI3 is the new version of YUI, there are many speed and size improvements and we changed the way YUI works significantly to make it more secure, performant and allows you to write much less code to achieve your goal.

Performance

JavaScript performance is one thing, but in order to deliver really successful web sites there are many tricks to apply to create happy end users. The exceptional performance section of the Yahoo Developer Network has them all listed.

YSlow – a Firefox extension allows you to test any web site against these tips and rules and you get immediate, relevant information how to improve the performance of your site.

July and August is a very slow time for me this year as there aren’t any conferences I am speaking at and no hack days planned. September will be packed again, though.

This is why I am right now planning to do some brown bag presentations in the London area. I’ve already been at IPC media, will go next Friday to ebay/gumtree/skype/paypal/shopping.com in Richmond and have had Sky and Thomson Group/TUI show some interest on twitter.

Brown bag presentations are by definition talks during lunch break where people can bring their food along. That way your employees can get some information or discuss their issues with an invited expert without disruption of your normal office hours and times.

I’d be happy to come around and give a talk on all the things I am passionate about. Currently these things are:

• Opening your own and reaching data on the web easily with YQL
• Mining your content and enhancing it with geographical information (Yahoo Placemaker)
• Building specialist search engines (BOSS)
• Accessibility and that it is not magic
• Professional Web Development
• Performance of web products and how to improve it

I am open to other topics, though.

If you are interested in some of that and you are in the Oyster card area, comment here or contact me on Twitter and we can talk date, location, topic and time. I don’t expect any pay, but normally a coffee, some water and a sandwich :). The whole idea of the exercise is to be able to reach developers that normally do not get the chance to go to conferences.

Things that made me happy this morning:

• Naughty photos with the people removed is just not the same
• Panic has a 3 day half-price sale on Transmit, Coda, Candybar and Unison and even a corny commercial for it (you gotta love the Kleenex)
• The German mail released a pretty cool collaborative magazine creator in Flash – in German of course.
• The sales of this 3 Wolves howling at the moon T-shirt allegedly went up 2300% after the fake overly excited comments started.
• Simon Willison creates geocoders – a meta geocoder using three different APIs under the hood
• Keyboard cat the true Internet story gives us all the info about keyboard cat. Now let’s bury it.
• More videos: Transforminators mashup, Damien Walters show reel (parcours, free running – also read the hilarious b3ta comments and check his terrible myspace profile and Bach Bach revolution – Two women play Bach’s Toccata and Fugue in D minor on a giant piano.
• Good quote “Direct HTML input is the autobahn to invalid XHTML.” -Carl Camera