• Posts Tagged ‘YUI’

    Participating in the Web of Data with Open Standards

    Wednesday, March 17th, 2010

    These are the detail notes for my talk at the mix10 conference in Las Vegas. The description of my talk was the following:

    Web development as we do it right now is on the way out. The future of the web is what its founders have planned a long time ago: loosely joined pieces of information for you to pick and choose and put together in interfaces catered to your end users. In this session, see how to build a web portfolio that is always up-to-date, maintained by using the web rather than learning a bespoke interface and high in performance as the data is pulled and cached for you by a high traffic server farm rather than your server. If you wondered how you can leave your footprint on the web without spending thousands on advertising and development, here are some answers.

    The slides

    The detailed notes / transcript

    Welcome to the web

    When I started using the web I was working for a radio station as a newscaster and producer. I’ve always dabbled with computers and connecting them world-wide and I was simply and utterly amazed at how easy it was to find information. This was 1996 and I convinced my boss at the radio station that it is of utmost importance that I get an internet access to be the first in our town to get the news from Associated Press and other very important sources. In reality, I just fell in love with the web and its possibilities.

    Working as a professional web developer

    I quit my job soon after that and built my first web sites. Together with some friends we maintained a banner exchange and I was admin for a few mailing lists, IRC and built a lot of really bad web sites which – back then – were the bee’s knees.

    I went through the first .com boom living in a hotel in Los Angeles with all expenses paid for writing HTML.

    Joining the Enterprise crew

    I then worked for an agency and delivered lots of products with enterprise level content management systems. Massive systems intended to replace the inadequate systems of the past.

    This is when things went wrong

    For a few years I released things that were incredibly expensive and meant that the people using them had to be sent on £3K+ trainings just to be able to do the things they already did before that – only much less effective. Or did they?

    Joining the corp environment (to a degree)

    Again, this was a very interesting step as it meant I moved away from the large world of delivering for clients to concentrating on one single company – one of the companies that defined the world wide web as it is now and also a large player in a newly emerging world.

    Hey Ho, Web 2.0 highfive

    The web 2.0 came around and run rampant in the media and in the mind of early adopters. User Generated Content sounded like an awesome scam to make millions of dollars with content that comes for free – all you need to do is set up the infrastructure and find the right people.

    And it went wrong again.

    The main purpose of the first round of web2.0 was to be as visible as possible – doesn’t matter if all the content added by your users is really just “first” and “you’re a fag” comments – as long as the user numbers were great you won.

    And now?

    Makes you wonder what comes now, doesn’t it?

    The web as the platform, the Mobile Web and GeoLocation

    This is where we are. We can use the web as our platform, we work with virtual servers, hosted services and are happy to mix and match different services to get our UGC fix. What is missing though is the glue.

    Market changes leave a track

    The thing we keep forgetting though are the users. All the changes we’ve gone through bred generations of users that are happy to use what they learnt to do in their job but are not happy having to re-learn basic chores over and over again.

    Time to shift down a gear

    We have the infrastructure in place, we can already make this work. I get a real feeling that we are not innovating but moving the web to a mainstream media. Streaming TV without commenting, massive successes like Farmville and MafiaWars makes me realize that the web is becoming ubiquitous but that it is also boring for me as someone who wants to move it further. In order to accelerate with a car you need to shift down a gear (or kick down the accelerator with an automatic). This is the time to do so.

    Finding the common denominator

    What is the common denominator that drove all the innovation and movement in the past? Data. Information becoming readily available and interesting by mixing it with other information sources to find the story in the data.

    Tapping into the world of data

    Data is around us, it is just not well structured. People cannot even be bothered to write semantic HTML and it is tough to teach editors to enter tags, alternative text for images and video descriptions. The reason is that we lack success stories and learning our CMS takes too long. Instead teaching people how to use systems we should teach people how to structure information and simplify the indexing process rather than empowering to make things pretty and shiny – this is what other experts do better.

    Why APIs work

    APIs or Application Programming Interfaces are the best thing you can think of if you want to build something right now. If you remove the information from the interface – both in terms of data entry and when it comes to consumption we can build a web that works for everybody. By making information searchable and allowing filtering out-of-the-box you can scale to any size or cut down to the utmost necessary.

    APIs made easy

    The Yahoo Query language or short YQL is a very simple language that allows you to use the web like you would use a database. In its simplest form a YQL query looks like this:

    select {what} from {where} where {conditions}

    Say for example you want to find photos in Flickr with the keyword donkey. The YQL statement for this is the following:

    select * from flickr.photos.search
    where text="donkey" and license=4

    The license=4 means that the photos you are retrieving and displaying are licensed with Creative Commons, which means that you are allowed to show them on your page. This is very important as Flickr users are happy to show photos but not all are happy for you to use their photos in your product. Play nice and we all get more good data.

    YQL is not limited to Yahoo APIs and data – anything on the web can become accessible with it. If you want to find a flower pot in the Bay Area you can use for example Craigslist with YQL:

    select * from craigslist.search where
    location="sfbay" and type="sss" and query="flower pot"

    If you want to get the latest news from Google about the topic of healthcare, you can use:

    select * from google.news where q="healthcare"

    If you want to collate different APIs and get one data set back, you can use the query.multi table. The following for example searches the New York Times archive, Microsoft Bing News and Google News for the term healthcare and returns one collated set of results:

    select * from query.multi where queries in (
    'select * from nyt.article.search where query="healthcare"',
    'select * from microsoft.bing.news where query="healthcare"',
    'select * from google.news where q="healthcare"'
    )

    YQL even allows you to get information when there is no API available. For example in order to get the text of all the latest headlines from Fox News you can use YQL’s HTML table:

    select content from html where
    url="http://www.foxnews.com/" and xpath="//h2/a"

    This goes to the foxnews.com home page, retrieves the HTML, runs it through the W3C HTML Tidy tool to clean up broken HTML and filters the information down to the text content of all the links inside headings of the second level. This is done via the xpath statement //h2/a which means all links inside h2 elements and you retrieve the content instead of everything with the *.

    You can then do more with this content – for example using the Google translation API to translate the headlines into French:

    select * from google.translate where q in (
    select content from html where url="http://www.foxnews.com/"
    and xpath="//h2/a"
    ) and target="fr"

    As you can see it is pretty easy to mix and match different APIs to reach your goal – all in the same language.

    YQL goes further than just reading data though. If you have an API that allows for writing to it, you can do insert, update and delete commands much like you can with a database. You can for example post a blog post on a WordPress blog with the following command:

    insert into wordpress.post
    (title, description, blogurl, username, password)
    values&nbsp;("Test&nbsp;Title",&nbsp;"This&nbsp;is&nbsp;a&nbsp;test&nbsp;body",&nbsp;"http://yqltest.wordpress.com",&nbsp;"yqltest",&nbsp;"password")</pre>
    
    	

    The insert, update and delete tables require you to use HTTPS to send the information, which means that although your name and password are perfectly readable here they won’t leak out to the public – unless you are using them inside JavaScript where they would be readable.

    The <span class="caps">YQL</span> endpoint

    As YQL is a web service, it needs an endpoint on the web. For tables that don’t need any authentication this end point is the following URL:

    http://query.yahooapis.com/v1/public/yql?
    q={query}
    &#38;format=xml|json
    &#38;callback={callbackfunction}</pre>
    
    	

    You can get the information back as XML or as JSON format. If you choose JSON then you can use the information in JavaScript and by providing a callback function even without any server interaction.

    Benefits of using <span class="caps">YQL</span>

    YQL is a way to use the web as a database. Instead of using your time reading up on different APIs, requesting access and learning how they work you can simply access and mix the data and you get it back ready to use a few minutes later. YQL does all this for you:

    • No time wasted reading API docs – every YQL table comes with a desc command that tells you what parameters are expected and what data will come back for you to use.
    • Creating complex queries with the console. – the YQL console allows you to play with YQL and quickly click together complex queries in an easy to use interface. It also previews the information directly to you so you can see what comes back and in which format.
    • Filter data before useYQL allows you to select all the data in a certain query using the * or specifically define what you want – down to a very granular level. In very limited application environments like mobile devices this can be a real benefit. It also means that you don’t need to spend a lot of time converting information to something useful after you requested it but even before you send it to the interface layer.
    • Fast pipesYQL is hosted on a distributed server network that is very well connected to the internet. Chances are that the server is fast to reach and a few times faster than your own server when it comes to accessing API servers from all over the world.
    • Caching + convertingYQL by default gives out XML or JSON which can be useful to convert any data that is on the web in another format to these highly versatile and open data formats. YQL also has an in-built caching system that only gives you new data when it is available and returns it very fast if all you need to do is request in another time.
    • Server-side JavaScript – if the out-of-the-box filtering, sorting and converting methods are not enough for you you can use YQL execute tables to run the returned data through JavaScript before YQL gives it back to you. This allows for all the conversion and extension power JavaScript comes with in a safe and powerful environment as we use Rhino to run it server-side.

    All in all YQL allows you to really use and access data without the hassle of resorting to XSLT, Regular Expressions, scraping and other tried-and-true but also complex ways of getting things web-ready.

    Government as a trailblazer?

    One thing that gets me very excited lately is governments throwing out information for us to use. Data that has been gathered with our tax money is now available for experts and laymen to play with it, look at it and see where the interesting parts are.

    Conjuring APIs out of thin air

    The problem is that the government data is not available for us as APIs – instead you will find it entered into Excel spreadsheets and in all kind of other data formats coming out of – yes – Microsoft products. We could now bitch about this and claim this is old school or do something about it. So how can we do this?

    A few weeks ago I build http://winterolympicsmedals.com – an interface to research the history of the Medals won in the Winter Olympics from 1924 up to now. There is no API for that and this data is also not available anywhere.

    What happened was that the UK newspaper the Guardian released a dataset of this information on their data blog (this is a free service the newspaper provides). The data was provided as an Excel sheet and all I did was upload it to Google Docs. I then selected “Share” and “export as CSV” which gives me the following URL:

    http://spreadsheets.google.com/pub?key=tpWDkIZMZleQaREf493v1Jw&output=csv

    Now, using YQL with this we have a web service:

    select&nbsp;*&nbsp;from&nbsp;csv&nbsp;where&nbsp;url="http://spreadsheets.google.com/pub?key=tpWDkIZMZleQaREf493v1Jw&#38;output=csv"&nbsp;and&nbsp;columns="Year,City,Sport,Discipline,Country,Event,&nbsp;Gender,Type"

    By giving the CSV some columns we can now filter by them, for example to get all the Silver medals of the 1924 games we can use:

    select&nbsp;*&nbsp;from&nbsp;csv&nbsp;where&nbsp;url="http://spreadsheets.google.com/pub?key=tpWDkIZMZleQaREf493v1Jw&#38;output=csv"&nbsp;and&nbsp;columns="Year,City,Sport,Discipline,Country,Event,&nbsp;Gender,Type"&nbsp;and&nbsp;Year="1924"&nbsp;and&nbsp;Medals="Silver"

    Spreadsheet to web services made easy!

    This makes it easy for us to turn any spreadsheet into a web service. With Google docs as the cloud storage and YQL as the interface and doing the caching and limiting for us it is not that hard to do.

    In order to make the creation of a search form and results table easier, I built a PHP script that takes this job on:

    &lt;?php
    include('csvtoservice.php');
    $content&nbsp;=&nbsp;csvtoservice('http://winterolympicsmedals.com/medals.csv');
    if($content){
    
    if($content['form']){
    echo&nbsp;'&lt;h4&gt;Filters&lt;/h4&gt;';
    echo&nbsp;$content['form'];
    }
    
    if($content['table']){
    echo&nbsp;'&lt;h4&gt;Results&lt;/h4&gt;';
    echo&nbsp;$content['table'];
    }
    
    }
    ?&gt;

    Some examples

    Let’s quickly go through some examples that show you the power of YQL and using already existing free, open systems to build a web interface.

    • GooHooBi is a search interface that allows you to search the web simultaneously using Google, Yahoo and Bing. You can see how it was done in a live screencast on Vimeo.
    • UK House Prices is a mashup I build for the release of the UK government open data site. It allows you to compare the house prices in England and Wales from 1996 up to now and see where it makes sense to buy a place and how stable the prices are in that area.
    • GeoMaker is a

    In summary

    • We have the network and we have the technology.
    • We have people who work effectively with the tools they use.
    • We have a new generation coming who naturally use the internet and are happy with our web interfaces.
    • If we use our efforts 50/50 on new and building APIs and converters to get the data of the old the web will rock.

    Homework

    As a homework I want people to have a look at the open tables for YQL on GitHub
    and see what is missing. I’d especially invite people to add their APIs to the tables using the open table documentation. We want your data to allow people to play with it.

    Learn more

    If you want to learn more about building sites like the ones I showed with this approach, there is a Video of me talking you through the building of UK-House-Prices.com available on the YUI blog.

    GeoPlanet Explorer – another showcase for quick development with YQL and YUI

    Friday, February 26th, 2010

    A few days ago Gary Gale pinged me on messenger and subsequently carried a cup of coffee to my desk to pester me with another challenge. This time he talked about just how rich and cool the GeoPlanet data is and that it is tough to show people this in a simple interface. Internally we have a few pretty cool tools for testing and analyzing the data but most of them are too loaded with information only understandable for the geo folk out there. So in essence, the benevolent overlord of geo technologies in Yahoo was asking to build a simple interface to navigate the GeoPlanet data.

    Well, this morning I got a chance to have a go at his request and here’s the GeoPlanet Explorer interface for you. Check the following screencast to see it in action:

    Building the interface wasn’t magic – I used YQL to access the data, write a few lines of PHP to display it in a nested list and then added a few lines of YUI3 JavaScript to collapse and expand the location details.

    Notice that the whole interface uses progressive enhancement throughout. If you have no JavaScript at your disposal you get a static map and all the information in one single page. The lat/lon links open in Yahoo Maps and you can see the location there.

    If you have JavaScript enabled the interface collapses and the map is Ajax and will be refreshed every time you click on a lat/lon link.

    The source code of the GeoPlanet Explorer is available on GitHub and it can give you a few pointers how to use the GeoPlanet API with YQL for your own solutions.

    How I build my data.gov.uk mashup – UK-House-Prices.com

    Thursday, January 21st, 2010

    UK-House-Prices.com is a web site to see how the prices in a certain area changed over the years using a data set released by the UK government as part of the data.gov.uk initiative.

    Here’s a screencast showing the app:

    The first step was to get the right data. I was lucky enough to be invited to the initial “hack day” and pre-release of the data and looked around for something to mash up. Initially I wanted to do something with environmental data but I found a lot of it to be very old. Therefore I just did a search for “2009” at data.gov.uk and found that the house prices data from 1996 to now in England and Wales is available. The plan was set. This was it:

    • I wanted to build an interface to show this information that was very fast, very portable and show a nice map of the area next to the numbers.
    • I wanted to build this as a web app and as an application for the Yahoo homepage (as I needed to build one as a demo anyways)
    • Traffic and speed was the most important issue – as this might get huge.

    Cleaning and converting data

    I got the spreadsheet and was confronted with my old nemesis: Excel. After saving the sheet as CSV and spending some fun time regular expressions and split() I had the data in a cleaner, and more usable version (JSON, specifically). One fun part is that when there was no data available for a certain area the field was either “..”, “n/a” or just empty. Something to work around. The numbers were also formatted like 100,312 which is nice on the eye but needs un-doing when you want to sort them outside Excel.

    Once I had the list of locations and their numbers I wanted to turn them into geographical locations to display maps of the area. For this I used Yahoo Placemaker, especially the YQL table (see an example for Rugby in the YQL console). This is the script I ran over the list of locations:

    
    $out&nbsp;=&nbsp;'';
    for($i=0;$i&lt;sizeof($lines);$i++){
    $select&nbsp;=&nbsp;preg_replace('/,.*/','',$lines[$i]);
    $select&nbsp;=&nbsp;preg_replace('/&nbsp;UA/','',$select);
    $url&nbsp;=&nbsp;'http://query.yahooapis.com/v1/public/yql?q=select%20match.place.woeId%2Cmatch.place.centroid%20from%20geo.placemaker%20where%20documentContent%20%3D%20%22'.urlencode($select.',uk').'%22%20AND%20documentType%3D%22text%2Fplain%22%20and%20appid%20%3D%20%22%22%20limit%201&#38;format=json&#38;env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys';
    $ch&nbsp;=&nbsp;curl_init();
    curl_setopt($ch,&nbsp;CURLOPT_URL,&nbsp;$url);
    curl_setopt($ch,&nbsp;CURLOPT_RETURNTRANSFER,&nbsp;1);
    $output&nbsp;=&nbsp;curl_exec($ch);
    curl_close($ch);
    $data&nbsp;=&nbsp;json_decode($output);
    echo&nbsp;'{"place":"'.$select.'",';
    echo&nbsp;'"w":"'.$data-&gt;query-&gt;results-&gt;matches-&gt;match-&gt;place-&gt;woeId.'",';
    echo&nbsp;'"lat":"'.$data-&gt;query-&gt;results-&gt;matches-&gt;match-&gt;place-&gt;centroid-&gt;latitude.'",';
    echo&nbsp;'"lon":"'.$data-&gt;query-&gt;results-&gt;matches-&gt;match-&gt;place-&gt;centroid-&gt;longitude.'"'."},n";
    ;
    }

    That was that – I had a data set I can work with.

    Adding more information

    The next thing I wanted to add was some more information about the area which meant using maps. As both Yahoo and Google maps have static map versions but are rate limited I wondered if there is a free version of that. And there is. Openstreetmap was the answer, especially the somewhat unofficial API I found with Google. To play safe, I wrote a script that gets the images and I cache it on my server to avoid killing this API.

    I also wanted to show currently available houses in the area in case you are looking to buy. For this the natural choice for me was to use Nestoria as they also have an open YQL table (see the Nestoria table in the YQL console). So I used YQL and sorted the results by date:

    select * from nestoria.search where place_name="Rugby" | sort(field='updated_in_days')

    Using this I can get offers in the area live:

    $url&nbsp;=&nbsp;'http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20nestoria.search%20where%20place_name%3D%22'.urlencode($city).'%22%20|%20sort%28field%3D%27updated_in_days%27%29&#38;format=json&#38;env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys&#38;diagnostics=false';
    $ch&nbsp;=&nbsp;curl_init();
    curl_setopt($ch,&nbsp;CURLOPT_URL,&nbsp;$url);
    curl_setopt($ch,&nbsp;CURLOPT_RETURNTRANSFER,&nbsp;1);
    $output&nbsp;=&nbsp;curl_exec($ch);
    curl_close($ch);
    $data&nbsp;=&nbsp;json_decode($output);
    if($data-&gt;query-&gt;results){
    $i=0;
    $results&nbsp;=&nbsp;array_slice($data-&gt;query-&gt;results-&gt;listings,0,5);
    if(sizeof($results)&gt;0){
    echo&nbsp;'&lt;div&nbsp;class="listings"&gt;&lt;h3&gt;Current&nbsp;property&nbsp;listings&nbsp;(powered&nbsp;by&nbsp;&lt;a&nbsp;href="http://www.nestoria.co.uk/"&gt;Nestoria&lt;/a&gt;)&nbsp;&lt;input&nbsp;type="button"&nbsp;class="hide"&nbsp;value="hide"&gt;&lt;/h3&gt;&lt;ul&gt;';
    foreach($results&nbsp;as&nbsp;$r){
    echo&nbsp;'&lt;li&gt;&lt;img&nbsp;src="'.($r-&gt;thumb_url).'"&nbsp;alt=""&gt;';
    echo&nbsp;'&lt;h4&gt;&lt;a&nbsp;href="'.($r-&gt;lister_url).'"&gt;'.($r-&gt;title).'&lt;/a&gt;&lt;/h4&gt;';
    echo&nbsp;'&lt;p&gt;&nbsp;Price:&nbsp;'.($r-&gt;price_formatted).',&nbsp;Type&nbsp;of&nbsp;property:&nbsp;'.ucfirst($r-&gt;property_type).',&nbsp;Updated:&nbsp;'.($r-&gt;updated_in_days_formatted).'&nbsp;('.($r-&gt;updated_in_days).'&nbsp;days)&lt;/p&gt;';
    echo&nbsp;'&lt;p&gt;Listed&nbsp;at:&nbsp;'.($r-&gt;datasource_name).'&nbsp;by&nbsp;'.($r-&gt;lister_name).'.&lt;/p&gt;';
    echo&nbsp;'&lt;/li&gt;';
    }
    echo&nbsp;'&lt;/ul&gt;&lt;/div&gt;';
    }
    }

    Finding a charting solution

    Adding interactive charts was the next step. I had a few issues with that:

    • While Google charts are full of win, they are rate-limited and I didn’t want to pull images. As the app was also meant to become a Yahoo application every image would have to be run through Caja for safety reasons which slowed it down.
    • Canvas and Flash solutions like YUI charts or Raphael were also not possible because of the performance of the YAP app.

    So I wrote my own pure CSS bar charts to work around that issue.

    Building the API

    I put all these solutions together and built a small API that will give me the search results with three parameters: the location as an id and the start and end of the time range.

    http://uk-house-prices.com/graphs.php?loc=1&#38;start=10&#38;end=20

    Building the interface

    To build the interface, I went all-out YUI. I took the YUI grids builder to create the main layout, the AutoComplete demo, the dual slider demo and the button and put them all together. Add an Ajax call to the form, and you are done. OK, I admit, there was quite a bit of cleaning up to be done :)

    Notice that I am using progressive enhancement all the way. Without JavaScript you get dropdowns:

    UK House Prices - without JavaScript by  you.

    That’s it

    The next thing I had to do is move the app over to the Yahoo Application Platform which was easy as I based it on an API - but this is another blog post :)

    Things you can use – my talk at the Deveoper Evening with Yahoo and Opera in Oslo, Norway

    Thursday, December 3rd, 2009

    I am currently in (freezing) Oslo in Norway and about to leave for the NITH university for the Yahoo and Opera developer evening. Here are the slides and notes of what I am about to present tonight. Audio will follow laterAudio is now available.

    For those who will attend tonight: stop right here, you’ll just spoil it for yourself :)

    Slides

    Audio

    The audio recording is available on archive.org or as a 54MB MP3 file.

    Notes

    Things you can use (by the Yahoo Developer Network and friends)

    In the following hour or so I will be talking to you about some of the things that have been done for you that you can build on. Web development is confusing enough as it is. There is no need to make it more complex by not using what has been done for us already. But first, a few facts about me:

    I am Chris and I’ve been developing web sites for about 12 years. I’ve worked with numerous frameworks and CMS and I’ve delivered various international and high-traffic sites. I’ve written several books and dozens of articles and hundreds of blog posts on the subject of web development, accessibility, performance, maintainability, internationalization and many other things I had to battle in my day to day job.

    Right now I work as a Developer Evangelist, which is a pretty new job in the market and as many people were confused about it, I wrote a Developer Evangelism Handbook which is free and creative commons online, but you can also buy a hard copy on Lulu.com.

    I work for the Yahoo Developer Network and you will find most of the things I will talk about today there.

    Learning the basics

    The good thing about these days we live in is that we have great resources on the web to learn good basics of web development. That we still learn it by viewing source and trial and error after copy and paste is human nature but also keeps us from evolving. The Opera Web Standards curriculum is an amazing resource to learn web standards based development (as is the WaSP interact) and the Yahoo Developer Network theatre is full of videos of talks and tutorials. All of this is free to use and to build upon.

    Starting with a clean canvas

    One thing I learnt is that in order to deliver products that work and are fun to maintain you need to work on a solid base. Web development is hindered by the fact that our deployment environment is totally unknown to us. This is why we need to even the playing field before we should go out on the pitch to play.

    You achieve this by defining what you call “support” for browsers. In the case of Yahoo this is the graded browser support document. If you aim to make your web product look and work the same on every browser out there you are doing it wrong. Web design is meant to go with the flow and accustom itself to the ability of the user agent (browser in most cases). At Yahoo, we have the Graded Browser Support for this – browsers that are not capable of supporting new technologies will not get them. Web sites are not meant to look and work the same everywhere. On the contrary – the ability to accustom the interface to different user agents is what makes web development so powerful.

    CSS frameworks and frontend libraries

    The next step is to free ourselves from the limitations of browsers and especially their differences. This is what CSS frameworks and front end libraries like jQuery, Mootools, Dojo, YUI and many more are for. All of these have the same goal: allow you to build code that is predictable and limited to the bare necessities. We should not have to bloat our code just to make our products work with random browser implementation problems. These are a moving target as there is all kind of weirdness happening across the board. Libraries make our job predictable and allow us to use web standards without catering for browsers. If you build your code based on libraries you can fix your product for the next browser by upgrading the library. If you choose to do everything yourself – good luck.

    Building interfaces that work

    The next thing to consider is that an application interface is not just the look and feel. In order to make it work for everybody we’ll need to understand the ways users interact with our products. This includes simple usability (not overloading the user, not confusing the user) but also means knowing about different interaction channels – for example keyboard users. A great resource for starting the journey towards usable interfaces is the Yahoo Design pattern library. There we collected information how our end users use the web and reach a goal easily. If anything, have a look at these patterns before you start building your first widget or application. They even come with stencils for different designer tools.

    Using the web

    The big change in web development over the last few years was that we stopped trying to do everything ourselves. The web is full of specialized systems such as YouTube, Flickr and Google Maps that allow you to host data in specific formats and make it dead easy for you to convert and re-use that data in web formats. Using this information happens via Application Programming Interfaces or short APIs. These allow you to demand data in a certain format and get back only what you need. There are hundreds of APIs available on the web. For an idea of what is available, check out programmableweb.com.

    Thinking data first

    The main thing to be aware of if you want to build great products is to separate your data from your presentation. This is essential to allow for localization, internationalization and to keep your code maintainable. In the case of a mashup of different data sources this means you need to think about making it as easy as possible for you to use different APIs. The complexity of your product increases with the number of APIs you use. Every API has different ways to authenticate, expects different parameters and returns data in different formats.

    Mixing the web with YQL

    YQL is a solution that Yahoo built for its own needs. All of our products are built on APIs – for scalability reasons. Having to learn all these APIs and negotiating access cost us a lot of our time so we thought we’d come up with a better solution. This solution is called YQL. YQL is a SQL-style language to get data from the web. The following query would get us photos of London from flickr:

    select&nbsp;*&nbsp;from&nbsp;geo.places&nbsp;where&nbsp;text="london"

    Or would it? Actually it would give us photos with the text “london” and not photos taken in Oslo. If we wanted that we need to use another API. The Yahoo Geo APIs allow you to define any place on earth and get it back as a “where on earth ID” or short woeid. This format is supported by flickr, so we can use these APIs together. Twitter will also soon support this.

    select&nbsp;*&nbsp;from&nbsp;flickr.photos.search&nbsp;where&nbsp;woe_id&nbsp;in&nbsp;(
    select&nbsp;woeid&nbsp;from&nbsp;geo.places
    where&nbsp;text="london"
    )

    This gives you some data of the photos you want to show but not all, so let’s use another API method to get that information.

    select&nbsp;*&nbsp;from&nbsp;flickr.photos.info&nbsp;where&nbsp;photo_id&nbsp;in(
    select&nbsp;id&nbsp;from&nbsp;flickr.photos.search&nbsp;where&nbsp;woe_id&nbsp;in(
    select&nbsp;woeid&nbsp;from&nbsp;geo.places&nbsp;where&nbsp;text="london"
    )
    )

    This is a lot of data so in order to only retrieve what we really need we can filter the data down by replacing the *:

    select&nbsp;farm,id,secret,owner.realname,
    server,title,urls.url.content
    from&nbsp;flickr.photos.info&nbsp;where&nbsp;photo_id&nbsp;in(
    select&nbsp;id&nbsp;from&nbsp;flickr.photos.search&nbsp;where&nbsp;woe_id&nbsp;in(
    select&nbsp;woeid&nbsp;from&nbsp;geo.places&nbsp;where&nbsp;text="london"
    )
    )

    Using this query in the YQL console, choosing YQL as the output format and “flickr” as the callback will give us a valid URL to use in a browser or script:

    http://query.yahooapis.com/v1/public/yql?
    q=select%20*%20from%20flickr.photos.info%20where%20photo_id%20in%20(
    select%20id%20from%20flickr.photos.search%20where%20woe_id%20in%20(
    select%20woeid%20from%20geo.places%20where%20text%3D%22london%22
    ))&format=json&diagnostics=false&callback=flickr

    Using this in the src attribute of a script tag and writing a few lines of Dom Scripting displays the photos.

    Is this limited to Yahoo?

    No, of course not. First of all you can use any data on the web as a source using the atom, csv, feed, html, json, microformats, rss and xml tables.

    Scraping HTML with YQL

    The HTML table is quite interesting as it allows you to get data from any HTML document, cleans it up by running it through HTML Tidy and then allows you to access parts of it using XPATH.

    This allows you for example to take a simple web site like this one about TV jokes and turn it into a wiget to include into other pages with a few lines of JavaScript.

    Extending YQL

    You can also add your own data by providing a simple XML schema called an open table. In this schema you need to tell YQL what the data endpoint is, what parameters are expected and what gets returned.

    Open tables also allow for an execute block which allows you to write JavaScript that will be executed by YQL on the server side using Rhino and has full e4x support.

    Using this we can turn the earlier example of the Flickr photos returned as a list into an open table and make it much easier to get Flickr photos in the right format:

    select&nbsp;*&nbsp;from&nbsp;flickr.photolist&nbsp;where&nbsp;text="me"&nbsp;and&nbsp;location="uk"&nbsp;and&nbsp;amount=20

    Using the JSON-P-X output this means we can simply use innerHTML to render out photos.

    If you want your tables to be available in YQL, all you need to do is to add to the open tables repository on GitHub. For more information, check out the YQL documentation.

    Building with Blocks

    The best thing you can do right now if you want to build a web application is using already tested and working building blocks. The Yahoo User Interface library is full of these as this is exactly how we build our own tools.

    For creating layouts with CSS without having to know all the hacks that browsers need you can use the YUI grids and if you are really lazy you can even use the WYSIWYG grids builder.

    Using the CSS grids and some of the YUI widgets it is very easy to build a working application that is tested across all the browsers defined in the graded browser support. Examples are this geographical Flickr search and a showcase to get information for Delhi.

    One thing that is really useful about the widgets provided in YUI is that they are all driven by custom events. That way you can extend and change their functionality without having to mess around with the code. Simply write an event listener for the custom event, add your functionality and prevent the original functionality.

    Another great benefit of YUI is the very detailed documentation and the hundreds of examples you can use to get you started.

    Wanna get super famous?

    The last thing I want to talk about today is the Yahoo Application Platform or short YAP. Using YAP you can build a web application using JavaScript, HTML and CSS and add it to the Yahoo Homepage, My Yahoo and in the future even more properties of Yahoo.

    You start at and develop your application to the largest part in your Applications Dashboard. Yahoo apps have two views: a small view which is more or less static (but allows for some Ajax) and a large view which gives you full access and much wider screen space. The small view is overlayed over the Yahoo page and will show Yahoo ads next to it. The large view can be monetized by you.

    One thing that can be a bit of a frustration about YAP when you go at it with a normal web development mindset is that not all CSS/HTML and JavaScript is allowed as YAP uses Caja to keep our applications secure. Therefore we’ve put together some Caja-ready code examples to get you on track.

    The easiest way to build YAP apps is by using the Yahoo Markup Language (YML) and YUI as YUI was re-written to be Caja compliant.

    If you want to take a look at what a YAP application looks like, check out the source of TweetTrans on GitHub. In essence it is a simple PHP API call using YQL and a YML interface to display the results using Ajax. No JavaScript involved as YML does that for us.

    You can install TweetTrans by clicking the following link: http://yahoo.com/add?yapid=zKMBH94k. This is also the way to promote your own applications (simply replace the application ID with yours) until the YAP application gallery is up and running.

    The more powerful way of promoting your application in Yahoo is to piggy-back on our social connections and you can do this by diving into the social graph API. The easiest way to do that is to use the social SDK also available on GitHub. Notice that the SDK will not work on a localhost – you need to run it inside the application dashboard or a Yahoo container on the homepage.

    Elevator pitches

    Yahoo User Interface Library – YUI

    YUI is the system that Yahoo uses to build its web sites. It is constantly tested to work for the largest amount of users, free, open source and covers everything from design patterns to out-of-the-box widgets. It is modular and you can use only what you need. You can either host it yourself or get it from a network of distributed servers.

    Yahoo Query Language – YQL

    YQL is a web service that allows you to mash-up any data on the web from various sources with a simple SQL-style language. You can filter the data down to what you need and you can convert the data with server-side JavaScript before returning it. Data providers can use YQL to publish an API on the web on top of Yahoo’s infrastructure and cloud storage.

    Yahoo Application Platform – YAP

    YAP is the Yahoo Application Platform which allows you to build applications that run on the Yahoo homepage and soon other properties. You can dive into Yahoo’s social graph to promote your applications and you can create highly secure web apps as YAP uses Caja to ensure code quality.

    On being cleverly lazy – my talk at the WebExpo in Prague

    Monday, October 19th, 2009

    I am currently in Prague, Czech Republic and gave a talk on re-use, professionalism and ease of development at the WebExpo.

    Webexpo Prague by  you.Webexpo Prague by  you.

    Today I am going to have a longer chat with the people at the University about accessibility and creating a WAI-ARIA enabled framework for web applications.

    For now, here are the slides of the WebExpo presentation and the audio recording of the talk. There was a video streaming, too, and it is up to Microsoft now to get this out as a recording.

    Slides

    Audio recording

    You can download on being cleverly lazy as an MP3 - 36MB or see other audio options on archive.org

    Notes / Transcript

    Today I will talk about the fact that being cleverly lazy will make you a better developer.

    First of all, I am Chris, a developer evangelist. This means I am a developer who evolved into a role where I can tell other developers how to have an easier life and tell my company what other things developers would want to make it even easier for them. I’ve written a developer evangelism handbook if you are interested in learning more about this role and why your company would benefit from having someone like me.

    Cleverly lazy

    Let me quickly define cleverly lazy here. Lazy is “I don’t want to do it” wheres cleverly lazy is “I don’t want to do that ever again so I do it right this time”. In development the difference between lazy and cleverly lazy is that lazy products do the job but are impossible to maintain whereas cleverly lazy products do the job, are easy to understand, extend and are built on a solid base.

    Evolving the web

    Our job as developers, designers, planners and organisers of web products is to evolve the web. We should have the chance to concentrate on building solutions that make people happy to use them rather than having to work out how one browser or another fails or how we convert data into a format the web technologies can display and understand.

    In order to achieve this, we need to be free to do new things rather than worrying about the past.

    The feature loop trap.

    One problem is that as developers, we stand in our own way. The biggest trap that there is for developers is the feature loop:

    • As developers we love to take a complex problem and tackle it. We love solving problems, that is why we become developers.
    • We then normally and quite quickly find a simple solution to the complex problem.
    • We then release the solution to the world and get feedback.
    • This is where it goes pear-shaped. We get stuck in a feature adding and feedback loop until the elegant, easy and simple solution becomes a complex one again.
    • Other developers will find our former simple solution and will think “hey, this is a complex problem, let’s make that easy!”

    This keeps us from evolving as web developers. We don’t develop the web, we fill it up with solutions to the same problem.

    My solution

    The next issue is that as developers we love to find solutions ourselves and not use other people’s information. We appreciate other people’s work but in the end we only trust ourselves to come up with the best solution ever.

    Short attention span

    The problem with finding and building own solutions is that we tend to lose interest in them really fast and then leave them behind as we already another complex problem to simplify. This leads to lots of half-finished solutions on the web that don’t get any love any longer.

    Unmaintained code is a security problem

    All the code that is not getting any love stays unmaintained on the web and becomes a wonderful attack vector for new security threats or vulnerability exploits. Nothing beats being up-to-date when you want to build secure systems.

    Things web developers must know to do their job.

    Looking at our job as web developers, here are the things you have to be aware of to build things that people can use.

    • The technologies involved
    • How browsers deal with these technologies and how they fail to support them
    • Security concerns and attack vectors
    • Usability and accessibility of the product
    • Internationalisation of our products
    • Performance concerns
    • Multiple platform support
    • Flexibility of the interface

    Browsers suck

    Browsers are the bane of our existence as they are unpredictable, unreliable, there are hundreds of them (in various configurations) and they are not being updated by people out there.

    That is a lot to know and deal with. The good news is that you don’t need to know all of it.

    Good developers are like librarians

    Librarians are great people. Instead of knowing everything that is in the library they know exactly where to find the information to solve a problem. As a cleverly lazy developer you should do the same.

    Build on a solid foundation

    Web development libraries came around for one simple purpose: Make our life as developers easier and our work as web developers predictable. They make browsers suck less and work around differences in browsers. They allow us to use web standards and get results instead of bugs to fix.

    Build with components

    If you build web applications and interfaces take a look around for what has already been created for you. Almost every library comes with widgets that are tested, work for everybody regardless of ability and can be changed to your needs. If you build your own component you are very likely indeed to forget some very necessary functionality that a collaborative product already had put in.

    Use a good debugging environment

    You can’t write great products without being able to know what is going on. Luckily enough nowadays we have great debugging tools like Firebug, Yahoo Profiler, JSLint and validators of all kind.

    Plan for extensions

    If you build a solution, plan a way so that people can extend it without having to change the central code. The central code should only have to change when there is a new browser or platform to be supported or there’s a security fix. Other than that people should be allowed to extend by listening for events or using other API hooks into your product.

    Read, use and write documentation

    If you build something, document it. There is no such thing as code that explains itself, that is a myth and an arrogant one at that.

    Use the web

    One of the really cleverly lazy things to do these days is to use the web to build your product rather than building your product and putting it on the web. Distribute your content on specialised systems like Flickr, YouTube, Delicious and LinkedIn and then put it together in a centralised CMS. That way your site will be easy to maintain and not vulnerable to a single point of failure.

    Use APIs

    To achieve this you need to use APIs. APIs can take a lot of your time to read up on and understand which is why it is a good plan to use YQL as an in-between to avoid going crazy on authentication and understanding what parameters go in and what data comes out.

    An example

    As an example, let’s take a look at a web site built solely from data on the web, using YQL and PHP.

    Thanks!

    Thank you, remember if you use what is out there you have time to be creative. I want us to write code most of our time, not to fix bugs.