Christian Heilmann

Posts Tagged ‘yql’ and displaying your words in an HTML document using YQL

Tuesday, January 4th, 2011

Whilst I am still not quite sure why, I signed up for, a service where people can describe you in three words. My results range from expected over disturbing up to WTF, so that’s all good.

Now, has no API as far as I can see and in the admin section there is a nice list of 10 most used words. To work around that I thought I use YQL - the statement is pretty easy:

select * from html where
url=”” and
| unique(field=”strong”)
| sort(field=”yahoo:repeatcount”)
| reverse()

You can Try in console to see that it does what you ask it to – get the words, make them unique and sort them by frequency.

Add a dash of JavaScript and you have a list you can display in your pages:

display of the list of words

You can see the script in action here or you can also get the source on GitHub.

Using YQL sensibly – my first talk at YUIConf2010

Tuesday, November 9th, 2010

I am currently at YUIConf in Sunnyvale, California to learn lots about YUI3, Node.js and YQL straight from the horse’s mouth. Yesterday I gave my first talk on using YQL in a sensible manner – coming from using webservices in JavaScript up to using localstorage to speed up your apps and keep state for your end users.

The Slides

The Audio

Now I need to write my second talk for later on – stay tuned for that one.

A bit of harmless Friday fun – what is your Ninja name?

Friday, October 29th, 2010

Going through my RSS feeds this morning I found this Ninja name translation table:

Ninja translation table

I thought that was a perfect candidate for a YQL table, so I made one. So find out your Ninja name in the YQL console.

Conversion was easy – simply convert the table to a hash, check that the user has not entered any non-alpha characters, lowercase and loop over the different characters with a lookup. You can see the source of the table on github

Yours, mirishikiari.

The book that never was – the why of YQL

Wednesday, October 27th, 2010

YQL is great, it is a technology that turns the web into a database and allows you to mix and match and filter before writing your first line of code. It also allows you to release an API without any infrastructure, knowledge of authentication and access control. In essence you can use Yahoo’s server infrastructure and processing power to unleash the awesome of the web into your products or the awesome of your data on the web.

There is a truckload of documentation, forums and blog posts about YQL on the YDN web site but as it is people love books so I was asked to write a YQL guide for Yahoo Press.

Specifically I was asked to write a demo chapter for the book to pitch to O’Reilly over Christmas, which I did. I then waited for the paperwork to be signed off and so on and time came and went until I was offered to write the YQL Guide for O’Reilly together with my colleague Tom Hughes-Croucher (as he had also sent in a proposal for a YQL book). We agreed on some collaboration and then Tom moved on to write another book.

As I am leaving Yahoo and yet have to see a contract about this book to sign, I announced that I am not going further with this project. Being the techno-hippie that I am though I thought it would be a waste to not give the first chapter to the world, so here it is:

You can read the chapter online, download the PDF or browse the “source” on GitHub in case you want to translate or quote it. I licensed it with Creative Commons so go nuts!

The annoying thing about YQL’s JSON output and the reason for it

Wednesday, September 22nd, 2010

A few days ago, Simon Willison vented on Twitter about one of the things that annoyed me about YQL, too:

Battling YQL's utterly horrible JSON output, where a response with one result is represented completely differently from a response with two

The interesting thing about this is that it is not a YQL bug – instead it is simply a problem of generic code and data conversion.

The problem with YQL JSON results

Let’s have an example. If you use YQL to get for example the geo information about “London” you get several results in XML:

select * from geo.places where text=”london”:

XML output with multiple places

If you use JSON as the output format this becomes an array:

JSON output with multiple places

Cool, so you can loop over, right? No, cause when there is only one result, we have a different situation. Instead of being an array, place becomes an object:

select * from geo.places where text=”london,uk”:

Single results in JSON become an object instead of an array

This is what ailed Simon as it means you need to write a function that works around that issue. For example in the Geoplanet Explorer, I use a render() function to show each place result and I use the following to work around the JSON output issue:

function renderlist($p){
foreach($p as $pp){
$o .= render($pp);

} else{
$o = render($p);

return $o;

In JavaScript you’d have to check the length and either show the results directly or loop over the results.

Generic conversion vs. creating a single API

This is a problem when you convert XML to JSON - as the resulting XML from the GeoPlanet API doesn’t have a places element with place in it but instead repeats the place element for every result the JSON parser must make a decision: if it is only one element, this becomes a property of the results object. If there are more than one place element it becomes a property that is an array (as you cannot repeat the same property name).

If you build your own API and you know the format of the different results you can force this to be an array even when there is only one result. If you do not know the XML schema there is no way of assuming what should be an array and what should not be an array. So the generic solution for XML to JSON conversion of an unknown XML with 1 to n results would be to always create an array. That way you couldn’t get to place.admin1.code for example but you’d always have to do place[0].admin1[0].code[0] – not really a sensible thing to do.

As YQL is an open system and the XML results are provided by anyone in their open table definition the only way around I could think of is to tell YQL in the table that some elements should always be forced to become arrays.

Do you have a solution?