Christian Heilmann

Posts Tagged ‘JSON’

The annoying thing about YQL’s JSON output and the reason for it

Wednesday, September 22nd, 2010

A few days ago, Simon Willison vented on Twitter about one of the things that annoyed me about YQL, too:

Battling YQL's utterly horrible JSON output, where a response with one result is represented completely differently from a response with two

The interesting thing about this is that it is not a YQL bug – instead it is simply a problem of generic code and data conversion.

The problem with YQL JSON results

Let’s have an example. If you use YQL to get for example the geo information about “London” you get several results in XML:

select * from geo.places where text=”london”:

XML output with multiple places

If you use JSON as the output format this becomes an array:

JSON output with multiple places

Cool, so you can loop over, right? No, cause when there is only one result, we have a different situation. Instead of being an array, place becomes an object:

select * from geo.places where text=”london,uk”:

Single results in JSON become an object instead of an array

This is what ailed Simon as it means you need to write a function that works around that issue. For example in the Geoplanet Explorer, I use a render() function to show each place result and I use the following to work around the JSON output issue:

function renderlist($p){
foreach($p as $pp){
$o .= render($pp);

} else{
$o = render($p);

return $o;

In JavaScript you’d have to check the length and either show the results directly or loop over the results.

Generic conversion vs. creating a single API

This is a problem when you convert XML to JSON - as the resulting XML from the GeoPlanet API doesn’t have a places element with place in it but instead repeats the place element for every result the JSON parser must make a decision: if it is only one element, this becomes a property of the results object. If there are more than one place element it becomes a property that is an array (as you cannot repeat the same property name).

If you build your own API and you know the format of the different results you can force this to be an array even when there is only one result. If you do not know the XML schema there is no way of assuming what should be an array and what should not be an array. So the generic solution for XML to JSON conversion of an unknown XML with 1 to n results would be to always create an array. That way you couldn’t get to place.admin1.code for example but you’d always have to do place[0].admin1[0].code[0] – not really a sensible thing to do.

As YQL is an open system and the XML results are provided by anyone in their open table definition the only way around I could think of is to tell YQL in the table that some elements should always be forced to become arrays.

Do you have a solution?

Stackoverflow DevDays – My “slide deck”

Wednesday, October 28th, 2009

I am about to leave for the London Stackoverflow DevDay, and speakers were asked to keep their slides to a minimum of two pages and then show code instead. Now, here are the “slides” of my talk at the end of the day today:

presentation = {
intro:’oh, hai’,



I will add more and more bookmarks on delicious with the tag stackoverflowlondon while presenting, so you can check what I will show there.

If there is no sturdy internet connection, I will have to improvise.

TTMMHTM: Evangelist Handbook, Billboard charts API, collaborative editing, IE6 bashing, pretty JSON, fancy fast food and terrible bugs.

Friday, July 24th, 2009

Things that made me happy this morning

Another interesting YQL feature: XML with callback (JSON-P-X)

Thursday, July 9th, 2009

Yesterday’s announcement of YQL now supporting insert, update and delete somehow overshadowed another interesting new feature: XML with callback!

XML with callback?

On first glance having a callback parameter on an XML output doesn’t make any sense. The normal use case of callbacks is to turn JSON data into JSON-P so that you can use the data immediately in JavaScript. The reason for XML with callback is to make it easier to use the same data when getting data from the web. Say I want to use YQL to get the images and links of the people I follow from twitter. The YQL statement is:

select * from html where url=””
and xpath = “//div[@id=’following_list’]”

The XML output of this call is a pretty hefty XML document with all the HTML as an XML node structure.

The JSON (and JSON-P output) is even worse as it gives you a structure of all the elements and their attributes as properties of nested objects:

div: {
id: “following_list”
span: [

a: {
href: “/jemimakiss”
hreflang: “en”
rel: “contact”
title: “Jemima Kiss”
img: {
alt: “Jemima Kiss”
height: “24”
src: “http://s3.amazonaws.c…glasto_mini.jpg”
width: “24”


//... and so on …

Converting this back into HTML could be quite an annoying job – not to say slow. This is why YQL now offers the callback parameter for XML. The JSONP-X output as it is called in the YQL changelog makes this task a lot easier by returning a JSON object with the XML as a string:



Jemima Kiss [...]



This makes it dead easy to render the results back as HTML:

Nice? I think so!

Safer JSON-P? An interesting feature of the Bing API

Monday, June 15th, 2009

I just looked through the API of Microsoft’s new Bing search (which is really a re-branding of the live search – also, switch to “low bandwidth view” to be able to use the docs much more smoothly) and I found an interesting step in protecting code from throwing errors.

When you provide a JSON output for developers it does make sense to also allow for a callback parameter. That way your code can be used in script nodes without having to use any backend at all. If you for example provide an API to return the names of the Beatles you could have a data endpoint like getBeatles and a parameter for output format:

The return data then will be:


‘Paul’:{ ... more info … },
‘Ringo’:{ ... more info … },
‘John’:{ ... more info … },
‘George’:{ ... more info … }


This I cannot use in JavaScript without hacks. I’d need to eval (or to be safe JSON parse) the results and with conventional Ajax I cannot reach data outside my domain. To make JSON work as easy as possible you can provide a callback parameter.

This should wrap the code in a function call which means the output is already evaluated and the user has to define a callback method to read that information. The output would be:


‘Paul’:{ ... more info … },
‘Ringo’:{ ... more info … },
‘John’:{ ... more info … },
‘George’:{ ... more info … }


If I define a function like eleanorRigby(o){} o will be the object returned from the data and I can use it immediately:

Now the issue there is if eleanorRigby is not defined it throws an error.

The Bing API is the first instance where I have seen that they worked around that as the output is this:

if(typeof eleanorRigby 'function') eleanorRigby(



SearchTerms hard day's night"


{TranslatedTerm harten Tag-Nacht "}


} /* pageview_candidate */);

I have no clue what the /* pageview_candidate */ is about and frown upon omitting the {} of the if statement, but I must say I do like this.

The issue is now that errors are silent, which might make debugging a pain. Maybe a better option would be to have an error case where the API writes out an error to the console when the callback is not defined:

if(typeof callback = ‘function’) {
callback(... data … );
} else {
if (typeof console!==’undefined’ &&
typeof console.log !== ‘undefined’){
console.log(‘Error: Callback method not defined’);


All in all an interesting approach though!