Last week I visited Sky in London for a brown bag session during lunch. Their head of innovation, Paul Kane had asked me to come around and talk about API design, the history of YQL and in general tips and tricks how to get innovation and developer outreach working in companies and media. One part of the presentation was the following talk.

Here are the detailed notes:

Reaching Out

Strategies and ideas

• Third party developers make your product better
• Reaching them can be tough
• Here are some ideas and strategies that worked in the past

Third party developers are an amazing force to make your products better and more relevant to the users you want to reach. Reaching them can be a bit of a headache and today I am going to talk about some of the ideas and strategies that worked well in the past and some of them I’d love to see followed more consistently.

The importance of being open

• Opening your data and services multiplies your reach
• Developers are happy to use your products and test them at the same time
• You can not test for all environments, other people can

The first thing to be aware of is that being open to developers multiplies your reach and makes it much easier to create a solid product. With each developer using your APIs or data feeds you reach audiences you don’t have access to and have another developer looking over your product.

Developers are very happy to point out issues and in fact do report issues – users are not that vocal about these issues. This means that a lot of companies are happily polluting the web with terribly unusable products without being aware of it. You can’t anticipate all environments and use cases to test in – if you open your data to the world the developers using it can do that for you.

Building for web use

• Be Platform independent
• Be RESTful
• Release all documentation and code examples in a portable format

The first thing to make sure is that to achieve ultimate reach you should concentrate on building systems that are meant to be used on the web. This means first of all that your web service should be platform independent.

REST does this for you. If you build a system that gives back data over HTTP implementers can use a browser, cURL on the command line or any framework they want to get your information.

Your documentation, information about the API and code examples should be available in as easy as possible formats that are portable and independent of platform. HTML, PDF and zipped code does all this – you cannot expect people to download and install an SDK like a piece of software – we are busy developers and hate polluting our machines with unnecessary installs.

Thinking of endpoints

• Good URL structures
• Easy to understand method names and parameters
• Versioning of APIs

The success of a REST based API stands and falls with the quality of its endpoints. Make sure that your URLs are human readable and make sense. A good web service should be navigable in the browser bar and I shouldn’t have to read up the docs to understand where I want to go.

Keep your method names and parameters short and to the point – nobody likes long URLs as there is no tab completion in the browser bar (yet).

One thing that is terribly important is to version your API in the URL. If you find a bug or you need to rename something in a newer version as new dependencies come up you don’t want to break implementations that were based on an older version. A v2,v3,v4 in the URL is not long but prevents that from happening.

Thinking of output

• Provide data that people need – a lot of data is good but don’t add information that only makes sense internally.
• Provide data formats that people can use: xml, rss, json
• Allow for callback parameters for json to use as JSON-P
• Maybe allow for object transportation / ID for JSON-P calls
• Allow people to turn off diagnostics and filter down to specific needs

These are very important points. As a general rule it is a good idea to provide as much data as possible in your APIs. However if this data is of no use to a third party developer or there is simply no documentation for the data then it shouldn’t be in there as it simply distracts from the necessary information. The Yahoo music API for example has a catzilla and hotzilla ID in the results – I have no clue what that is.

The data format question is pretty easy, XML for a full-fat result, RSS to really make it easy for people to use the data and JSON for people like me who don’t like to wade through XML with namespaces.

JSON-P is JSON with a callback parameter that allows the end user to use the data immediately in JavaScript without having to use a proxy to reach your content. The callback ID is an idea I’ve mentioned before. The problem with using JSON-P and generated script nodes is that you can never be sure in which order several calls are returned. Therefore it is a good idea to allow for an ID to be sent and returned with the call so I can match up data with request in the callback method. This could be as simple as repeating the request terms in the result set, but a real transaction ID would make it even more handy.

Diagnostic parts of API calls can help me find out what went wrong or what was going on but it is good when you can turn it off. YQL for example offers that option, check this call vs. this one – the difference is the diagnostics=false parameter. In general it is a good idea to allow developers to filter down the amount of data being sent back (f.e. with an amount parameter) as there are environments where really every byte counts.

Providing easy access

• For data APIs that are read only, don’t bother with authentication – just use a developer key
• Have sandboxes where developers can play without having to sign up

Nothing is more annoying than having to authenticate before even getting to see the kind of content you can access with your API. Therefore it is a good idea to keep the pain of authentication as low as possible.

A great idea is to cluster the access to different developers or have a no-frills access version. Vimeo are doing a great job offering different APIs and so is Yahoo Maps. Flickr is providing an API explorer and the Guardian API also has a console for running and filtering down requests. The Rolls Royce of course is the YQL console.

Documentation

• Provide proper documentation that is quick to read and easy to navigate
• Offer cheatsheets and quick introductions
• Have a documentation team – developers are not writers and are too close to the product to actually write readable docs

None of these need much more explanation. Simply be aware that documentation is something you shouldn’t take lightly as developers go to docs when they get stuck and when they encounter only more frustration and hard to understand information there you are not likely to be a big success.

Personally I don’t think you should bother localising documentation unless you need to provide different content to different markets and have to explain developers why some content is not available to them. Let bloggers do the advertising in the local markets for you.

Demo Code

• Provide demo code and SDKs (if you really must)
• Have excellent demo code – not ‘look, just two lines of code’
• Offer different language demos

Demo code is a very powerful tool and shows the quality of your development team. Of course every developer can wade through quickly put together code and get something done, but your code should entice developers to use your systems and create excellent solutions.

Your demo code should not have any security holes, accessibility omissions or simply bad code practices. This code is your advertisement to other developers – it should be easy to copy and paste it and hit the ground running but you should not leave a trail of security concerns and global variables behind.

If you provide copy and paste examples make sure they work outside your environment – all links to dependencies should be absolute and not relative (starting with http at best) so that you don’t get complaints about code not working. I found it best to list a whole example and the explain it chunk by chunk what is going on.

Consistency is key here – keep to an agreed code standard and you won’t confuse implementers. Other than that your code should be customizable as easy as possible – use config objects to allow for that.

Be responsive

• Have a mailing list / forum that is monitored
• Keep your eyes out for things people do with your code (blogsearch, twitter)

Developers will do a lot for you and nothing gives them more satisfaction than getting their problems answered by an expert. Have a mailing list or forum with your expert developers or at least people that can ask the right person. Monitor the buzz that is going on and chime in to give praise or set things straight. There is nothing cooler than seeing your tweet about an implementation you’ve built with a certain API getting re-tweeted from an official account or by experts connected with this API.

On the other hand there is nothing more deadly than a real implementation problem remaining unanswered for days. Be visible, be there for people and you’ll have happy users.

Invite people to play

• Invite selected people for a private beta – this is amazingly powerful as people’s integrity will work for you – or you get great feedback what to change
• Give people a chance to build “official mashups” before release to show on the site and in the press conference.
• Maybe launch with a developer day

This is pretty important. Releasing an API is one thing, having other people that the whole audience trusts in their decisions advocate for you is a much bigger impact. Open up betas to some people beforehand, have a handful of API keys upfront for people to play with and have keys that people can give out on twitter or their blogs – all of this will have quite an impact.

The Guardian did a great job with this when they released their APIs. It also works for product launches, as spotify has shown. Having people build official mashups with official blog coverage also allows these people to write “making of” articles and blog posts which show tips and tricks you might not have thought of before.

You can take this idea further – have some undocumented but cool new features every few weeks or so (or previews of upcoming features) and leak them out to twitter or on developer lists. A lot of developers are very happy to be the first to show the world that your API is working on something new and will try to hack around the non-support by writing own docs.

Collaborate with other API providers.

• Reach out to other API providers
• Take part in groups dealing with the same topic
• Create an open YQL table, add your data to gnip

Last but not least remember that you are not alone. We all release APIs and can learn from each other’s mistakes and gains. This presentation is living proof for that and I am not alone in reaching out to companies who want to join the ranks of those who build and maintain the web of data. An open YQL table is written pretty fast and gives you a chance to let developers play with your data without having to worry about limitations and caching for now – the YQL access restrictions and server does that for you.

I just looked through the API of Microsoft’s new Bing search (which is really a re-branding of the live search – also, switch to “low bandwidth view” to be able to use the docs much more smoothly) and I found an interesting step in protecting code from throwing errors.

When you provide a JSON output for developers it does make sense to also allow for a callback parameter. That way your code can be used in script nodes without having to use any backend at all. If you for example provide an API to return the names of the Beatles you could have a data endpoint like getBeatles and a parameter for output format:

http://example.com/API/getBeatles?output=json

The return data then will be:

{

“members”:

[

	
‘Paul’:{ ... more info … },

‘Ringo’:{ ... more info … },

‘John’:{ ... more info … },

‘George’:{ ... more info … }

]
	
}

This I cannot use in JavaScript without hacks. I’d need to eval (or to be safe JSON parse) the results and with conventional Ajax I cannot reach data outside my domain. To make JSON work as easy as possible you can provide a callback parameter.

http://example.com/API/getBeatles?output=json&callback=eleanorRigby

This should wrap the code in a function call which means the output is already evaluated and the user has to define a callback method to read that information. The output would be:

eleanorRigby({

“members”:

[

	
‘Paul’:{ ... more info … },

‘Ringo’:{ ... more info … },

‘John’:{ ... more info … },

‘George’:{ ... more info … }

]
	
});

If I define a function like eleanorRigby(o){} o will be the object returned from the data and I can use it immediately:

Now the issue there is if eleanorRigby is not defined it throws an error.

The Bing API is the first instance where I have seen that they worked around that as the output is this:

if(typeof eleanorRigby  'function') eleanorRigby(

{

	
"SearchResponse":

{
	
Version",

"Query":

{
	
SearchTerms hard day's night"

},

"Translation":

{
	
"Results":

[
	
{TranslatedTerm harten Tag-Nacht "}

]
	
}

}
	
} /* pageview_candidate */);

I have no clue what the /* pageview_candidate */ is about and frown upon omitting the {} of the if statement, but I must say I do like this.

The issue is now that errors are silent, which might make debugging a pain. Maybe a better option would be to have an error case where the API writes out an error to the console when the callback is not defined:

if(typeof callback = ‘function’) {

callback(... data … );

} else {

if (typeof console!==’undefined’ &&

typeof console.log !== ‘undefined’){

console.log(‘Error: Callback method not defined’);

}

	
}

All in all an interesting approach though!

As part of my talk on YQL for Open Hack London I’ve thought of an easy way to get Flickr photos that you are allowed to display in your products and hacks.

The wrapper API getFlickrBy works around the somewhat convoluted data returned from Flickr’s API methods and uses YQL to cut the information you want down to the bare necessities. Furthermore the API only returns photos that are licenced with Creative Commons’ “By” license to avoid you using photos you have no right to use (which is a big thing with the Flickr crowd).

The API endpoint is:

http://isithackday.com/api/getFlickrBy.php

You have several parameters to play with:

location

The location you want photos of as defined in the Yahoo Geo API

search

A word you want to search for

format

The format of the returned data, XML, HTML or JSON - preset is XML

callback

A name of a JavaScript function call to wrap the JSON data in, in case you want to use the API in a script node

amount

The amount of photos returned, max is 100 – preset is 20

size

the photo size, “s” for 75×75px, “t” for thumbnail, “m” for medium, omit to get big size

The API returns only what you need: the image title, the owner, the url of the image and the link to Flickr. If you use html as the output format it returns an HTML list of linked images. For output demos and explanations simply call the API without any parameters

Here are some examples how to use the API:

• Get 5 photos from Paris,France as JSON calling the function foo()
• Get 20 photos from Ankara as HTML
• Get 20 photos from London as XML
• Get 20 photos of bridges in munich as XML
• Get 10 photos of kittens json calling the function meow()

Things that made me happy this morning:

• May the fourth be with you – it is Star Wars Day!
• The YQL console now lists community open tables from github
• The Bay Area’s public transport system has a very nice API
• A nice photo series of old game controllers and consoles as X-Rays
• James Broad has a good step by step tutorial how to use iSight for a timelapse video
• PopeyesJokes on YouTube shows that you can create a lot of UGC with a simple video editing suite and a lot of enthusiasm
• What not the crochet tells you exacty that.
• The iSnort is an iPhone app to either make you look properly Hollywood, or to get you kicked out of any club. Also check the disclaimer on the bottom of the page.
• A great list of code comments
• Jacob Kaplan-Moss has info on the worst REST practices to avoid
• Wedesigner Depot praises 1,2,3 as a good usability practice
• Some very dedicated hackers found 48,000 lbs of 70mm tape – the only copy of extremely high-resolution images of the moon in an abandoned McDonald’s and started digitizing them.
• An art student in the UK shows how to make a car disappear. Take that James Bond.

Distributing your information all over the web has become a common practice over the last few years and it makes a lot of sense. By covering lots of distribution channels you can reach various audiences and get comments and feedback from them.

You also make yourself independent of a single online resource – if your server is unavailable your data is still around. I could go on with the benefits of distribution (after all I’ve written a book on the subject) but let’s take a look on the flipside: by spreading your data all over the web you also spread yourself thin and you want a single resource to act as your main URL.

People have been telling me for a while that they don’t have time to find all the things I leave across the web and that they are wondering if there’s a single entry point. One Solution is FriendFeed but you want to be able to style your “online profile” more than that.

This is where YQL comes into the equation. Using YQL, a YUI CSS grid, a few dozen lines of PHP and a bit of CSS I managed to pull together My online portfolio http://icant.co.uk and you can do this as easily. The following screencast shows you how it is done:

You can also download a readable version of the screencast for ipods.

Since I put together the screencast (which was a bit hurried as I needed to catch a flight) I’ve updated the idea with yet another script that scrapes the resulting HTML document to create an RSS feed of all my data on the web.

• Check out the source of index.php
• Check out the source of feed.php

Using YQL has a few more benefits than reading all the different sources yourself and mixing them up: the results are cached for you, YQL’s connection to the web is very much likely to be faster than yours which makes the fetching process easier and you have full control over what’s happening as YQL output gives you diagnostics information.

I’ll talk more about in YQL in various talks in the nearer future, and there are even more interesting changes to the system itself around the corner. Stay alert for awesome updates.