Researching something on the web can be pretty annoying. Search engines get better every year, but there is a whole world of social sites that are not indexed. For example if I search for a nice photo of a red panda I use Google image search. If I want to use this photo later on I am better off using Flickr or Picasa and see what license the photo is.

Yahoo’s researchers had the same problem which is why they assembled all the social updates in one XML feed – the Yahoo! Firehose. This, in contrast to other Yahoo APIs also comes with commercial terms and conditions and is available through YQL. In terms of data, the Firehose aggregates a lot of different sources:

Yahoo! 360, AOL, Bebo, Blogger, Bloglines, Digg, Diigo, Goodreads, Google, Google Reader, Last.fm, Ma.gnolia, Movable Type, Netflix, Pandora, Picasa, Pownce, Seesmic, Slideshare, SmugMug, StumbleUpon, ThisNext, TravelPod, Tumblr, Twitter, TypePad, Vimeo, Vox, Webshots, Xanga, Yelp, YouTube, Zooomr, Yahoo! Avatars, Yahoo! Buzz, Yahoo! Profiles, Wisteria, Yahoo! Answers, Yahoo! Shopping, Yahoo! Autos, Bix for Yahoo!, Yahoo! Bookmarks, Yahoo! Briefcase, Yahoo! Calendar, Yahoo! Classifieds, Delicious, Yahoo! Family, Yahoo! Sports, Yahoo! Finance, Flickr, Yahoo! Food, Yahoo! Games, Yahoo! Geocities, Yahoo! Green, Yahoo! Greetings, Yahoo! Groups, Yahoo! Health, Yahoo! Hotjobs, Yahoo! Kids, Yahoo! Local, Yahoo! Movies, Yahoo! Music, MyBlogLog, Yahoo! News, OMG! from Yahoo!, Yahoo! Personals, Yahoo! Pets, Yahoo! Status Updates, Yahoo! Guestbook Comments, SearchMonkey from Yahoo!, Yahoo! Shopping, Yahoo! Sports, Yahoo! Tech, Yahoo! Travel, Yahoo! TV, Yahoo! Video.

You can do the data junkie part and use it in the YQL console:

This can be annoying though, especially as you cannot see the photos and videos. This is why I put together a research interface on top of the Yahoo Firehose:

You can see the research interface in action here but more importantly, the source code of the interface is available on GitHub which means that you can host it yourself – for example behind a firewall or make it part of your Intranet.

For a local install you need to sign up for a developer key, edit the keys.php file, put all the files up on your PHP enabled server and you are done. If you get stuck you can get help on the YDN Forums.

Notice that I am keeping the state of your last search by storing it in local storage when your browser supports it – this can be useful for larger searches.

Things that made me happy this morning:

• The data.gov.uk release yesterday was a great event. UK House Prices got applauded, I sat next to Tim Berners-Lee and I managed to pester people from the government to have me come around and showcase YQL!
• Gource is a very sexy visualisation of Git commits, forks and so on.
• This Saturday Yahoo will host CrisisCamp Haiti
• Metal Slug sprites is a collection of all the sprites in all their pixely goodness. Me having spent a lot of time pixeling stuff (notice the browser support info – this is how old this is), I am impressed.
• Jennifer Aniston and Matthew Perry show us how Windows 95 works.”Look Matthew I’m computing!”
• Want to buy a jumbo jet? Comes with slight water damage – yes, this is the Hudson River one.
• Tickery shows you which other people two people on Twitter follow.

Displaying photos from Flickr can be daunting. The API needs authentication and the RSS, JSON or LOLcode output is very limited. The way around is using YQL and its Flickr tables. That way it is pretty easy to search Flickr:

select * from flickr.photos.search where text=”panda”

Try out the Panda search in the YQL console.

The output format has a lot of information in there, but sadly enough, not all. For example the real name of the owner or the description is missing. Therefore you need to go through yet another Flickr API to get the whole data set:

select * from flickr.photos.info where photo_id in(

select id from flickr.photos.search where text=”panda”

)

Guess what? You can also try this more detailed query in the console.

I’ve shown before how easy it is to display Flickr Photos retrieved that way:

The issue with that though is that it uses JavaScript and JavaScript may be turned off (think Blackberries). Of course you can do the same thing in PHP but I’d wager to say more people to JavaScript than PHP these days.

The main issue is that Flickr returns the photos in a pretty weird format and that you need a script like the one above to turn it into a simple HTML list.

The good news is that YQL with the execute command allows you to embed JavaScript in your open tables. That way you can write a table that does all the necessary transformations and returns the data as a simple list for immediate use:

select * from {table} where location=”london,UK”
Christian Heilmann
http://www.wait-till-i.com/2009/11/01/getting-a-list-of-flickr-photos-by-location-andor-search-term-with-a-yql-open-table
Searches Flickr by location and/or search term and returns an HTML list that can be immediately used in a mashup.
















var amt = amount || 10;

var query = ‘select farm,id,secret,owner.realname,server,title,’+

‘urls.url.content from flickr.photos.info where ‘+

‘photo_id in (select id from flickr.photos.search(‘+

amount + ‘) where ‘;

if(location!==null){

query += ‘woe_id in (select woeid from geo.places where text=”’ +

location+’”) and ‘;

}

	
query += ’ text=”’ + text + ‘” and license=4)’

var x = y.query(query);

var out = 
;

for each(var cur in x.results.photo){

var li = 
;

var a = ;

a.@[“href”] = cur.urls.url;

var img = ;

var url = ‘http://farm’ + cur.@farm + ‘.static.flickr.com/’ +

cur.@server + ‘/’+cur.@id + ‘_’ + cur.@secret +

‘_s.jpg’;

img.@[“src”] = url;

img.@[“alt”] = cur.title;

a.img = img;

li.a = a;

out.li += li;

}


	
response.object = out;

]]>

You’ll notice that while the E4X support is very powerful, it can be a bit confusing to look at on first sight. Once you got your head around though it becomes much cleaner that way.

You can use this table like any other open table via the use command in YQL:

use “http://github.com/codepo8/yql-tables/raw/master/flickr/flickr.photolist.xml” as flickr;

select * from flickr where text=”me” and location=”uk” and amount=20

try it in the console.

I’ve wrapped one more API in there – the Yahoo Geo API to determine a place from a name should you want to search by location. All in all you have three parameters in this open table – all of which are optional:

• text – the search text
• location – the geographical location
• amount – the amount of photos to be returned

If you look at the table source, you can also see that I hard-wired the license of the photos to 4 which is CC-BY. So if you link the photos back to Flickr you both satisfied Flickr’s terms and the original photographer’s.

Now, the easiest way to use this output is by using YQL’s JSON-P-X output format. This is XML with a callback which returns a JSON object with the HTML as a string instead of a convoluted JSON object. See the JSON-P-X output here.

That way you can easily use it in JavaScript:

And also in PHP:

$url = ‘http://query.yahooapis.com/v1/public/yql?q=use%20%22http://github.com/codepo8/yql-tables/raw/master/flickr/flickr.photolist.xml%22%20as%20flickr;%20select%20*%20from%20flickr%20where%20text=%22me%22%20and%20location=%22uk%22%20and%20amount=20&format=xml&diagnostics=false’;

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, $url);

curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

$output = curl_exec($ch);

curl_close($ch);

$output = preg_replace(‘/.*
/’,’
‘,$output);

$output = preg_replace(‘/
.*/’,’
‘,$output);

$output = preg_replace(‘//’,’‘,$output);

$output = preg_replace(‘//’,’‘,$output);

echo $output;

?>

You can see both in action on the demo page.

So by using YQL open tables you can not only access complex APIs with YQL, but you could also write complete mashups in JavaScript and have them executed in a safe environment on Yahoo’s servers. Your end of the mashup is simply the display which could be a form that works with Ajax when JavaScript is available and renders a static page in PHP (or whatever other server-side language) when JavaScript is turned off. You only need to do one HTTP request – the rest is executed and cached on the YQL server farm – everybody wins.

Things that made me happy this morning:

• The Developer Evangelist Handbook is going nuts. 10300 hits yesterday, so far 12400 today.
• Amazing photos of naked Obama and unicorns – in case you wondered if the web is sane by now.
• Reconstructing 3D models of cities from Flickr photos is a pretty amazing research/tool/design.
• Proof that the quick brown fox really jumps over the lazy dog%
• CSS Prism is a tool to analyse CSS files and show the colours used in them
• Snook blogged about CSS text rotation and dug up the old IE transitions as a fallback – they are terrible though.
• 100 things our kids may never know about is a cool list of tech nostalgia
• BootXChanger allows you to change the Apple logo when your computer boots up
• Gheat is a pretty cool tool to overlay heatmaps on Google Maps

Things that made me happy this morning:

• Keep your meetings interesting with the Meeting Money Ticker
• The iPhone 3.0 firmware release now allows us to access the JavaScript Geolocation API
• A surprisingly thorough article on TechCrunch about the Twitter crack some time ago
• An interesting story about a husband finding the photo of his wife as a hot single in his area in a Facebook advertisement – better check how your photos are used.
• A very thorough decision chart on whether to use a Flash intro or not
• A great new web site Geekspeakr tries a very nice thing – “Connecting tech women speakers with event organisers”.
• Bedjumping is something I have thought of many a time on my travels, too
• If you want to develop for the palm pre – here are some Palm Pre PSD design stencils
• Pretty cool robots recycled from scrap metal
• Douglas Coupland is coming to London to read from Generation X.
• Runpee is a web site and now iPhone app that tells you when you can take a break in a movie to use the toilet – and tells you the plot parts you miss. Also wait for the logo animation.
• Flickery seems to be a great replacement for the Flickr uploadr.