Decreasing the server load of Keywordfinder with YQL

Back in the days when Yahoo BOSS came out, I built Keywordfinder to show off the “keywords” functionality of BOSS. BOSS returns you keywords people entered for a certain web site in its results and I used that to list the most popular search terms for a term you enter.

Now, yesterday I got an alert by my host that Keywordfinder is exceeding my GPU limit and checking the logs I realized that it is heavily hit by users from South America. So I decided to add a JavaScript layer to the app that would take away some of the traffic and a lot of the computation from the server.

I could have just replicated the PHP functionality in JavaScript but I thought it is more beneficial for everybody to turn the functionality into a YQL table and use Yahoo’s servers to do the heavy lifting for me.

You can see the keywordfinder open table on GitHub and it will be soon replicated over to datatables and show up in the YQL console. For now you can apply it using the “use” command:

use "http://github.com/yql/yql-tables/raw/master/keywordfinder/keywordfinder.xml" as keywordfinder;
select * from keywordfinder where term="oil spill";

try it in the console or see the results as XML

The results right now are: oil, oil spill, spill, the Gulf of Mexico, BP, the Gulf, BP oil, Oil Spill, Deepwater, gallons, Gulf of Mexico, Gulf oil, Horizon oil, crude oil, Gulf, Gulf Coast, Gulf Oil, oil rig, birds, Mexico oil.

The table returns the search term, the region, the language and the terms. It sanitizes all of them for HTML display. By default, the table uses English as the language and USA as the region. If you provide a different language and region you get other results:

use "http://github.com/yql/yql-tables/raw/master/keywordfinder/keywordfinder.xml" as keywordfinder;
select * from keywordfinder where term="fussball weltmeister"
and language="de" and region="de";

try it out in the console or see the result as XML.

I think it is a nice example how you can speed up an app and help others by adding the functionality to YQL. Enjoy.

Tags: BOSS, keywordfinder, keywords, opentable, yql