• Posts Tagged ‘geo’

    Adding map links and a small map to any text using JavaScript – addmap.js

    Friday, January 29th, 2010

    As part of a larger article, I am currently building some tools that use geographical information. The first one is a pure JavaScript solution to link locations in a text to Google Maps and to show a small map under the text. You can see it in action by clicking the screenshot below.

    Analyse text and add a map with its locations in pure JavaScript by  you.

    All you need to do to use the script is get your own Google Maps key and embed the script in the page you want to analyse, giving it the ID of the main text element as a parameter:

    <script src="http://github.com/codepo8/geotoys/raw/master/addmap.js"></script>
    <script>
    addmap.config.mapkey = 'YOUR_API_KEY';
    addmap.analyse('content');
    </script>

    You can customise the look and feel by writing your own CSS for the generated HTML and setting the addmap.config.width and addmap.config.height properties to resize the map.

    Under the hood here is what happens:

    The code of addmap.js is available on GitHub.

    Finding the current location by IP and with the W3C Geo API

    Monday, January 25th, 2010

    I was always fascinated by those ads that show you “hot contacts in {city}” not because of the hot contacts, but because they always changed to where I was at the time.

    I was also quite interested in the fact that they were always a bit off (again, the city, not the contacts). As I like to find things out, here’s a demo of just how far the location of these ads can be off:

    The difference between IP location and Geo location by  you.

    You can see your results by visiting the demo page and allowing it to get your location if you use a browser that supports the W3C geo location API.

    Here’s how this works:

    You can guess the location of a user by their IP and Rasmus Lerdorf wrote a nice API to do that at http://geoip.pidgets.com/. Using that, you can read the IP in PHP and call the API with cURL:

    if ($_SERVER['HTTP_X_FORWARD_FOR']) {
    $ip = $_SERVER['HTTP_X_FORWARD_FOR'];
    } else {
    $ip = $_SERVER['REMOTE_ADDR'];
    }
    $url = 'http://geoip.pidgets.com/?ip='.$ip.'&format=json';
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    $output = curl_exec($ch);
    curl_close($ch);
    $data = json_decode($output);
    $lat = $data->latitude;
    $lon = $data->longitude;

    This API is only a wrapper for the Maxmind API and for some reason rounds the latitude and longitude from time to time. You can get more accurate results using the Javascript Web Service of Maxmind directly:

    <script src="http://j.maxmind.com/app/geoip.js"></script>
    <script>
    var lat = geoip_latitude();
    var lon = geoip_longitude();
    </script>

    The most detailed data can be obtained with the W3C Geo API though:

    if(navigator.geolocation){
    navigator.geolocation.getCurrentPosition(function(position){
    var lat = position.coords.latitude;
    var lon = position.coords.longitude
    });
    }

    And that is all there is to it :)

    TTMMHTM: Meeting ticker, iPhone JavaScript Geo API, Geekspeakr, Flickr uploadr replacement and when to pee during movies

    Sunday, July 19th, 2009

    Things that made me happy this morning:

    RSS2map – a Placemaker/Yahoo Maps mashup generator script

    Tuesday, July 14th, 2009

    This morning I went to the Telegraph Media Group to talk about GeoMaker and Placemaker. The main thing they wanted to know was how to place an RSS feed that does not have any Geo data on a map and how to style it.

    To make this easier I created rss2map, a demo PHP script with all the CSS and JavaScript needed to show a feed on a map:

    RSS 2 Map example page by  you.

    Introducing Placemaker – a talk at the YDN Tuesday in London

    Tuesday, July 7th, 2009

    This is a talk I’ve given at the YDN Tuesday on 7th of July 2009 in London. It is an introduction to Placemaker, a new geolocation service by Yahoo. Check the slides, audio and the notes below.

    Notes

    Introducing Placemaker

    Hello, I am Chris. Hacker by passion. When I went to the first WhereCamp about two years ago I thought nobody can out-geek me. I was wrong. Geolocation and Geocoding is quite some hard-core branch of geekery. So let me tell you about a nice little product that makes things easy for you.

    Placemaker

    This is Yahoo Placemaker and it is an API. You give it a URL to get data from or a text to extract geographical information from. Here are the docs. Now go forth and build cool stuff. OK then… Let’s take a look at the need for something like Placemaker.

    A web of information

    The web is full of information. Which is cool. The problem is that we accumulated and still accumulate more and more information without giving it proper structure.

    Searching and finding

    Search engines help us find stuff. However, as being found means making money the first search results are not necessarily the best – only the ones that have been promoted the best way.

    Analysing and deciding

    Analysing all the data of the web is a massive job. And computers are stupid. Computers are decision engines that would be thoroughly stumped when asked “do I look fat in this dress” as they forget the underlying dangers in answering this question in one way or another.

    Human additives

    This is why we need humans. By enriching our content with structured, easier to parse data we make it easier for machines to harvest only the necessary parts of our documents. In the past that was keywords, now we use microformats and tagging. The latter is very useful as it can be crowdsourced. People tagging my photos on flickr or my site on del.icio.us make it easier for them to find them later on and give me an idea what keywords I hadn’t thought of.

    Mobility

    This is all fine and good, but the real change we see in behaviour of web users is that we become more and more mobile. Laptops, Mobile devices and Netbooks are a very common sight and wireless networks and fast 3G connectivity allows people to enjoy the web on the go.

    This also means that people can locate themselves on the planet and expect information from their physical surroundings rather than just looking for words, matching and hoping the “night in paris” information they are looking for doesn’t end up in imagery of a disappointing night vision movie experience. In other words, for our content to be found we need to have geographical information in there that defines the locality of the text, not only what it talks about.

    Finding the hidden goodies

    And this is what Placemaker does for us – give it a text or a url and it returns you the geographical information in it, defined as names, a where on earth ID and as latitude and longitude.

    Say I throw the following text at it:

    First we take Manhattan and then we take Berlin.

    If you get an API key you can POST this information to the Placemaker API endpoint like this:

    http://wherein.yahooapis.com/v1/document
    
    documentContent=First+we+take+Manhattan+and+then+we+take+Berlin.
    documentType=text/plain
    appid=my_appid

    Using different parameters

    Placemaker takes different parameters that help you filter down the results to what you want.

    appid
    nothing happens without it!
    inputLanguage
    fr-CA,de-DE…
    outputType
    xml or RSS
    documentContent
    text to analyse
    documentTitle
    additional title
    documentURL
    url to analyse
    documentType
    MIME type of doc
    autoDisambiguate
    remove duplicates, set to false to get more results
    focusWoeid
    filter around a woeid – 400km radius

    Placemaker result sets

    With the above data and parameters you will get the following XML document back:

    
    <?xml version="1.0" encoding="utf-8"?>
    <contentlocation xmlns:yahoo="http://www.yahooapis.com/v1/base.rng" xmlns:xml="http://www.w3.org/XML/1998/namespace" xmlns="http://wherein.yahooapis.com/v1/schema" xml:lang="en">
    <processingTime>0.001987</processingTime>
    <version> build 090508</version>
    <documentLength>48</documentLength>
    <document>
    <administrativeScope>
    <woeId>0</woeId>
    <type>Undefined</type>
    <name><![CDATA[]]></name>
    <centroid>
    <latitude>0</latitude>
    <longitude>0</longitude>
    </centroid>
    </administrativeScope>
    <geographicScope>
    <woeId>1</woeId>
    <type>Supername</type>
    <name><![CDATA[Earth, ZZ]]></name>
    <centroid>
    <latitude>0</latitude>
    <longitude>0</longitude>
    </centroid>
    </geographicScope>
    <extents>
    <center>
    <latitude>52.5161</latitude>
    <longitude>13.377</longitude>
    </center>
    <southWest>
    <latitude>40.6838</latitude>
    <longitude>-74.0477</longitude>
    </southWest>
    <northEast>
    <latitude>52.6675</latitude>
    <longitude>13.7262</longitude>
    </northEast>
    </extents>
    <placeDetails>
    <place>
    <woeId>638242</woeId>
    <type>Town</type>
    <name><![CDATA[Berlin, Berlin, DE]]></name>
    <centroid>
    <latitude>52.5161</latitude>
    <longitude>13.377</longitude>
    </centroid>
    </place>
    <matchType>0</matchType>
    <weight>1</weight>
    <confidence>8</confidence>
    </placeDetails>
    <placeDetails>
    <place>
    <woeId>12589342</woeId>
    <type>County</type>
    <name><![CDATA[Manhattan, New York, NY, US]]></name>
    <centroid>
    <latitude>40.791</latitude>
    <longitude>-73.9659</longitude>
    </centroid>
    </place>
    <matchType>0</matchType>
    <weight>1</weight>
    <confidence>8</confidence>
    </placeDetails>
    <referenceList>
    <reference>
    <woeIds>12589342</woeIds>
    <start>14</start>
    <end>23</end>
    <isPlaintextMarker>1</isPlaintextMarker>
    <text><![CDATA[Manhattan]]></text>
    <type>plaintext</type>
    <xpath><![CDATA[]]></xpath>
    </reference>
    <reference>
    <woeIds>638242</woeIds>
    <start>41</start>
    <end>47</end>
    <isPlaintextMarker>1</isPlaintextMarker>
    <text><![CDATA[Berlin]]></text>
    <type>plaintext</type>
    <xpath><![CDATA[]]></xpath>
    </reference>
    </referenceList>
    </document>
    </contentlocation>
    

    Working with Placemaker results

    Placemaker results have a lot of cool things in them, all explained in detail in the docs. Let’s concentrate on the things we really want to play with here.

    First up is a list of places the API found in the text. These are PlaceDetails elements with a nested place element:

    <placeDetails>
    <place>
    <woeId>12589342</woeId>
    <type>County</type>
    <name><![CDATA[Manhattan, New York, NY, US]]></name>
    <centroid>
    <latitude>40.791</latitude>
    <longitude>-73.9659</longitude>
    </centroid>
    </place>
    <matchType>0</matchType>
    <weight>1</weight>
    <confidence>8</confidence>
    </placeDetails>

    This is cool, but it doesn’t tell us where this information came from. For this there is a referenceList element with an array of references

    <reference>
    <woeIds>638242</woeIds>
    <start>41</start>
    <end>47</end>
    <isPlaintextMarker>1</isPlaintextMarker>
    <text><![CDATA[Berlin]]></text>
    <type>plaintext</type>
    <xpath><![CDATA[]]></xpath>
    </reference>

    Notice the element with the name woeIds, as – oh joy of joys – a reference can have several woeids it is connected with. In order to find out where the text Placemaker found as a match is located in the document, you either get start and end for text content of the XPATH for structured content (XML/RSS). This is pretty sweet, of course.

    Annoyances

    There are a few annoyances when it comes to working with Placemaker.

    The first is a limit of 50,000 bytes for the text to be analyzed which is less than you think when you remember just how much we pack into our web documents.

    The second is that the web is simply not a clean and nice dataset. When you read an HTML document from a live site you’ll find that in many cases Placemaker chokes – for starters only valid UTF-8 documents go through.

    The third is that Placemaker has no JSON output at the moment, which means you cannot use results in JavaScript without writing an own converter.

    Placemaker only allows for POST requests which makes it a bit less easy to play with than with GET enabled APIs (as you can simply open them in a browser window).

    My biggest annoyance is the disconnect of places and references. This is not a problem of Placemaker as it wasn’t meant exclusively to match content to places – just find places. But it makes my favourite use cases – embedding geo location at the right spot in a document – harder.

    Workarounds

    Of course there are workarounds for all these issues (except for the 50000 byte limit).

    Fixes

    As with anything on the web, there are ways to work around these annoyances.

    Fixing the broken web with YQL

    The first trick is to use YQL to load the HTML and filter it before you send it to Placemaker.

    <?php
    $key = 'YOUR_API_KEY';
    if(isset($_GET['url'])){
    $realurl ='http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20html%20where%20url%20%3D%20%22'.urlencode($_GET['url']).'%22&format=xml';
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $realurl);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    $c = curl_exec($ch);
    curl_close($ch);
    if(strstr($c,'<')){
    $c = preg_replace("/.*<results>|</results>.*/",'',$c);
    $c = preg_replace("/<?xml version="1.0"".
    " encoding="UTF-8"?>/",'',$c);
    $c = strip_tags($c);
    $c = preg_replace("/[r?n]+/","",$c);
    $ch = curl_init();
    define('POSTURL',  'http://wherein.yahooapis.com/v1/document');
    define('POSTVARS', 'appid='.$key.'&documentContent='.urlencode($c).
    '&documentType=text/html&outputType=xml');
    $ch = curl_init(POSTURL);
    curl_setopt($ch, CURLOPT_POST, 1);
    curl_setopt($ch, CURLOPT_POSTFIELDS, POSTVARS);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    $x = curl_exec($ch);
    header('content-type:text/xml');
    echo $x;
    }
    }
    ?>

    The YQL implementation of select * from html runs the HTML through Tidy to fix issues and is encoding agnostic. That way Placemaker now can get data from sources it normally chokes on. Notice that it is a good idea to filter out tags and whitespace to save on byte-size.

    Matching references and places

    The best way to explain this is to build a small implementation. For example a simple form that allows a user to enrich a text with Geo Microformats:

    The code is not hard, the main trick is to create an array from the known places and then match them with the IDs of a reference in a nested loop.

    <?php
    
    // if some text was sent through
    if(isset($_POST['analyze'])){
    $content = $_POST['analyze'];
    $template = $_POST['template'];
    // define the API key and do the call to Placemaker
    $key = 'C8meDB7V34EYPVngbIRigCC5caaIMO2scfS2t'.
    '.HVsLK56BQfuQOopavckAaIjJ8-';
    $ch = curl_init();
    define('POSTURL',  'http://wherein.yahooapis.com/v1/document');
    define('POSTVARS', 'appid='.$key.'&documentContent='.
    urlencode($content).
    '&documentType=text/plain&outputType=xml');
    $ch = curl_init(POSTURL);
    curl_setopt($ch, CURLOPT_POST, 1);
    curl_setopt($ch, CURLOPT_POSTFIELDS, POSTVARS);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    $x = curl_exec($ch);
    
    // create an object from the XML
    $places = simplexml_load_string($x, 'SimpleXMLElement',
    LIBXML_NOCDATA);
    // <sup></sup> WTF?
    // loop over places and create an array with
    // the woeid as the key
    $foundplaces = array();
    foreach($places->document->placeDetails as $p){
    $woeid = 'woeid'.$p->place->woeId;
    $foundplaces[$woeid] = array(
    'name' => str_replace(', ZZ','',$p->place->name.''),
    'type' => $p->place->type.'',
    'woeId' => $p->place->woeId.'',
    'lat' => $p->place->centroid->latitude.'',
    'lon' => $p->place->centroid->longitude.''
    );
    }
    
    // loop over the references and over the woeids
    $refs = $places->document->referenceList->reference;
    $microformats = array();
    foreach($refs as $r){
    foreach($r->woeIds as $wi){
    // get dataset connected with the current woeid
    $currentloc = $foundplaces["woeid".$wi];
    
    // check if all interesting data exists
    // get the template and replace the
    // placeholders
    if($r->text != '' && $currentloc['name'] != '' &&
    $currentloc['lat'] != '' && $currentloc['lon'] != ''){
    $lat = $currentloc['lat'];
    $lon = $currentloc['lon'];
    $mf = preg_replace('/%place%/',$r->text,$template);
    $mf = preg_replace('/%lat%/',$lat,$mf);
    $mf = preg_replace('/%lon%/',$lon,$mf);
    $content = preg_replace('/'.$r->text.'/',$mf,$content);
    }
    }
    }
    }
    ?>

    Making Placemaker GET it with YQL

    Another thing YQL allows developers to do is to extend it with own open tables that run JavaScript conversions on the server side. One of those is the YQL open table which does all the things Placemaker does but on the server and offers JSON output.

    The great thing about the JSON output is that it already matches up places and references for us:

    {
    "query":{
    <a href=""1" title="">count</a>",
    <a href=""2009-07-05T06:30:53Z" title="">created</a>",
    <a href=""en-US" title="">lang</a>",
    <a href=""2009-07-05T06:30:53Z" title="">updated</a>",
    <a href=""http://query.yahooapis.com/v1/yql?q=SELECT+*+FROM+geo.placemaker+WHERE+documentContent+%3D+%22First+we+take+Manhattan+and+then+we+take+Berlin%22+AND+documentType%3D%22text%2Fplain%22+AND+appid+%3D+%22%22" title="">uri</a>",
    "diagnostics":{
    <a href=""true" title="">publiclyCallable</a>",
    "url":[{
    <a href=""2" title="">execution-time</a>",
    <a href=""http://datatables.org/alltables.env" title="">content</a>"
    },
    {
    <a href=""183" title="">execution-time</a>",
    <a href=""http://www.datatables.org/geo/geo.placemaker.xml" title="">content</a>"
    },
    {
    <a href=""2" title="">execution-time</a>",
    <a href=""http://wherein.yahooapis.com/v1/document" title="">content</a>"
    }
    ],
    "javascript":{
    <a href=""11805" title="">instructions-used</a>"
    },
    <a href=""338" title="">user-time</a>",
    <a href=""187" title="">service-time</a>",
    <a href=""1949" title="">build-version</a>"
    },
    "results":{
    "matches":{
    "match":[{
    "place":{
    <a href=""638242" title="">woeId</a>",
    <a href=""Town" title="">type</a>",
    <a href=""Berlin" title="">name</a>, Berlin, DE",
    "centroid":{
    <a href=""52.5161" title="">latitude</a>",
    <a href=""13.377" title="">longitude</a>"
    }
    },
    "reference":{
    <a href=""638242" title="">woeIds</a>",
    <a href=""41" title="">start</a>",
    <a href=""47" title="">end</a>",
    <a href=""1" title="">isPlaintextMarker</a>",
    <a href=""Berlin" title="">text</a>",
    <a href=""plaintext" title="">type</a>",
    "xpath":""
    }
    },
    {
    "place":{
    <a href=""12589342" title="">woeId</a>",
    <a href=""County" title="">type</a>",
    <a href=""Manhattan" title="">name</a>, New York, NY, US",
    "centroid":{
    <a href=""40.791" title="">latitude</a>",
    <a href=""-73.9659" title="">longitude</a>"
    }
    },
    "reference":{
    <a href=""12589342" title="">woeIds</a>",
    <a href=""14" title="">start</a>",
    <a href=""23" title="">end</a>",
    <a href=""1" title="">isPlaintextMarker</a>",
    <a href=""Manhattan" title="">text</a>",
    <a href=""plaintext" title="">type</a>",
    "xpath":""
    }
    }
    ]
    }
    }
    }
    }

    Using the open table we can easily use Placemaker in JavaScript:

    function gotit(o){
    var matches = o.query.results.matches.match;
    for(var i=0,j=matches.length;i<j;i++){
    console.log('Name:  ' + matches[i].place.name);
    console.log('lat:   ' + matches[i].place.centroid.latitude);
    console.log('lon:   ' + matches[i].place.centroid.longitude);
    console.log('Match: ' + matches[i].reference.text);
    }
    }
    var content = 'First we take Manhattan and then we take Berlin';
    var yql = 'select * from geo.placemaker where documentContent = "' +
    content + '" and documentType="text/plain" and appid = ""';
    var url = 'http://query.yahooapis.com/v1/public/yql?' +
    'format=json&callback=gotit&env=' +
    'http%3A%2F%2Fdatatables.org%2Falltables.env&q=' +
    encodeURIComponent(yql);
    var s = document.createElement('script');
    s.setAttribute('src',url);
    document.getElementsByTagName('head')[0].appendChild(s);

    Implementations

    • Yahoo News Map – Yahoo News Map uses the Yahoo RSS feed run through Placemaker to show news on a map and allow to navigate with the map.
    • TweetLocations – Tweetlocations shows a map of your latest tweets.
    • Geo this! (Greasemonkey) – Geo This! is a Greasemonkey script for WordPress that adds a button to analyze and tag the content before submitting the blog post.
    • GeoMaker – GeoMaker is a frontend to Placemaker that turns a URL or a text into a map.
    • GeoMaker API – GeoMaker also has an own API that makes it easy to convert URLs to all kind of handy formats.
    • JS-PlacemakerJS Placemaker is a JavaScript wrapper for Placemaker using the open YQL table.

    You have the data, you have the tools, now go and make some ideas a reality

    That’s all I have for you today. Check the resources coming up and have a play with Placemaker. Contact me once you’ve done something cool and I’ll be happy to tell the team about it. Here are some ideas:

    Flickr knows woeid

    Using YQL and open tables you can get geolocated photos from a text analysed with Placemaker:

    select * from flickr.photos.info where photo_id in
    (
    select id from flickr.photos.search where woe_id in
    (
    select match.place.woeId from geo.placemaker where
    documentContent = "First we take Manhattan and then we take Berlin"
    and documentType="text/plain" and appid = ""
    )
    and license=4
    )

    Other Yahoo Geo resources

    The Yahoo Geo section of the Yahoo Developer Network has all the other Geo goodies for us. Maps, FireEagle and even the Placemaker dataset for download is all there for you to use.

    The Guardian Data Store

    The Guardian has a really nice blog/resource that always has new data for you to play with: the Guardian Data Store. Have a look and a play.