Derick Rethans - tag: openstreetmap

Importing OpenStreetMap data into MongoDB

Tue, 11 Jun 2013 08:49:00 +0000

Importing OpenStreetMap data into MongoDB

Tuesday, June 11th 2013, 09:49 BST

In many recent MongoDB related presentations I have used OpenStreetMap data as basis for most of my examples. I wrote a script that imports OpenStreetMap (OSM) nodes, ways and to a lesser extend relations into MongoDB with a specific and optimal schema. I have written about this briefly before in Indexing Freeform-Tagged Data, but now MongoDB received numerous updates to geospatial indexes I find it warrants a new article.

In MongoDB 2.2 and before, the index type that MongoDB used was the 2d type, which was basically a two dimensional flat earth coordinate system index—with some spherical features built on top. MongoDB 2.4 has a new index type: 2sphere. Instead of just being able to index points described by x/y coordinates (longitude/latitude) it now has support for indexing points, line strings and polygons as defined by GeoJSON.

I have modified my import script to use those new types, and also added support for very simple multi-polygons that OpenStreetMap records through its relation tag. The script also creates an indexes on { l: '2dsphere' } (the GeoJson object), { ts: 1 } (the tags), and { ty: 1 } (the type).

The structure it converts a node from OpenStreetMap to looks like:

{
        "_id" : "n26486695",
        "ty" : NumberLong(1),
        "l" : {
                "type" : "Point",
                "coordinates" : [
                        -0.1580359,
                        51.4500055
                ]
        },
        "ts" : [
                "addr:housenumber=97",
                "addr:postcode=SW12 8NX",
                "addr:street=Nightingale Lane",
                "amenity=pub",
                "name=The Nightingale",
                "operator=Youngs",
                "source:name=photograph",
                "toilets=yes",
                "toilets:access=customers",
                "website=http://www.youngs.co.uk/pub-detail.asp?PubID=430"
        ],
        "m" : {
                "v" : NumberLong(5),
                "cs" : NumberLong(11229430),
                "uid" : NumberLong(652021),
                "ts" : NumberLong(1333911628)
        }
}

There are several sections that make up the document:

_id: Is a combination of n and the OSM node id.
ty: Is the type. For nodes, this is always 1.
l: The point's location in GeoJson format. An OSM point is translated to a GeoJson Point feature with an array describing the longitude and latitude.
ts: Are the tags that describe the node. Each tag is stored as a concatenation of its key and its value. This creates both a smaller index and it still allows for exact tag/value matches as well as matching against specfic keys through a regular expression match. For example, we could find the above document with:

db.poiConcat.find( { ts: 'name=The Nightingale' } );

And the index would also be used when we look for all amenities:

db.poiConcat.find( { ts: /^amenity=/ } );
m: Contains meta information that describes the node. The following fields are currently present:
- v: The object's version. This is the version number of the version that was found in the imported file. There is always just one version per object.
- cs: The changeset ID in which this object was last updated.
- uid: The user ID of the OpenStreetMap contributor who uploaded the latest version.
- ts: The Unix timestamp of when this object was last updated.

OpenStreetMap ways are stored in two different types. Unclosed ways are stored as GeoJSON LineStrings (think roads):

{
        "_id" : "w2423886",
        "ty" : NumberLong(2),
        "l" : {
                "type" : "LineString",
                "coordinates" : [
                        [ -0.1044769, 51.508462 ],
                        [ -0.1044093, 51.5106306 ],
                        [ -0.1044139, 51.5107814 ],
                        [ -0.104427, 51.5108453 ],
                        [ -0.1044459, 51.5109208 ],
                        [ -0.1045131, 51.5110686 ]
                ]
        },
…

And closed ways are stored as GeoJSON Polygons (think buildings and parks):

{
    "_id" : "w24257746",
    "ty" : NumberLong(2),
    "l" : {
        "type" : "Polygon",
        "coordinates" : [
            [
                [ -0.0745133, 51.560977 ],
                [ -0.0742252, 51.5609742 ],
                [ -0.0742308, 51.5606721 ],
                [ -0.0745217, 51.5606721 ],
                [ -0.0745133, 51.560977 ]
            ]
        ]
    },
    "ts" : [
        "amenity=park",
        "leisure=park",
        "name=Kynaston Gardens"
    ],
    "m" : {
        "v" : NumberLong(1),
        "cs" : NumberLong(357805),
        "uid" : NumberLong(5139),
        "ts" : NumberLong(1210169336)
    }
}

Both ways and areas (closed ways) will have a ty value of 2, as they both come from a way primitive as stored in OpenStreetMap.

The script is available on GitHub as part of the 3angle repository. The latest version is at https://raw.github.com/derickr/3angle/master/import-data.php and it also requires https://raw.github.com/derickr/3angle/master/classes.php for some GeoJSON helper classes and https://raw.github.com/derickr/3angle/master/config.php where you can set the database name and collection name (in my case, demo and poiConcat).

Map data © OpenStreetMap contributors (terms).

Crowd-serfing

London, UK

Friday, April 12th 2013, 09:27 BST

Yesterday, Google announced that they have made Google Map Maker available in the United Kingdom. Like OpenStreetMap it allows everybody to update and add things to the map. But there is one big difference: With MapMaker you don't get access to the data.

Fellow OpenStreetMapper, Richard Fairhurst, describes this as Crowd-serfing:

Crowd-serfing, n.: when a large corp uses crowd-sourced volunteering for its own financial gain, without giving back. See: @googlemapmaker.

I will never understand why people do work for a commercial company without getting any real benefit back: http://www.bbc.co.uk/news/business-21226623

Unfortunately, today's BBC coverage on the availability of Google MapMaker in the UK read more like a manual on MapMaker than a nicely unbiased piece on crowd-sourced maps. Only after one of the OpenStreetMappers reached out to the journalist that wrote the piece, they added some background on OpenStreetMap:

"The biggest problem with Google Map Maker is that anything people contribute may appear on Google's map, but only Google can get at the underlying data to be able to do anything else with it," said Chris Hill.

"If someone includes a Google map on their web site to show where their business is they may also be showing where their competitors are and they can't change that."

Sure, it's nice to have some roads on the Google Map, but you will never even have full access back to your data, unlike OpenStreetMap where you can download and work with all the data. Even nicer is that often, OpenStreetMap still has better maps than GoogleMaps - for example, have a look at this in North Korea: http://tools.geofabrik.de/mc/?mt0=mapnik&mt1=googlemap&lon=125.74677&lat=39.01863&zoom=14 . And closer to home in the United Kingdom, compare some of the hiking trails in the Peak District: http://tools.geofabrik.de/mc/?mt0=mapnik&mt1=googlemap&lon=-1.99876&lat=53.17827&zoom=15 . Even in places like London, the accuracy of the locations of addresses and points-of-interest (POIs) is often a lot better, as OpenStreetMap doesn't use web site scraping and post-code-centroid locations to place POIs: http://tools.geofabrik.de/mc/?mt0=mapnik&mt1=googlemap&lon=-0.12405&lat=51.50862&zoom=18 . OpenStreetMap mostly relies on surveys, done by individuals (like you!) to verify things are actually there, aided a little by the availability of Bing Maps as background imagery.

One of the things that most people forget, is the terms and conditions that commercial entities state. An except from MapMaker's reads:

"You give Google a perpetual, irrevocable, worldwide, royalty-free, and non-exclusive licence to reproduce, adapt, modify, translate, publish, publicly perform, publicly display, distribute, and create derivative works of the user submission."

Note that it never mentions that you can do anything with the data yourself…

If you are contributing time and knowledge, why not allow yourself to benefit from it as well? I realise that OpenStreetMap might not be as accessible, and the map-tiles on their web site aren't the prettiest, but the real benefit is in the access to the raw data that makes up the images in the map. The OpenStreetMap wiki also has an article on this.

Access to the data allows you to do so many more things. From creating your own fancy map-styles, creating "washable, wearable, all-weather maps designed for the real outdoors" such as SplashMaps, to powering web-sites that show accessibility. If you don't like the way map data is rendered, you can produce something in your own style, just like Nike did with this campaign.

Also, there is nothing better if some of your handiwork shows up in a "best of OSM" poster :-) I would never spend my time adding data to Google's Maps. Instead, I prefer to contribute to OpenStreetMap and do awesome things with the data. In the meanwhile, OpenStreetMap continues to support humanitarian relief efforts in Mali and many other places.

If you're near London, come and join us this summer! (Other places will also run mapping parties).

OpenStreetMap: A Year of Edits 2012

London, UK

Tuesday, January 1st 2013, 00:04 GMT

Just like last year I've analysed all the edits to OpenStreetMap and produced a visualisation. It was a bit trickier this year as I also wanted to visualize the work of the redaction bot as well as the "redaction period" and the switch to ODbL for OpenStreetMap data.

The result is: OpenStreetMap: A Year of Edits (2012):

OpenStreetMap: A Year of Edits (2012)

This animation shows all additions and modifications of nodes (flashes) and cumulative edits (fading to purple) in 2012. Additional visualisations are the redaction bot's work (flying away yellow spheres) and the shift to ODbL (green flash). It gives a good overview of the effort that 1000s of contributors put in to make OpenStreetMap the best source of mapping data.

You can see the video in HD on Vimeo.

I have started to make the code for this available through GitHub, but the code is very much not modularized and it is likely it will only work on my set-up. Expect this to change for next year!

Mappy New Year!

Address lookups with Leaflet and Nominatim

London, UK

Tuesday, November 20th 2012, 10:52 GMT

I recently wrote a patch for joind.in to add a map of an event's location to the event detail page. With the same patch, I also replaced the location part of the event edit page with a solution that uses JQuery, Leaflet as map API, OpenStreetMap tiles and Nominatim for doing address lookups. This article forms a small tutorial on how to use this same set-up yourself.

The Basics

To start, we create a new directory for our project:

mkdir addresses
cd addresses

Then I downloaded the Leaflet and jQuery libraries and extracted them in the js directory of the project:

mkdir js
curl -L https://github.com/CloudMade/Leaflet/zipball/v0.4.5 -o leaflet.zip
unzip leaflet.zip
mv CloudMade-Leaflet-*/dist/* js
rm -rf CloudMade-Leaflet-*
rm leaflet.zip
curl http://code.jquery.com/jquery-1.8.2.min.js -o js/jquery-1.8.2.min.js

As first step, we are simply going to show a map on a web page. The map is going to be full screen, and will not have any bells and whistles. The code to embed a map is small, but we will separate it into three files for clarity: a CSS file for our styles (site.css), an HTML file for the structure (index.html) and a JS file for all our JavaScript functions (js/map.js).

Let's start with the HTML file:


  
    Leaflet and Nominatim example

This HTML file includes the Leaflet and jQuery libraries, as well as the default CSS file that Leaflet needs. We are also including our own CSS file (site.css):

body {
  margin: 0;
}
div#map {
  width: 100%;
  height: 100%;
}

In the body of the HTML file, we place a

as contained for the map, and then include a JavaScript file that is responsible for embedding them map:

var map;

function load_map() {
  map = new L.Map('map', {zoomControl: false});

  var osmUrl = 'http://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png',
    osmAttribution = 'Map data © 2012 OpenStreetMap contributors',
    osm = new L.TileLayer(osmUrl, {maxZoom: 18, attribution: osmAttribution});

  map.setView(new L.LatLng(51.538594, -0.198075), 12).addLayer(osm);
}

window.onload = load_map;

If you request the index.html page now through the browser, you will see something like:

Adding the Address Search

In order to add an address lookup form, we need to add more HTML. Our HTML will feature an input box (for the address), a submit button, and a place holder to show our results in. We add this code between the

and the

 tags:
      
  
  
  



      To style this, we add the following at the end of our CSS file:
      div#search {
  background-color: white;
  position: absolute;
  bottom: 40px;
  left: 40px;
  width: auto;
  height: auto;
  padding: 10px;
}
div#search input {
  width: 200px;
}
div#results {
  font-style: sans-serif;
  color: black;
  font-size: 75%;
}


      If we reload the page in our browser, we will see something like:
      
      Now the only thing left to do is to implement the addr_search function. In our JS file (js/map.js) we add before window.onload = load_map; the following lines (split over multiple sections in this tutorial):
      function addr_search() {
  var inp = document.getElementById("addr");

  $.getJSON('http://nominatim.openstreetmap.org/search?format=json&limit=5&q=' + inp.value, function(data) {


      This above line uses jQuery's AJAX capabilities to request a URL, parse the JSON result and issue a callback if it worked. We query Nominatim here, with as format json and limiting the result to 5 items. Nominatim also supports other parameters, which are documented here. 
      var items = [];

$.each(data, function(key, val) {
  items.push(
    "" + val.display_name +
    ''
  );
});


      For each of the items in our result, we create an 
 element which has an  containing a call to a JavaScript function (chooseAddr). This function is responsible for re-centering the map according to the picked latitude and longitude. 
          $('#results').empty();
    if (items.length != 0) {
      $('', { html: "Search results:" }).appendTo('#results');
      $('
', {
        'class': 'my-new-list',
        html: items.join('')
      }).appendTo('#results');
    } else {
      $('', { html: "No results found" }).appendTo('#results');
    }
  });
}


      This processes the results that came back from Nominatim. If there are results, we shows those including a Search results: header, and if there are no results, we show No results found.
      Then we need to add one more function, the chooseAddr function which looks like:
      function chooseAddr(lat, lng, type) {
  var location = new L.LatLng(lat, lng);
  map.panTo(location);

  if (type == 'city' || type == 'administrative') {
    map.setZoom(11);
  } else {
    map.setZoom(13);
  }
}


      We simply use the latitude and longitude from the function invocation, and in order to make things slightly nicer we zoom in a bit less if the item type is either a city or an administrative border. As each of the returned results actually includes a full bounding box, we probably can use that to zoom in better, but I will leave that for your own experiments - you'd want the panInsideBounds() method of Leaflet's Map class for that.
      In the end, if we click on the Search button, a list is presented of all our search results:
      
      And after clicking one of the links, we see the map centered on Paris:
      
      The code for this example is available on github in my osm-tools repository at https://github.com/derickr/osm-tools/tree/master/leaflet-nominatim-example



Displaying OpenStreetMap data from MongoDB
Tue, 16 Oct 2012 09:12:00 +0000

  
    
      Displaying OpenStreetMap data from MongoDB

      
      
         Mainz, Germany
        Tuesday, October 16th 2012, 11:12 CEST
      
      In an earlier article I have already written about several different ways on how you could store OpenStreetmap data in MongoDB.
      
        
        
      

      
      
    

  




Indexing Freeform-Tagged Data
Tue, 26 Jun 2012 07:11:00 +0000

  
    
      Indexing Freeform-Tagged Data

      
      
         Dieren, Netherlands
        Tuesday, June 26th 2012, 09:11 CEST
      
      At last week's MongoDB UK I presented on "Geolocation, Maps and MongoDB". During this talk I presented a few methods on how to store OpenStreetMap's tags on geographical objects. OpenStreetMap's tags are free form and for example for a road can look like:
      alt_name:    The Strand
highway:     primary
name:        Strand
oneway:      yes
postal_code: WC2


      For this article, I am using a data extra from OpenStreetMap for just London. There are about 2.2 million items in my database after importing taking up 413MB of storage (without indexes).
      
        Tags as a normal object
      
      The first method for storing into MongoDB basically uses a tags field with each of the OpenStreetMap tag taking up a property in the array:
      "_id" : "w1572026939",
"type" : NumberLong(2),
"tags" : {
        "alt_name" : "The Strand",
        "highway" : "primary",
        "name" : "Strand",
        "oneway" : "yes",
        "postal_code" : "WC2"
},
"loc" : [ [ -0.125262, 51.5086046 ], …, [ -0.1270609, 51.5076398 ] ]


      In order to make looking up tags and values faster, we will need to create some indexes. We would like to look for streets by name:
      db.poi.ensureIndex( { 'tags.name': 1 } );
db.poi.find( { 'tags.name': 'Strand' } );


      And also find roads and paths:
      db.poi.ensureIndex( { 'tags.highway': 1 } );
db.poi.find( { 'tags.highway' : { $exists: 1 } } );


      As you can see, for every tag that we want to search on we would have to create an index. More indexes make inserts, updates and deletes slower and MongoDB is also limited to 64 indexes per collection. In order to have an index on every tag we need to find another solution. Also note that the second query doesn't even use an index when you check it with calling explain().
      
        Tags as key/value pairs
      
      The second method I introduced was the one where instead of storing the tags as properties of the tags field, I stored objects containing a tag/tagvalue pair each:
      "tags_indexed" : [
        { "k" : "alt_name", "v" : "The Strand" },
        { "k" : "highway", "v" : "primary" },
        { "k" : "name", "v" : "Strand" },
        { "k" : "oneway", "v" : "yes" },
        { "k" : "postal_code", "v" : "WC2" }
],


      The index we then can create is:
      db.poi.ensureIndex( { 'tags_indexed.k' : 1, 'tags_indexed.v' : 1 } );


      To answer the same questions as above we have to rewrite the queries. To find all documents where the name is Strand we can not simply do:
      db.poi.find( { 'tags_indexed.k' : 'name', 'tags_indexed.v' : 'Strand' } );


      This query will find all documents where either the key is name or the value is Strand. This includes documents where there is an array element like { "k" : "name", "v" : "Strand" } in the tags_indexed array, but also every other document where there is one "k" : "name" and a "v" : "Strand" element, such as in:
      {
        "_id" : ObjectId("4fe4c4b044670a41222088b5"),
        "tags_indexed" : [
                { "k" : "addr:housenumber", "v" : "79" },
                { "k" : "addr:postcode", "v" : "WC2R 0DW" },
                { "k" : "addr:street", "v" : "Strand" },
                { "k" : "building", "v" : "yes" },
                { "k" : "name", "v" : "Stamp Centre" }
        ]
}


      In order to match only documents where there is one array element with both the "k" : "name" and "v" : "Strand" properties, you need to use $elemMatch:
      db.poi.find( {
        'tags_indexed' : { $elemMatch: { 'k' : 'name', 'v' : 'Strand' } }
} );


      Let's look at the explain output of both variants. In the first case (with 'name': 'Strand'), the explain output looks like:
      > db.poi.find( { 'tags.name': 'Strand' } ).explain();
{
        "cursor" : "BtreeCursor tags.name_1",
        "nscanned" : 12,
        "nscannedObjects" : 12,
        "n" : 12,
        "millis" : 1,
        "nYields" : 0,
        "nChunkSkips" : 0,
        "isMultiKey" : false,
        "indexOnly" : false,
        "indexBounds" : {
                "tags.name" : [ [ "Strand", "Strand" ] ]
        }
}


      And in the second case, the explain output looks like:
      > db.poi.find( {
... 'tags_indexed' : { $elemMatch: { 'k' : 'name', 'v' : 'Strand' } }
... } ).explain();
{
        "cursor" : "BtreeCursor tags_indexed.k_1_tags_indexed.v_1",
        "nscanned" : 172145,
        "nscannedObjects" : 172145,
        "n" : 12,
        "millis" : 429,
        "nYields" : 0,
        "nChunkSkips" : 0,
        "isMultiKey" : true,
        "indexOnly" : false,
        "indexBounds" : {
                "tags_indexed.k" : [ [ "name", "name" ] ],
                "tags_indexed.v" : [ [ { "$minElement" : 1 }, { "$maxElement" : 1 } ] ]
        }
}



      
      In the first case, 12 documents are scanned and found. But in the second case, the same 12 documents are returned, but 172145 documents are scanned.  This is because at the moment, $elemMatch does not use the index fully and will only use the first part of the compound index1 — tags_indexed.k in our case.
      In order to make index usage better, it therefore makes more sense to have the property with the highest cardinality to be the first key in the compound index. In my data-set, the cardinality of tags_indexed.v is (a lot) higher than tags_indexed.k:
      > db.poi.distinct( 'tags_indexed.k' ).length;
1281
> db.poi.distinct( 'tags_indexed.v' ).length;
151581


      So let's change the keys in the compound index around, and run the explain for the second query again:
      > db.poi.dropIndex( 'tags_indexed.k_1_tags_indexed.v_1' );
> db.poi.ensureIndex( { 'tags_indexed.v' : 1, 'tags_indexed.k' : 1 } );

> db.poi.find( {
... 'tags_indexed' : { $elemMatch: { 'k' : 'name', 'v' : 'Strand' } }
... } ).explain();
{
        "cursor" : "BtreeCursor tags_indexed.v_1_tags_indexed.k_1",
        "nscanned" : 44,
        "nscannedObjects" : 44,
        "n" : 12,
        "millis" : 0,
        "nYields" : 0,
        "nChunkSkips" : 0,
        "isMultiKey" : true,
        "indexOnly" : false,
        "indexBounds" : {
                "tags_indexed.v" : [ [ "Strand", "Strand" ] ],
                "tags_indexed.k" : [ [ { "$minElement" : 1 }, { "$maxElement" : 1 } ] ]
        }
}


      Now with the higher cardinality on the first part of the compound key only 44 documents are scanned. And instead of 429 milliseconds it now took less than 1 millisecond. Excellent news!
      The second question from the first section ("find roads and paths") can be rewritten as:
      db.poi.find( { 'tags_indexed.k': 'highway' } );


      However, this query can not be serviced by our rewritten index ('tags_indexed.v' : 1, 'tags_indexed.k' : 1) as MongoDB can only use keys starting from the left side of a compound index for queries. We can now either add another index that only encompasses the k part or perhaps find a different solution.
      
        Tag keys and values combined
      
      There is one other alternative that I would like to explore, and that is where I combine the key and value of each tag into one string, and store those all in an array in the object, like:
      "tags_combined" : [
        "alt_name=The Strand",
        "highway=primary",
        "name=Strand",
        "oneway=yes",
        "postal_code=WC2"
],


      And then we index this as:
      db.poi.ensureIndex( { 'tags_combined': 1 } );


      The first query we used earlier in this article ("all items where the name is Strand") now looks like:
      > db.poi.find( { 'tags_combined': "name=Strand" ).explain();
{
        "cursor" : "BtreeCursor tags_combined_1",
        "nscanned" : 12,
        "nscannedObjects" : 12,
        "n" : 12,
        "millis" : 0,
        "nYields" : 0,
        "nChunkSkips" : 0,
        "isMultiKey" : true,
        "indexOnly" : false,
        "indexBounds" : {
                "tags_combined" : [ [ "name=Strand", "name=Strand" ] ]
        }
}


      As you can see the index is very good at resolving this case, as it only scanned the 12 documents that the query also returned—just like where we had a direct index on tags.name.
      The second query (all paths and roads) and its explain syntax are:
      > db.poi.find( { 'tags_combined': /^highway=/ } ).explain();
{
        "cursor" : "BtreeCursor tags_combined_1 multi",
        "nscanned" : 210441,
        "nscannedObjects" : 210440,
        "n" : 210440,
        "millis" : 741,
        "nYields" : 0,
        "nChunkSkips" : 0,
        "isMultiKey" : true,
        "indexOnly" : false,
        "indexBounds" : {
                "tags_combined" : [
                        [ "highway=", "highway>" ], [ /^highway=/, /^highway=/ ]
                ]
        }
}


      Here we use a regular expression search (/^highway=/) anchored to the start of the string (with the ^). This can use an index, and is almost as good as the the case where tags_indexed.k was the first part of the compound key, except that it is a bit slower (527 vs 741 milliseconds):
      > db.poi.find( { 'tags_indexed.k': 'highway' } ).explain();
{
        "cursor" : "BtreeCursor tags_indexed.k_1_tags_indexed.v_1",
        "nscanned" : 210440,
        "nscannedObjects" : 210440,
        "n" : 210440,
        "millis" : 527,
        "nYields" : 0,
        "nChunkSkips" : 0,
        "isMultiKey" : true,
        "indexOnly" : false,
        "indexBounds" : {
                "tags_indexed.k" : [ [ "highway", "highway" ] ],
                "tags_indexed.v" : [ [ { "$minElement" : 1 }, { "$maxElement" : 1 } ] ]
        }
}


      
        Conclusion
      
      Right now, I think that having an indexed on tags_combined where each tag is combined with key and value and an array element tags_combined is the best solution. Its index scanning is better for the name=Strand case than the other, and not significantly slower in the highway=* case. I would still like to see how the speed differences progress when I add more objects to the database.
      
        
        
      

      
      
    

    
      
        1
        : https://jira.mongodb.org/browse/SERVER-4520
      
    
  



Ignite London: Crowd Sourcing a Map of the World
Tue, 15 May 2012 08:47:00 +0000

  
    
      Ignite London: Crowd Sourcing a Map of the World

      
      
         London, UK
        Tuesday, May 15th 2012, 09:47 BST
      
      Almost  two weeks ago, I gave a talk at Ignite London about OpenStreetMap, titled "Crowd Sourcing a Map of the World". Ignite's presentation style is 20 slides which automatically advance every 15 seconds. Having never done this before I actually wrote the whole talk out. The presentation that I gave slightly diverges from this but I thought it'd still be good to reproduce here. I did add some links to more information, and if you want to see the recording, you can find it at the end of this post.
      
      1. This talk is about a project, started here in the UK with as its major goal to create a free map of the whole planet. From roads and motorways to country-side footpaths, restaurants and of course pubs. This talk is about OpenStreetMap, the free map of the world.
      
      2. There are of course already plenty of mapping solutions available. Maybe one of the best maps can be acquired through Ordnance Survey. They can be regarded as the national authority on this subject. It's however expensive to get access to their maps, especially the very detailed maps from OS MasterMap. Additionally, it's only for the UK.
      
      3. Besides the commercial solutions, you might wonder why we simply can't do with GoogleMaps? It's mostly freely available for use and also provides you with satellite imagery and StreetView. They even allow you in some areas to update the map through Google MapMaker.
      
      4. But one thing Google doesn't give you access to, is the data behind the map. All you will ever see, is the rendered map tiles and perhaps some APIs to lookup locations and points of interest. Even for data that you have added yourself through MapMaker.
      
      5. Both aspects; the cost of commercial maps, as well as the access to the data that is behind the map tiles is something that the OpenStreetMap project addresses. But which steps have to be taken to obtain this enormous amount of geographical data?
      
      6. We start by getting our wellies and trusty GPS out. Maybe even some pen and paper. We find a location that looks rather empty on the map and travel to that area to see what's on the ground. This is step one: data gathering in the field.
      
      7. In urban areas such as London the roads have often already been mapped and a GPS is not accurate enough to be useful. Then we just use pen and paper to record points of interest, such as shops, landmarks, restaurants and postboxes, my personal favourite.
      
      8. In the country side, donated aerial imagery makes it possible for us to easily trace tracks and footpaths. However, you can't be sure whether the imagery is up-to-date, and you can't always see where fences, streams and local wild life create barriers.
      
      9. In both situations, surveys are best done in groups: at mapping parties. It helps spread the workload and a larger area can be surveyed in one go. As an additional benefit, it allows us to go the pub and discuss our mapping adventures!
      
      10. Doing a survey is important. We take photographs, video and notes with pen and paper of everything that seems to be of interest. This leaves a record that everything we map is actually existing and we can prove that nothing has been copied from other copyrighted maps.
      
      11. After collecting the data, we enter it into the database. This includes basic information such as street names, but we also record whether a café has wheelchair access, or whether a pub has wifi. Updates to the map show up on the site close to real time.
      
      12. Every map object has tags associated with it. Tags tell whether a line is a road, or perhaps a fence. All the tags are free form so you can generally add as much information about an object as you want. Sometimes however, this gets slightly out of hand and people tag pandas in trees and eyes on postboxes.
      
      13. Once the data has been added to the map, we can make use of it. One of the primary uses is obviously showing the data as map tiles. But with all the extra data, we can generate maps that show all the information you're interested in-and nothing more.
      
      14. Clockwise, we have four different visualisations of the map data: we have a cycling-specific style, a style that shows transport routes, a rendering with MapQuest's style sheets and even a water colours inspired style.
      
      15. Having mentioned MapQuest; they were one of the first companies to make use of OpenStreetMap data. They provide, free of charge, map tiles with their own rendering style as well as an instance of Nominatim, OpenStreetMap's geolocation sister project.
      
      16. Lots of companies have already switched to OpenStreetMap.  The property search site Nestoria recently switched from using GoogleMaps to OpenStreetMap. Partly because of their costs, but also partly because "The maps are equal or better". geocaching.com, TfL's countdown website and Apple also use OpenStreetMap maps and data.
      
      17. Although OpenStreetMap provides a free and editable map of the world, there are certain requirements for using the data as well. The most important one is that you always need to attribute the OpenStreetMap project.
      
      18. In order to help people start using OpenStreetMap for their mapping needs, the Switch2OSM site has been launched. This website provides background information, case studies and technical information on how to use OpenStreetMap data.
      
      19. Right now, OpenStreetMap has very good data coverage in the country, but we are not nearly finished. A lot of work still has to be done, and we rely on you to improve the data too, even if you add just a postbox.
      
      20. In the last 5 minutes we have looked at what OpenStreetMap is, how the data is gathered and how the data is added to the map. Further more, we had a look at different use cases of the data. OpenStreetMap in the UK: Footpaths and pubs a speciality!
      And then I planned showing the "Year of Edits" video, but that sadly didn't work out. I'm including it for good measure here though. (If you want it in HD, follow the link).
      
        
        OpenStreetMap: A Year of Edits (2011)

      

      The video of the talk itself is at http://vimeo.com/41626116 and is embedded here:
      
        
        Crowdsourcing a Map of the World

      

      
        
        
      

      
      
    

  




OpenStreetMap: A Year of Edits
Sun, 01 Jan 2012 00:00:00 +0000

  
    
      OpenStreetMap: A Year of Edits

      
      
         London, UK
        Sunday, January 1st 2012, 00:00 GMT
      
      In the past weeks I've been working on some visualisations related to additions and changes being made to OpenStreetMap. To start of the new year, I hereby present: OpenStreetMap: A Year of Edits (2011):
      
        
        OpenStreetMap: A Year of Edits (2011)

      

      You can see the video in HD on Vimeo. Happy New Year!
      
        How Did He Do That?
      
      This animation is all made with Open Source software. I haven't had the time to clean up the code to make it releasable yet, but I am intending to do that in the near feature and add it to my OpenStreetMap tools git repository on github.
      The first thing I did, was download all the hourly replication files from http://planet.openstreetmap.org/hour-replicate for the year.
      With a script, I scanned over all those XML files and for every three hours I created an image showing the edits of those last three hours (in white), as well as all the other edits in the background (in purple). With 4 frames per day, this gives me 1460 frames.
      Each of the frames is an equirectangular projection which I then map onto a sphere with a faded background of "earthmap10k" of http://planetpixelemporium.com/earth.html (I downloaded it years ago when it was still free to download). For the 3D rendering of the Earth with the edits layed on top of it I used POV-Ray.
      I generated two different loops so that I can make two full rotations at different angles. This gives me in total 2882 frames which is about two minutes. The POV-Ray generated images I post-processed by overlaying the progress bar in the lower left and added the fading and merging effects. Then I used mencoder to create a MPEG video out of it.  Finally I added the sound track again by using mencoder and uploaded it to Vimeo.
      
        
        
      

      
      
    

  




South Kensington Mapping Party Animation
Tue, 19 Jul 2011 08:17:00 +0000

  
    
      South Kensington Mapping Party Animation

      
      
         London, UK
        Tuesday, July 19th 2011, 09:17 BST
      
      Every two weeks or-so, the London OpenStreetMap group comes together for a mapping party. The main idea is to go to an area of London to improve the map there. Sometimes that's adding streets, but as most of those are now completed, time is spend on buildings, address information and points-of-interest.
      As an experiment I recorded a snapshot of the map for the area (http://mapcraft.nanodesu.ru/pie/73) every hour to follow the progress of people adding new and improved data to the map. I then rendered an image of each map snapshot and put those together to create a video:
      
        
        South Kensington Mapping Party animation — Map data © OpenStreetMap contributors. CC-BY-SA

      

      You can see the video in HD on Vimeo.
      
        How Did He Do That?
      
      Well, first of all, I already have a script to keep a local version of the London XML map data (as london.osm) up to date. Each hour, this script:
      
        
          Read the old london.osm file and merge in changes through the OpenStreetMap replication service and write it as london-new.osm.
        
        
          Compare the old file and the new file and write out a changes file to changes/diff-.osc. This is not needed for this to work, but I am interested in this information as well.
        
        
          Move the new file to the old file.
        
        
          Import it into my local PostGreSQL/PostGIS instance.
        
      
      In script form, that looks like:
      #!/bin/bash

NOW=`date +%Y%m%d-%H%M`

export JAVACMD_OPTIONS="$JAVACMD_OPTIONS -Djava.net.preferIPv4Stack=true"
../osmosis-0.39/bin/osmosis --rri --simc --rx london.osm --ac --bb left=-0.563 right=0.28 top=51.68 bottom=51.26 --wx london-new.osm
../osmosis-0.39/bin/osmosis --rx london-new.osm --rx london.osm --derive-change --write-xml-change file="changes/diff-$NOW.osc"
mv london-new.osm london.osm
../try4/bin/osm2pgsql/osm2pgsql -S ../try4/bin/osm2pgsql/default.style --slim -d gis -C 2400 london.osm #-e 15 -o expiry.log


      And from that I then can render maps of London quite easily:
      cd /backup/osm/try4/bin/mapnik
./generate_tiles_london.py | grep -v " exists"


      In order to make a snapshot of South Kensington, or at least, the area that we were going to map in, I added another task. I placed it just after the moving of the old file to the new file. This task cuts out the South Kensington area and puts it in a file south-ken/south-kensington-.osm:
      ../osmosis-0.39/bin/osmosis --rx london.osm --bb left=-0.18256 right=-0.16853 top=51.49426 bottom=51.48733 --wx south-ken/south-kensington-$NOW.osm


      This generated about 100 files in the south-ken directory, where each XML file contained a snapshot of the map on each specific hour.
      I then ran the following script to create a image of each of those map files:
      #!/bin/bash

export MAPNIK_MAP_FILE=/backup/osm/try4/bin/mapnik/osm.xml
mkdir -p images
LAT1=-0.18256
LON1=51.48833
LAT2=-0.16853
LON2=51.49326

for i in south*osm; do
  /backup/osm/try4/bin/osm2pgsql/osm2pgsql -S /backup/osm/try4/bin/osm2pgsql/default.style --slim -d gis -C 2400 $i
  /backup/osm/try4/bin/mapnik/generate_image.py ${LAT1} ${LON1} ${LAT2} ${LON2}

  mv image.png images/${i}-0.png
  LAT1=`echo "${LAT1} + 0.000005" | bc -q`
  LON1=`echo "${LON1} + 0.00001" | bc -q`
  LAT2=`echo "${LAT2} - 0.00001" | bc -q`
  LON2=`echo "${LON2} - 0" | bc -q`

  /backup/osm/try4/bin/mapnik/generate_image.py ${LAT1} ${LON1} ${LAT2} ${LON2}
  mv image.png images/${i}-1.png
  LAT1=`echo "${LAT1} + 0.000005" | bc -q`
  LON1=`echo "${LON1} + 0.00001" | bc -q`
  LAT2=`echo "${LAT2} - 0.00001" | bc -q`
  LON2=`echo "${LON2} - 0" | bc -q`
done


      I had to modify the generate_image.py script from the Mapnik distribution slightly to allow the bounding box to be passed in through the command line. I also set the image width and height to 1280 by 720:
      ...
ll = [];
for arg in sys.argv:
    ll.append(arg)
ll.pop(0);
for index, arg in enumerate(ll):
    ll[index] = float(arg)

z = 15
imgx = 1280
imgy = 720

m = mapnik.Map(imgx,imgy)
...


      Once all the PNG files were created, I converted them to the JPEG format and ran mencoder to generate the AVI film:
      for i in *png; do convert -quality 100 $i $i.jpg; done
mencoder "mf://*.jpg" -mf fps=10 -o test.avi -ovc lavc -lavcopts vcodec=msmpeg4v2:vbitrate=7200


      And this video I then uploaded to Vimeo. Voilà! One OpenStreetMap mapping party progress animation!
      
        
        
      

      
      
    

  




OpenStreetMap Quality Assurance with a Garmin GPS
Tue, 05 Jul 2011 09:43:00 +0000

  
    
      OpenStreetMap Quality Assurance with a Garmin GPS

      
      
         London, UK
        Tuesday, July 5th 2011, 10:43 BST
      
      I've recently bought myself a new Garmin eTrex Vista HCx GPS to replace my older Garmin eTrex Legend unit. Because I don't want to shell out money for outdated maps, I have spend some time building my own map images for the Garmin with mkgmap and OpenStreetMap data. I will write more on that later.
      
      Because most of the data in OpenStreetMap has been surveyed by volunteers, it is always possible that things are missing or done incorrectly. There are quite a few different tools on-line for quality assurance of the OpenStreetMap data, but I will be looking at two of them only.
      
        OSM Analysis
      
      First of all there is ITO World's OSM Analysis. This tool compares the OpenStreetMap data in the UK to the Ordnance Survey's Locator data. The latter is supposed to be the authoritative source of road names in the UK. For each of the 417 areas in the UK, the tool produces two lists; for example the one for Brent is here: http://www.itoworld.com/product/data/osm_analysis/area?name=Brent.
      The first column lists all the corrections that Ordnance Survey ought to make in an updated version because one of OpenStreetMap's surveyors found it incorrect. The second column lists all the streets that are missing from OpenStreetMap.
      Because it would be really useful to have this sort of data available on a hand-held GPS to be able to notice any of the highlighted missing roads on the go, I set out to convert this data to a map file suitable for the Garmin devices. I've published the scripts on github so that you can try it for yourself, but I've also put a ready made osmanal.img file on-line for download at http://derick.dev.openstreetmap.org/. In the near future I will use cron to run this conversion about once a week automatically.
      
        Musical Chairs
      
      Another tool that compares OpenStreetMap data with Ordnance Survey's Locator data is OS Locator Musical Chairs. Documentation on this project is available on the OpenStreetMap wiki.
      The author of the Musical Chairs tool has provided a data dump highlighting all the disagreements it finds (including missing roads) while comparing OpenStreetMap data against Ordnance Survey's Locator data.  I've written a tool that converts this data to another Garmin map image files with the scripts that I have published at github. With those scripts you can make your own image files, but like with the OSM Analysis I have made a ready-made file muschair.img available at http://derick.dev.openstreetmap.org/.
      
        Using the map image files
      
      Before you can use the two Garmin map image files, you either need to convert it to a stand-alone gmapsupp.img file by running:
      java -jar /backup/osm/try4/bin/mkgmap-r1946/mkgmap.jar --gmapsupp osmanal.img


      or:
      java -jar /backup/osm/try4/bin/mkgmap-r1946/mkgmap.jar --gmapsupp muschairs.img


      These commands produce a gmapsupp.img file that you can upload to your Garmin device. If you do this, you will not have any road data.
      You can also merge this file with all your other maps to keep the road data, by running something like:
      java -Xmx2048M -jar /backup/osm/try4/bin/mkgmap-r1946/mkgmap.jar \
    --gmapsupp /tmp/music.img /tmp/osmanal.img \
    UK/gmapsupp.img \
    contours/gmapsupp.img


      Which merges the Musical Chairs image file (music.img), the OSM Analysis image file (osmanal.img), the UK maps (UK/gmapsupp.img) as well as my UK contours line file (contours/gmapsupp.img). This again produces a gmapsupp.img file that you can upload to your Garmin GPS.
      For further mkgmap usage details I would like you to point to it's page on the OpenStreetMap wiki: http://wiki.openstreetmap.org/wiki/Mkgmap or more generally to http://wiki.openstreetmap.org/wiki/OSM_Map_On_Garmin. I am also intending to publish my UK and contour map creation scripts after I've streamlined this process.
      And now the maps have been on my GPS for a few days, I've identified and fixed a "disagreement" from Musical Chairs and a few others from the ITO World OSM analysis tool.
      
        
        
      

      
      
    

  




Spatial Indexes: Solr
Tue, 14 Jun 2011 08:04:00 +0000

  
    
      Spatial Indexes: Solr

      
      
         London, UK
        Tuesday, June 14th 2011, 09:04 BST
      
      In two previous articles I introduced the spherical Earth model, using SQLite as a geographical data storage and using MySQL as a geographical data storage.  In this article we're going to have a look at importing the data into something else than a relational database: the search platform Solr. (Yes, I know I've skipped PostgreSQL, but I'll come back to that).
      
        Solr
      
      Solr is "the popular, blazing fast open source enterprise search platform from the Apache Lucene project." Since version 3.1, Solr has support for spatial search, including geospatial search. It can store coordinates in two different field types: solr.PointType for n-dimensional points, and solr.LatLonType for a two-dimensional point for geospatial search. The main difference is that with solr.PointType, calculations are done according to the flat Earth model, and solr.LatLonType does calculations for a spherical Earth model—which is just what we need.
      In this example we will use the default Solr configuration, unless noted otherwise. First we download Solr 3.1 and then untar it with tar -xvzf
apache-solr-3.1.0.tgz.  Then we edit example/solr/conf/solrconfig.xml to comment out the section on Query Elevation Component so that it looks like:
      


      See here for the reason.
      
        Setting Up Solr
      
      Before we can start using Solr, we need to define a schema. This schema is quite analogous to a database schema, except that Solr only has one table.  By default Solr comes with a schema defining lots of fields in example/solr/conf/schema.xml.  I've changed the whole  section to look like:
      
        
        
        
        
        
        
        
        
        
        
        



      I've defined specific fields for the type of source (1=node, 2=way/area), the name of the amenity (name of the pub, hospital, etc), what sort of amenity it is (cafe, bank, etc), the address and postcode, the phone number, the type of cuisine as well as the location. For the location I've also defined two subtypes: location_0_coordinate and location_1_coordinate. Solr requires this for multi-dimensional types such as solr.LatLonType.
      The types for each field are also defined in schema.xml. text is for a string that is analysed and tokenized (broken down in smaller parts) so that you can search in only parts of the text, string is for a string of characters that is not analysed or tokenized, and location contains our latitude/longitude point. Each of those types are associated with a specific Java class with an entry such as:
      


      The subFieldSuffix in this line configures the suffix in the fields location_0_coordinate and location_1_coordinate.
      Now we have changed the default schema, we can start Solr with: cd example &&
java -jar start.jar.
      On the PHP side, we need to do a bit of set-up as well. I've been using the Solr PECL extension to talk to Solr which can be installed with pecl
install solr.  Please do not forget to add the extension to php.ini with extension=solr.so. Check whether it is installed correctly with: php --ri
solr.
      
        Importing
      
      For our import, we need to change the script we used previously quite substantially.  Instead of just using London, I've also downloaded the whole of the UK from Geofabrik and extracted all amenities from the dataset with Osmosis:
      wget http://download.geofabrik.de/osm/europe/great_britain.osm.bz2

./osmosis-0.39/bin/osmosis -v 5 \
--read-xml file=great_britain.osm.bz2 \
--tf reject-relations \
--tf accept-nodes amenity=* \
--tf reject-ways \
outPipe.0=POI \
\
--read-xml file=great_britain.osm.bz2 \
--tf reject-relations \
--tf accept-ways amenity=* \
--used-node outPipe.0=area \
\
--merge inPipe.0=POI inPipe.1=area \
--write-xml file=great_britain_amenity.osm


      This is going to take quite some time, and will result in a 500MB XML file.
      The main logic of the importing script remains the same, but instead of adding each item to a database, we now build a Solr import document and add it to the search index.  I've also split the address and postcode into two separate fields, and added the amenity field to store all the different amenities. For now, I'm filtering out post boxes, parking areas and grave yards. The script, parsepoi-solr.php is available here.
      After downloading, and renaming the downloaded file to parsepoi-solr.php we can run the script with:
      php -dmemory_limit=1G parsepoi-solr.php great_britain_amenity.osm


      When done, this should have imported more than 140.000 items into Solr. You can verify this, by going to http://localhost:8983/solr/select/?q=*:*&rows=0
      
        Querying
      
      As you can see, you can query Solr through its HTTP interface quite easily. In fact, all queries to Solr are done over HTTP. The Solr extension however abstracts this away from you for at least the import. I couldn't find any functionality in the extension to do spatial queries, so we'll do it manually.
      If we look at the URL above, we can divide it into different parts:
      
        
          http://localhost:8983/solr/: The base URL for this Solr instance.
        
        
          select/?: The action for running search queries.
        
        
          q=*:*: q is the search query parameter, and its value, *:* means all fields (the first *) and all possible values (the second *). In this case that means return everything.
        
        
          rows=0: Do not return any rows, so that we just get a count.
        
      
      We get back the result as an XML file. If we want JSON instead, we can simply append &wt=json. In case you want a CSV file, append &wt=csv.
      In a first example, all we want to do is to return all cafes within 100 meter. We construct the search query as follows:
      
        
          http://localhost:8983/solr/select/?: The base URL with search action.
        
        
          q=amenity:cafe: We search in the field amenity and look for the value cafe.
        
        
          fq={!geofilt}: We set a query filter to geofilt_. This filter restricts the result set according to the location (pt) and the maximum distance from this location (d) in km.
        
        
          sfield=location: The field to use for location information.
        
        
          pt=51.5375,-0.1934: The point that we center our search around.
        
        
          d=0.1: The maximum distance in kilometers.
        
        
          wt=csv: The format to return, in our case, CSV.
        
      
      Together this makes the full GET request: http://localhost:8983/solr/select/?q=amenity:cafe&fq={!geofilt}&sfield=location&pt=51.5375,-0.1934&d=0.1&wt=csv
      And the result is:
      id,phone,cuisine,location,address,location_1_coordinate,name,amenity,type,location_0_coordinate,postcode
w62088838,,,51.53750834,-0.19329616,75 Kilburn High Road,-0.19329616,Costa,cafe,2,51.53750834,
w78337118,,,51.53828298,-0.19410346,101 Kilburn High Road,-0.19410346,Caffè Nero,cafe,2,51.53828298,NW6 6JE
w105467205,,,51.537555925,-0.192445525,274 Belsize Road,-0.192445525,Belsize Cafe,cafe,2,51.537555925,NW6 4BT
w105467209,+44 207 3724002,,51.537624071429,-0.19228221428571,270 Belsize Road,-0.19228221428571,Lord Jim,cafe,2,51.537624071429,NW6 4BT
w107710475,+44 20 76245736,,51.53695276,-0.19263662,2 Kilburn Bridge,-0.19263662,Famished Cafe,cafe,2,51.53695276,NW6 6HT
w107710482,,,51.53717868,-0.19288912,Kilburn Bridge,-0.19288912,Mike's,cafe,2,51.53717868,NW6 6HT
w107710490,+44 20 76246942,,51.53732946,-0.1930577,12 Kilburn Bridge,-0.1930577,La Dolce Vita,cafe,2,51.53732946,NW6 6HT


      Which fields are returned can be configured. In this case, we are only interested in the location and name of each amenity, so we restrict the number of returned fields with: fl=id,location,name. The result now becomes:
      id,location,name
w62088838,51.53750834,-0.19329616,Costa
w78337118,51.53828298,-0.19410346,Caffè Nero
w105467205,51.537555925,-0.192445525,Belsize Cafe
w105467209,51.537624071429,-0.19228221428571,Lord Jim
w107710475,51.53695276,-0.19263662,Famished Cafe
w107710482,51.53717868,-0.19288912,Mike's
w107710490,51.53732946,-0.1930577,La Dolce Vita


      Sadly, the results do not come back ordered by distance from our starting point. In order to do that, we need to add one more query parameter: sort=geodist() asc.  The full query is now: http://localhost:8983/solr/select/?q=amenity:cafe&fq={!geofilt}&sfield=location&pt=51.5375,-0.1934&d=0.10&wt=csv&fl=id,location,name&sort=geodist()+asc
      And the result:
      id,location,name
w62088838,51.53750834,-0.19329616,Costa
w107710490,51.53732946,-0.1930577,La Dolce Vita
w107710482,51.53717868,-0.19288912,Mike's
w105467205,51.537555925,-0.192445525,Belsize Cafe
w105467209,51.537624071429,-0.19228221428571,Lord Jim
w107710475,51.53695276,-0.19263662,Famished Cafe
w78337118,51.53828298,-0.19410346,Caffè Nero


      It is however not possible to return the distance from our starting point directly with Solr. With the exception being that if you use only the geodist() function as query argument. In this case, make sure to include the special field score in the fl= argument as well. An URL showing this is: http://localhost:8983/solr/select/?q={!func}geodist()&fq={!geofilt}&sfield=location&pt=51.5375,-0.1934&d=0.10&wt=csv&fl=id,location,name,score&sort=geodist()+asc Of course, in this case you can not filter for specific amenities.
      
        Conclusion
      
      In currently released versions of Solr (3.1 at the time of writing), it is not possible to return the distance to the starting point for the search as part of the result set. The Solr developers have already implemented the capabilities to add the results of function queries to result documents for version 4.0. You can verify this by downloading a nightly build and running the query: http://localhost:8983/solr/select/?q=amenity:cafe&fq={!geofilt}&sfield=location&pt=51.5375,-0.1934&d=0.10&fl=id,location,name,score,geodist()&sort=geodist()+asc In this query I've added geodist() to the fl= parameter. The result now includes an extra field called geodist():
      

  
  
        
          w62088838
          Costa
          51.53750834,-0.19329616
          4.352648
          0.0072415726934058865
        
        
          w107710490
          La Dolce Vita
          51.53732946,-0.1930577
          4.352648
          0.030333097323281075
        


      I could not manage to alias it to a different field, or get it to work with the CSV format (wt=csv).
      If the return format is xml (the default) or json, then the result includes, besides the number of found items, the total search time for this query.  Look for this information in the QTime field. In all of the examples here, the QTime has been less than 5, meaning 5 milliseconds. Solr is extremely fast, even with huge amounts of data. I will try to import the amenities of the whole "planet" at some point, and report back with some benchmarking information.
      In the next installment of this series on storing geospatial data, I will be looking at using MongoDB as data store for geographical information.
      
        
        
      

      
      
    

  




What is OpenStreetMap?
Thu, 12 May 2011 08:04:00 +0000

  
    
      What is OpenStreetMap?

      
      
         London, UK
        Thursday, May 12th 2011, 09:04 BST
      
      With people more and more complaining that Google Maps gets it wrong, I often reply that the complainers have a look at OpenStreetMap instead.  But there are a few misunderstandings on what OpenStreetMap actually is.
      
      OpenStreetMap has its main site at http://openstreetmap.org. This shows a rendering of the map that we like to call a "slippy map". This is an online map, that can interactively be zoomed and paned. The slippy map on the site is nothing more than a display of map tiles, static images rendered from OpenStreetMap data as an example of what you can do with it. The real power of OpenStreetMap is not this default rendering, but the possibility to actually access the data behind this map rendering.  And this is where it differs from Google maps, Bing Maps and Yahoo! Maps and various others. OpenStreetMap is the only mapping service that allows you to do something more than just look at pretty map tiles. OpenStreetMap is a database project, with as its main purpose to have an exhaustive database of every street, city, road, building etc on the planet, and not a map display project.
      Now what does that actually mean? For one thing, it means that the sample map tiles (that are displayed in the slippy map) are not meant to be used heavily from applications.  It's all right to show a map tile of your neighbourhood on your web site, but it's not okay to write a mobile application that allows you to scrape map tiles of large areas. Have a look at the general tile usage policy for some background.  The only reason why there is a sample rendering is to aid mappers with improving the data that is the core of OpenStreetMap.
      Obviously, the OpenStreetMap project wants its data to be used. But if you're not allowed to use the map tiles en-masse, then how can you use the data?
      First of all, you can query the OpenStreetMap database in various ways. In most cases, you're going to get an XML file with descriptions of nodes (points of interest, facilities such as toilets, benches, addresses), ways (roads, water ways, transport routes) and areas (buildings, lakes). There is some more info on the (very simple schema) here. This data you can parse and import (as I've shown in a previous article) and do all kinds of cool tricks with, such as finding out the closest pub. This functionality is not available on the main OpenStreetMap site, because "OpenStreetMap does not aim to create and host every webservice possible, but to provide the data so that others can"  make awesome map-related applications.
      Secondly there is a growing number of web sites that uses the OpenStreetMap data to render specialized maps.  There is OpenCycleMap that renders information related to using your bicycle and Öpvnkarte for public transport. There is even a tool that uses the postcode data in OpenStreetMap.  It's quite possible to render your own maps as well, but that I will get back to in a later article.
      
      MapQuest is a good example of a company that uses OpenStreetMap data. They have made their own rendering with its own styles for a visually different map. Their site implements searching for pubs/bars and restaurants for example and MapQuest are also happy with you using their map tiles for your applications. Recently, they have also announced a simple API to generate a static map based on OpenStreetMap with their rendering style.
      The map above is generated by just requesting http://open.mapquestapi.com/staticmap/v3/getmap?size=320,260&zoom=12¢er=51.51,-0.15
      Another major difference with established mapping providers is the ability to edit the map yourself. You can add your own business, houses, addresses, postcodes, your favourite restaurants with either on-line or off-line editors. My list of recent edits is here.
      There is also a whole plethora of special renderings of the map data to assist with debugging issues with the map.  There is the "no name" rendering which shows roads without name, ITO World's osm analysis that shows deviations of road names from another source in the UK, keepright and OSM Inspector that have some automated checks and a tool to overlay postcodes for checking.
      Obviously, there are a few things missing that would make OpenStreetMap an even better resource for mapping related applications and web-applications. Feel free to suggest things that you find missing or lacking in the comments of this article. I'd be happy to hear why you would think that OpenStreetMap is not a good fit for your applications and usage.
      
        
        
      

      
      
    

  




Spatial Indexes: MySQL
Tue, 12 Apr 2011 08:04:00 +0000

  
    
      Spatial Indexes: MySQL

      
      
         London, UK
        Tuesday, April 12th 2011, 09:04 BST
      
      In two previous articles I introduced The spherical Earth model and importing data into SQLite for querying geographical data. In this article we're going to have a look at importing the data into MySQL and finding out how to best store and query spatial data in the databases.
      
        MySQL
      
      MySQL has some support for Spatial Extensions, but it's not particularly useful. For example, there is no way to query anything within the radius around a specific point. Their community pages list a method of implementing it, but it only calculates for a flat Earth model.
      Instead, we'll have to implement our own algorithms again. But first of all, let us import the data into MySQL. First we create the MySQL database:
      derick@whisky:~$ mysqladmin -u root -p create poi
mysql> CREATE TABLE poi(id int, type int, lat float, lon float, name char(255), address char(255), cuisine char(64), phone char(18));


      And then we take the script from the previous article on importing data, and change the third line from:
      $d = ezcDbFactory::create( 'sqlite://' . dirname( __FILE__ ) . '/pois.sqlite' );


      to:
      $d = ezcDbFactory::create( 'mysql://root:root@localhost/poi' );


      Of course, substitute the username and password (root, root) and the database name (poi) to one that suits yourself. We then run again:
      php parseoi.php.txt great_britain_pubs.osm


      And check that our import is complete:
      mysql> SELECT count(*) from poi\G
*************************** 1. row ***************************
count(*): 28147


      Now that we have all the POIs imported, we can query the database. Because MySQL doesn't have working spatial extensions, nor the register-function capabilities from SQLite such as we saw in the previous article, we have to come up with a new solution. The most obvious one is writing a stored procedure for the task, another (non-explored option) would be to write a user defined function. Writing the stored procedure is simple enough; we just have to convert the distance() function to SQL. On the MySQL command line you can enter:
      delimiter //

CREATE FUNCTION distance (latA double, lonA double, latB double, LonB double)
    RETURNS double DETERMINISTIC
BEGIN
    SET @RlatA = radians(latA);
    SET @RlonA = radians(lonA);
    SET @RlatB = radians(latB);
    SET @RlonB = radians(LonB);
    SET @deltaLat = @RlatA - @RlatB;
    SET @deltaLon = @RlonA - @RlonB;
    SET @d = SIN(@deltaLat/2) * SIN(@deltaLat/2) +
        COS(@RlatA) * COS(@RlatB) * SIN(@deltaLon/2)*SIN(@deltaLon/2);
    RETURN 2 * ASIN(SQRT(@d)) * 6371.01;
END//


      Fetching all the pubs within a 250 meter radius around 51.5375°N, 0.1933°W is than as easy as running:
      mysql> SELECT name, address, phone
       FROM poi
       WHERE DISTANCE(lat, lon, 51.5375, -0.1933) < 0.25;


      With the result being:
      | name            | address                        | phone           |
+-----------------+--------------------------------+-----------------+
| Mrs Betsy Smith | Kilburn High Road 77 , NW6 6HY | +44 20 76245793 |
| The Cock Tavern | Kilburn High Road 125          | NULL            |
| The Old Bell    | NULL                           | NULL            |
| The Westbury    | Kilburn High Road 34 , NW6 5UA | +44 20 76257500 |
+-----------------+--------------------------------+-----------------+
4 rows in set (0.72 sec)


      If we want to also include the distance from the centre point in the result, we need to modify the query to:
      mysql> SELECT name, DISTANCE(lat, lon, 51.5375, -0.1933) AS dist
       FROM poi
       HAVING dist < 0.25
       ORDER BY dist;


      with as result:
      | name            | dist                |
+-----------------+---------------------+
| Mrs Betsy Smith | 0.00825473345748987 |
| The Cock Tavern |     0.2420193460511 |
| The Old Bell    |   0.103123484090313 |
| The Westbury    |   0.150294300836645 |
+-----------------+---------------------+
4 rows in set (0.72 sec)


      In both cases, the query takes 0.72 seconds. This is not overly fast, and the main reason for this is that the distance() function has to be called for every row in the table. An index can not be created on this either. However, what we can do is to filter out the results roughly first, by calculating a bounding box of latitude/longitude pairs around the centre point. Calculating the latitude boundaries can be done by:
      d = (distInKm / 6371.01 * 2π) * 360
lat1 = centreLat - d
lat2 = centreLat + d


      In our example that becomes:
      d = (0.250 / 6371.01 * 2π) * 360 = 0.0022483
lat1 = 51.5375 - 0.0022483 = 51.5352.. -> 51.5351
lat2 = 51.5375 + 0.0022483 = 51.5397.. -> 51.5398


      Or in PHP:
      


      We round slightly up and down to combat inaccuracies in the calculations—it is a rough estimate after all.  After adding the index on the lat column, and the index on the lon column, we reissue the query:
      mysql> CREATE index poi_lat ON poi(lat);
mysql> CREATE index poi_lon ON poi(lon);

mysql> SELECT name, DISTANCE(lat, lon, 51.5375, -0.1933) AS dist
       FROM poi
       WHERE lat BETWEEN 51.5351 AND 51.5398
       HAVING dist < 0.25;


      Which returns the same result as before, but faster:
      | name            | dist                |
+-----------------+---------------------+
| The Westbury    |   0.150294300836645 |
| The Old Bell    |   0.103123484090313 |
| Mrs Betsy Smith | 0.00825473345748987 |
| The Cock Tavern |     0.2420193460511 |
+-----------------+---------------------+
4 rows in set (0.01 sec)


      We can pre-filter the result set even more, by also limiting on the longitude boundary. This involves a few more calculations than for the latitude boundaries. The distance in degrees longitude that belongs to a distance in km depends on the latitude of the location. So first we need to calculate the latitude boundaries as we have done above, and with that information calculate the longitude boundaries.
      
      In the first step (red), we calculate the northern and southern boundaries of the circle. We then calculate the western and eastern boundaries for the northern boundary (green), and southern boundary (blue).
      In PHP this algorithm becomes:
      


      If we use both functions we calculate as boundaries:
      


      Which returns the query to execute:
      SELECT name, DISTANCE(lat, lon, 51.5375, -0.1933) AS dist
FROM poi
WHERE lat BETWEEN 51.535251699514 AND 51.539748300486
  AND lon BETWEEN -0.19691479627963 AND -0.18968520372037
  HAVING dist < 0.25
  ORDER BY dist;


      If we run this query, we get as result:
      | name            | dist                |
+-----------------+---------------------+
| Mrs Betsy Smith | 0.00825473345748987 |
| The Old Bell    |   0.103123484090313 |
| The Westbury    |   0.150294300836645 |
| The Cock Tavern |     0.2420193460511 |
+-----------------+---------------------+
4 rows in set (0.00 sec)


      As you can see, we are getting the result a lot faster now—0.00 sec vs 0.72 sec.  Please do note, that if we would just have used the boundaries without using the HAVING dist < 0.25 clause, we would have gotten one extra result that is slightly too far away (0.281 km):
      | name            | dist                |
+-----------------+---------------------+
| Mrs Betsy Smith | 0.00825473345748987 |
| The Old Bell    |   0.103123484090313 |
| The Westbury    |   0.150294300836645 |
| The Cock Tavern |     0.2420193460511 |
| Bar Hemia       |   0.281138534935208 |
+-----------------+---------------------+


      
        Conclusion
      
      In this article we saw how we can use a MySQL stored procedure to find our pubs and bars within a certain distance from a central location.  In the next part, we will be looking on how to solve the same problem with PostgreSQL.
      
        
        
      

      
      
    

  




Spatial Indexes: Fetching Data/SQLite
Thu, 31 Mar 2011 08:17:00 +0000

  
    
      Spatial Indexes: Fetching Data/SQLite

      
      
         London, UK
        Thursday, March 31st 2011, 09:17 BST
      
      In a previous article I introduced 'The Flat Earth Model' and the 'The spherical Earth model'. In this article we're going to have a look at fetching a data set and importing them into a SQLite database to query from PHP. What better data set is there to import than all of the UK's pubs?
      
        Getting the Data
      
      In order to get a suitable data set we are going to use the data from the OpenStreetMap project. This project is mainly concerned with making an open map of the entire globe, but it also contains a vast database of points-of-interest (POI). OpenStreetMap contributes seems to be a big fan of pubs, and hence they are mapped really well. POIs can either be stored as a node (a single point with a geographical location) such as The Long Acre or as a closed way (an ordered collection of nodes where the first and last node are the same), such as Brondes Age.
      I will demonstrate two methods to fetch the pubs in an XML format containing nodes and ways.  The first method is with XAPI. XAPI is an interface to the OpenStreetMap database to allow users to query and filter items. In order to fetch data through it, you specify a bounding box and a predicate. A bounding box specifies the most Eastern, Southern, Western and Northern coordinates. The UK has roughly as bounding box 9.05°W, 48.77°N, 2.19°E, 58.88°N, or short: -9.05, 48.77, 2.19, 58.88. Fetching the data can simply be done by querying the server with wget:
      wget -O pubs.xml 'http://xapi.openstreetmap.org/api/0.6/*[amenity=pub][bbox=-9.05,48.77,2.19,58.88]'


      This is going to take a long time; and will most likely just not work or time-out. The current XAPI server, written in an obscure programming language called MUMPS, is extremely unreliable and is really slow. A new version of XAPI build in Java is on the way, but right now it's limited to 10 square degrees.
      Luckily, there is an alternative in the form of parsing (an extract of) the planet file. The planet file is an enormous database dump of OpenStreetMap's data. The people at Geofabrik have extracts for specific parts of the world at http://download.geofabrik.de/osm/. From them I downloaded the europe/great_britain.osm.pbf file (274 MB) for my example. With a tool called Osmosis we can then filter out all pubs into a similar formatted XML file. I am only giving the command to do our task at hand, but a detailed usage guide for Osmosis is available too. The command runs in about four minutes and looks like:
      ./osmosis-0.38/bin/osmosis -v 5 \
\
--read-pbf file=great_britain.osm.pbf \
--tf reject-relations \
--tf accept-nodes amenity=pub,bar \
--tf reject-ways \
outPipe.0=POI \
\
--read-pbf file=great_britain.osm.pbf \
--tf reject-relations \
--tf accept-ways amenity=pub,bar \
--used-node \
outPipe.0=area \
\
--merge inPipe.0=POI inPipe.1=area \
--write-xml file=great_britain_pubs.osm


      
        Importing the Data
      
      The resulting XML file has two important elements: nodes and ways. The XML for The Long Acre looks like:
      

  
  



      Important here are the latitude (51.51°N) and longitude (0.12°W) and of course, the name of the pub in the tag element.
      And the XML for Brondes Age (a way) is a bit more complex, and looks like:
      

  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  



      There is no latitude and longitude, but instead there are references to nodes (the nd elements). There is also a large collection of descriptive tags, such as addr:housenumber and contact:phone. In order to calculate the latitude and longitude of Brondes Age we can either take the correct approach by calculating the centroid or we can simply take the average latitude and longitude of all the nodes. To make things simple, I will take the simple approach.
      In order to find the latitude and longitude of all the nodes, we simply scan through the file again and find all the nodes that correspond to the ref attribute of each nd element. We should disregard the last one of each way though, as it is the same as the first one.
      
        SQLite
      
      To start, we will use a very simple RDBMS: SQLite. We will still have to define a database schema. We can simply do that with:
      derick@whisky:~$ sqlite pois.sqlite
sqlite> CREATE TABLE poi(id int, type int, lat float, lon float, name char, address char, cuisine char, phone char);


      The importing of data then can be done by this "simple" script (after adjusting the path to eZ Components/ Zeta Components) with:
      php parsepoi.php.txt great_britain_pubs.osm


      After running the import script, there should be about 28000 POIs in the database.
      
        Querying the Data
      
      Once we have imported the POIs into our SQLite database, we are ready to query them. SQLite does not have a very extensive set of functions so we can not do the calculation in the query directly. Just to iterate from the previous article in this series, the formula for calculating the distance in a spherical Earth model is:
      


      One solution would be to query the database, and then use the distance() function to filter out unwanted elements, like:
      createSelectQuery();
$q->select( '*' )->from( 'poi' );
$s = $q->prepare();
$s->execute();

foreach ( $s as $res )
{
  $e = distance( $lat, $lon, $res['lat'], $res['lon'] );
  if ( $e < $wantedD )
  {
    echo sprintf( '%.3f,%.3f %-40s %.2f km away',
      $res['lat'], $res['lon'], $res['name'],
      $e ), "\n";
  }
}
?>


      This will show all pubs in a 250 meter radius around 51.53°N, 0.19°W:
      derick@whisky:/home/httpd/html/test/maps$ php fetch-sqlite-simple.php
51.538,-0.193 Mrs Betsy Smith                          0.01 km away
51.539,-0.195 The Cock Tavern                          0.24 km away
51.537,-0.192 The Old Bell                             0.10 km away
51.537,-0.192 The Westbury                             0.15 km away


      Of course, this is not very efficient as all items are selected from the database, and then filtered out depending on their calculated distance. Luckily, SQLite supports user defined functions written in PHP. This would mean that we can increase performance a bit by letting PHP's internals call the distance function for every row:
      sqliteCreateFunction( 'dist', 'distance' );

// Centre point
$lat = 51.5375;
$lon = -0.1933;

// Distance (in km)
$wantedD = 0.25;

$q = $d->createSelectQuery();

// Use the user defined dist() function as additional column
$q->select( "*, dist($lat, $lon, lat, lon) as e" )
  ->from( 'poi' )
  ->where( "e < $wantedD" );

$s = $q->prepare();
$s->execute();

foreach ( $s as $res )
{
  echo sprintf( '%.3f,%.3f %-40s %.2f km away',
    $res['lat'], $res['lon'], $res['name'],
    $res['e'] ), "\n";
}
?>


      The result is naturally the same as before.
      
        Conclusion
      
      In this installment we have seen how to retrieve information from OpenStreetMap's database and import them into SQLite for querying. In the next installment we will have a look at how to import and query with MySQL and PostgreSQL.
      
        
        
      

      
      
    

  




Spatial Indexes: Calculating Distance
Wed, 09 Mar 2011 12:17:00 +0000

  
    
      Spatial Indexes: Calculating Distance

      
      
         Montréal, Canada
        Wednesday, March 9th 2011, 07:17 EST
      
      During my "Geolocation and Mapping with PHP" talk that I've given a few times I briefly touch on the subject of indexes on data-sets of spatial data. This isn't as simple as just solving Pythagoras theorem and this article is meant to clarify this.
      
        The flat Earth model
      
      Pythagoras theorem can be used to calculate the distance between two points quite easily; you take the square root of the square of the absolute vertical distance plus the square of the absolute horizontal distance; in short:
      d = √(|x1 - x2|² + |y1 - y2|²)


      or in PHP:
      $d = sqrt(pow(abs($x1 - $x2), 2) + pow(abs($y1 - $y2), 2));


      If you take for example London's coordinates (51.50°N, 0.13°W) and Amsterdam's coordinates (52.37°N, 4.90°E) we can calculate the distance with:
      d = √(|-0.13 - 4.90|² + |51.50 - 52.37|²)
d = √(5.03² + 0.87²)
d = √(26.0578)
d ≅ 5.10


      And on a map:
      
      But what does a difference of "5.10°" actually mean? How far is this in useful units, such as meters?
      If we show the whole map of which the above is an extract, we come to:
      
      The distance around the equator, and through the poles is roughly the same, 40.000km (please be aware that the blue line only shows half of it, the other half is going through the anti-meridian at 180°W/E). 5.03° in East-West difference is then about 40 000 ✕ (5.03/360) = 559 km and the North-South difference about 20 000 ✕ (0.87/180) = 97 km. Using those numbers within the Pythagoras theorem we end up with a distance of √(559² + 97²) = 567 km. Although the calculation is correct, the answer is still wrong. The real distance is closer to 360 km.
      
        The spherical Earth model
      
      If we look at the Earth in its original (mostly) spherical 1 shape, then it's clear that 10° longitude (East/West) at 60°N is going to be less of a distance than 10°E/W at the equator. It's actually fairly easy to calculate how much 1° longitude is at 60°N by using cos(60) * 1/360 * 6371 * 2π 2. Or in PHP:
      


      This returns 55.597 km per ° for 60°N. The same distance in degrees on the Equator gives 6371 * 2 * M_PI / 360 = 111.195.
      The following diagram shows ones more that latitudinal degrees always correspond with the same distance in kilometer, whereas longitudinal degrees differ.
      
      In the diagram the line A is a line from 0°N, 90°W to 10°N, 90°W. It has the same length as line B, from 30°N, 90°W to 40°N, 90°W: a 36th of the circumference of the Earth through the poles. Line C, from 0°N, 30°W to 0°N, 20°W has the same length. Line D however, from 50°N, 30°W to 50°N, 20°W is shorter by a factor of cos(50°) ≅ 0.64.
      If we look again at the distance between London and Amsterdam, ignore the differences in latitude and instead pick the average, we see:
      


      Which is a bit shorter than the expected 360km, but that's because we conveniently forgot about the difference in latitude.
      Sadly, we can't use Pythagoras's theorem to calculate the real distance with the latitude difference taken account as well. This is because the theorem is meant for Euclidean geometry, and a sphere does not follow the rules of this geometry. Instead we need to use a formula that is called the great-circle distance formula. The main concept behind it is that a circle is drawn across the whole sphere that connects both the start (point P) as well as the end (point V). Then with that circle the distance can be calculated. The following diagram shows this:
      
      I will spare you how the function is derived, but the distance calculation ends up as being:
      


      If we punch in our original numbers form London (51.50°N, 0.13°W) and Amsterdam (52.37°N, 4.90°E), we calculate the following:
      


      Which gives us the expected result of 358.07 km.
      
        Conclusion
      
      I hope that the above clarified the difference between 2D spatial indexing with the flat Earth model and spatial indexing of geo-located data (the spherical Earth model). In future articles I will go into specific implementations of spatial indexing by traditional databases such as SQLite, MySQL and PostGreSQL; NoSQL databases such as MongoDB and CouchDB; and Solr.
      
        
        
      

      
      
    

    
      
        1
        The Earth is not really a sphere, but an approximation of it. However, doing the same calculations for an ellipsoid can (as far as I know) only be done by approximation. The difference would hardly matter for finding the closest pub.
      
      
        2
        In this article, I've used an average radius of the Earth of 6371 km.