Derick Rethans

Importing OpenStreetMap data into MongoDB

Tue, 11 Jun 2013 08:49:00 +0000

Importing OpenStreetMap data into MongoDB

Tuesday, June 11th 2013, 09:49 BST

In many recent MongoDB related presentations I have used OpenStreetMap data as basis for most of my examples. I wrote a script that imports OpenStreetMap (OSM) nodes, ways and to a lesser extend relations into MongoDB with a specific and optimal schema. I have written about this briefly before in Indexing Freeform-Tagged Data, but now MongoDB received numerous updates to geospatial indexes I find it warrants a new article.

In MongoDB 2.2 and before, the index type that MongoDB used was the 2d type, which was basically a two dimensional flat earth coordinate system index—with some spherical features built on top. MongoDB 2.4 has a new index type: 2sphere. Instead of just being able to index points described by x/y coordinates (longitude/latitude) it now has support for indexing points, line strings and polygons as defined by GeoJSON.

I have modified my import script to use those new types, and also added support for very simple multi-polygons that OpenStreetMap records through its relation tag. The script also creates an indexes on { l: '2dsphere' } (the GeoJson object), { ts: 1 } (the tags), and { ty: 1 } (the type).

The structure it converts a node from OpenStreetMap to looks like:

{
        "_id" : "n26486695",
        "ty" : NumberLong(1),
        "l" : {
                "type" : "Point",
                "coordinates" : [
                        -0.1580359,
                        51.4500055
                ]
        },
        "ts" : [
                "addr:housenumber=97",
                "addr:postcode=SW12 8NX",
                "addr:street=Nightingale Lane",
                "amenity=pub",
                "name=The Nightingale",
                "operator=Youngs",
                "source:name=photograph",
                "toilets=yes",
                "toilets:access=customers",
                "website=http://www.youngs.co.uk/pub-detail.asp?PubID=430"
        ],
        "m" : {
                "v" : NumberLong(5),
                "cs" : NumberLong(11229430),
                "uid" : NumberLong(652021),
                "ts" : NumberLong(1333911628)
        }
}

There are several sections that make up the document:

_id: Is a combination of n and the OSM node id.
ty: Is the type. For nodes, this is always 1.
l: The point's location in GeoJson format. An OSM point is translated to a GeoJson Point feature with an array describing the longitude and latitude.
ts: Are the tags that describe the node. Each tag is stored as a concatenation of its key and its value. This creates both a smaller index and it still allows for exact tag/value matches as well as matching against specfic keys through a regular expression match. For example, we could find the above document with:

db.poiConcat.find( { ts: 'name=The Nightingale' } );

And the index would also be used when we look for all amenities:

db.poiConcat.find( { ts: /^amenity=/ } );
m: Contains meta information that describes the node. The following fields are currently present:
- v: The object's version. This is the version number of the version that was found in the imported file. There is always just one version per object.
- cs: The changeset ID in which this object was last updated.
- uid: The user ID of the OpenStreetMap contributor who uploaded the latest version.
- ts: The Unix timestamp of when this object was last updated.

OpenStreetMap ways are stored in two different types. Unclosed ways are stored as GeoJSON LineStrings (think roads):

{
        "_id" : "w2423886",
        "ty" : NumberLong(2),
        "l" : {
                "type" : "LineString",
                "coordinates" : [
                        [ -0.1044769, 51.508462 ],
                        [ -0.1044093, 51.5106306 ],
                        [ -0.1044139, 51.5107814 ],
                        [ -0.104427, 51.5108453 ],
                        [ -0.1044459, 51.5109208 ],
                        [ -0.1045131, 51.5110686 ]
                ]
        },
…

And closed ways are stored as GeoJSON Polygons (think buildings and parks):

{
    "_id" : "w24257746",
    "ty" : NumberLong(2),
    "l" : {
        "type" : "Polygon",
        "coordinates" : [
            [
                [ -0.0745133, 51.560977 ],
                [ -0.0742252, 51.5609742 ],
                [ -0.0742308, 51.5606721 ],
                [ -0.0745217, 51.5606721 ],
                [ -0.0745133, 51.560977 ]
            ]
        ]
    },
    "ts" : [
        "amenity=park",
        "leisure=park",
        "name=Kynaston Gardens"
    ],
    "m" : {
        "v" : NumberLong(1),
        "cs" : NumberLong(357805),
        "uid" : NumberLong(5139),
        "ts" : NumberLong(1210169336)
    }
}

Both ways and areas (closed ways) will have a ty value of 2, as they both come from a way primitive as stored in OpenStreetMap.

The script is available on GitHub as part of the 3angle repository. The latest version is at https://raw.github.com/derickr/3angle/master/import-data.php and it also requires https://raw.github.com/derickr/3angle/master/classes.php for some GeoJSON helper classes and https://raw.github.com/derickr/3angle/master/config.php where you can set the database name and collection name (in my case, demo and poiConcat).

Map data © OpenStreetMap contributors (terms).

Crowd-serfing

London, UK

Friday, April 12th 2013, 09:27 BST

Yesterday, Google announced that they have made Google Map Maker available in the United Kingdom. Like OpenStreetMap it allows everybody to update and add things to the map. But there is one big difference: With MapMaker you don't get access to the data.

Fellow OpenStreetMapper, Richard Fairhurst, describes this as Crowd-serfing:

Crowd-serfing, n.: when a large corp uses crowd-sourced volunteering for its own financial gain, without giving back. See: @googlemapmaker.

I will never understand why people do work for a commercial company without getting any real benefit back: http://www.bbc.co.uk/news/business-21226623

Unfortunately, today's BBC coverage on the availability of Google MapMaker in the UK read more like a manual on MapMaker than a nicely unbiased piece on crowd-sourced maps. Only after one of the OpenStreetMappers reached out to the journalist that wrote the piece, they added some background on OpenStreetMap:

"The biggest problem with Google Map Maker is that anything people contribute may appear on Google's map, but only Google can get at the underlying data to be able to do anything else with it," said Chris Hill.

"If someone includes a Google map on their web site to show where their business is they may also be showing where their competitors are and they can't change that."

Sure, it's nice to have some roads on the Google Map, but you will never even have full access back to your data, unlike OpenStreetMap where you can download and work with all the data. Even nicer is that often, OpenStreetMap still has better maps than GoogleMaps - for example, have a look at this in North Korea: http://tools.geofabrik.de/mc/?mt0=mapnik&mt1=googlemap&lon=125.74677&lat=39.01863&zoom=14 . And closer to home in the United Kingdom, compare some of the hiking trails in the Peak District: http://tools.geofabrik.de/mc/?mt0=mapnik&mt1=googlemap&lon=-1.99876&lat=53.17827&zoom=15 . Even in places like London, the accuracy of the locations of addresses and points-of-interest (POIs) is often a lot better, as OpenStreetMap doesn't use web site scraping and post-code-centroid locations to place POIs: http://tools.geofabrik.de/mc/?mt0=mapnik&mt1=googlemap&lon=-0.12405&lat=51.50862&zoom=18 . OpenStreetMap mostly relies on surveys, done by individuals (like you!) to verify things are actually there, aided a little by the availability of Bing Maps as background imagery.

One of the things that most people forget, is the terms and conditions that commercial entities state. An except from MapMaker's reads:

"You give Google a perpetual, irrevocable, worldwide, royalty-free, and non-exclusive licence to reproduce, adapt, modify, translate, publish, publicly perform, publicly display, distribute, and create derivative works of the user submission."

Note that it never mentions that you can do anything with the data yourself…

If you are contributing time and knowledge, why not allow yourself to benefit from it as well? I realise that OpenStreetMap might not be as accessible, and the map-tiles on their web site aren't the prettiest, but the real benefit is in the access to the raw data that makes up the images in the map. The OpenStreetMap wiki also has an article on this.

Access to the data allows you to do so many more things. From creating your own fancy map-styles, creating "washable, wearable, all-weather maps designed for the real outdoors" such as SplashMaps, to powering web-sites that show accessibility. If you don't like the way map data is rendered, you can produce something in your own style, just like Nike did with this campaign.

Also, there is nothing better if some of your handiwork shows up in a "best of OSM" poster :-) I would never spend my time adding data to Google's Maps. Instead, I prefer to contribute to OpenStreetMap and do awesome things with the data. In the meanwhile, OpenStreetMap continues to support humanitarian relief efforts in Mali and many other places.

If you're near London, come and join us this summer! (Other places will also run mapping parties).

Iceland trip

London, UK

Monday, March 25th 2013, 09:11 GMT

Aurora Borealis

I have always wanted to see Aurora Borealis (Northern Lights) and I never managed to see it the five years I lived in Norway. The Aurora is caused by the collision of energetic particles coming from the Sun with the atmosphere. Their typical green colour comes from the interaction of the particles with Oxygen atoms in the atmosphere. The auroral mechanism is better explained on Wikipedia. Because the particles are charged, they are directed towards the magnetic poles of the Earth.

Aurora gets more intense, and is visible on lower latitudes, when a geomagnetic storm is in progress. One of their causes is a coronal mass ejection (CME) on the Sun which sends huge quantities of matter and electromagnetic radiation out into space. When CMEs are directed towards Earth they cause a geomagnetic storm which increases the change of Aurora Borealis to occur.

The solar cycle, which is eleven years is a cycle in which the Sun's activity goes from very little activity to a lot of activity and back to very little. During solar maximum there is a lot higher chance of a CME to occur. Originally the most recent solar maximum was forecasted to be in 2010 or 2011 but more recent predictions expect it to be this autumn. For some unexpected reason, Auroras are strongest around the vernal and autumnal equinoxes.

With it being so close to both the vernal equinox and solar maximum we set off to Iceland in the hope to be able to see them. The original plan was to seek them out on the second night of our trip (March 18th) but just before we left I noticed through the SpaceWeather site that the Sun produced a solar flare as well as an Earth-bound CME. Instead of trying our luck the 2nd night we decided to book an extra excursion as chances where very high that we would see the Aurora on our first night (March 17th). The Aurora Buddy application that I had installed on my phone had constantly been warning me about high activity all through Sunday after all. The image above shows the extend of auroral activity on the night of the 17th. Just when we got on the bus to travel to some darker skies the Aurora already showed up in the sky visible through Reykjavik's city lights...

When we got to the viewing location, the sky was fully alight as you can see in this timelapse:

Aurora Timelapse, near Vogar, Iceland on March 17th.

This timelapse shows the auroras over a three minute period with a picture taken every 5 seconds. At a frame rate of 3 frames per second this is sped up 15 times. The show lasted until about midnight, when we headed back to Reykjavik. Some more still photos are available on flickr.

The Golden Circle

Of course, Iceland has much more to offer than just the occasional show of Auroras and the next morning we set off on the Golden Circle—a trip past Iceland's touristic highlights. Our first stop was Þingvellir, the site of Alþingi, the first Icelandic parliament that was founded in 930. It is situated in a rift valley that marks the crest of the Mid-Atlantic Ridge and is part of Þingvellir National Park.

After Þingvellir we proceeded to the geyser Strokkur (Icelandic for "churn"). It's quite spectacular to see a whole lot of water being launched in the air every 4 to 8 minutes.

Our next stop was Gullfoss, a waterfall in the river Hvítá. I had visited Gullfoss in summer many many years ago and this trip's experience was quite different. It was so increadible windy and cold that we could hardly make it to view the waterfall. However, with the ice surrounding it it was quite beautiful:

Smoke and Wind

The last day of the trip consisted of exploring the Rejkjanes peninsula which is a large geothermic area. We visited the Kerið crater, the geothermal fields Seltún and Gunnuhver, the cliffs near Reykjanestá and the "Bridge between two continents". The colours of some of the landscapes were beautiful, but there was a strong cold wind almost everywhere which made us want to go to the car very fast most of the time. Something to re-explore in summer I suppose.

Blue Lagoon

After a good night's dinner, drinks and rest we spend the last morning of our trip with our bottoms in the Blue Lagoon to relax. A perfect ending to a quick, but gorgeous trip in Iceland. We'll be back!

For further photos and timelapses, please see my flickr set.

MongoDB's aggregation framework

London, UK

Tuesday, January 22nd 2013, 09:15 GMT

As part of my preparations for my MongoDB workshop at PHP Benelux, I ran into a nice use case for MongoDB's aggregation framework. As I have already promised to write about it, this seems to be a good time to actually write the article.

The dataset that I am using for the workshop contains of restaurants and other points of interest, extracted from OpenStreetMap data. As an example, one of the documents that I am storing is:

{
        "_id" : "n558797601",
        "type" : NumberLong(1),
        "loc" : [ 4.4577708, 51.1611465 ],
        "tags" : [
                "addr:city=Edegem",
                "addr:country=BE",
                "addr:housenumber=398",
                "addr:postcode=2650",
                "addr:street=Mechelsesteenweg",
                "amenity=restaurant",
                "cuisine=regional",
                "name=La Rosa",
        ]
}

I wanted to find out which cuisines are used in all of the documents of my dataset. Described in a different way: I want all the different cuisine=… tags as used in my dataset. Traditionally you would write a really complex™ Map/Reduce job, but since MongoDB 2.2, there is a new feature called the aggregation framework. The aggregation framework is meant to be an easy way to do fairly complex aggregation jobs.

The idea behind it is that each aggregation job is defined by a pipeline of operators. Each operator does a specific task. There is for example $match, which allows you to restrict which documents pass through. If the document matches the predicate contained in the $match operator, then it is allowed through, and otherwise it is dropped from the pipeline.

On the MongoDB shell, you use the aggregate() shell helper to execute a pipeline of operators, and with PHP you use the MongoCollection::aggregate() method.

In order to let pass all the documents that have the tag amenity=restaurant, you would run on the shell:

db.poi.aggregate( { $match: { tags: "amenity=restaurant" } } );

And in PHP you would use the following script:

demo->poi;
$result = $c->aggregate(
        array(
                array( '$match' => array( 'tags' => 'amenity=restaurant' ) )
        )
);

var_dump( $result['result'] );
?>

If everything goes well, the return value of the aggregate() helper method is an array with two elements: ok with a value of double(1) as well as a result element containing an array of all the documents that made it through the whole pipeline. Because the aggregation framework returns all of its results as one document over the network, the full result is limited to 16MB. There are also memory limits internally, so it is always wise to restrict the data coming through the pipeline with an operator as soon as you can.

Let's try to construct a pipeline to get a list of all the different cuisine=… tags including how often they appear.

$match

The first thing to do is to match all documents that have the cuisine=… tag in the first place. We use the $match operator for that. If a $match operator is the first operator in a pipeline than it can make use of indexes. It is therefore important that you have indexes in place. The $match operator definition that we need is:

$allWithCuisine = array(
        '$match' => array( 'tags' => new MongoRegex( '/^cuisine=/' ) )
);

To fit that into our script, we change it to:

demo->poi;

$allWithCuisine = array(
        '$match' => array( 'tags' => new MongoRegex( '/^cuisine=/' ) )
);

$result = $c->aggregate(
        array( $allWithCuisine )
);

var_dump( $result['result'] );
?>

In my case, this returns an array of 117 documents in the result element.

For each pipeline step, we create a variable such as $allWithCuisine to define the operator, and then add those to the array that is passed to the aggregate() method.

$project

To reduce the amount of data going through the pipeline, our next step is to remove all the fields from the documents that we are not interested in. In fact, we are actually only interested in the tags field. In order to "re-shape" a document into a different structure, we use the $project operator. In its most basic form, it works the same as the $fields argument to MongoCollection::find(). It is a lot more powerful that that, as it supports changing the whole structure of a document, as well as computed fields. Have a look at the $project documentation for some more inspiration.

As we are only interested in the tags field of the documents, we just put that in the projection:

$justTheTags = array(
        '$project' => array( 'tags' => 1 )
);

and modify the aggregate() call:

$result = $c->aggregate(
        array( $allWithCuisine, $justTheTags )
);

$unwind

In order to be able to do some work on individual tags, we need to split up the tags array. The $unwind operator does just that. It is a rather tricky operator to explain, so I will try with an example. Take for example this document:

{
        _id: "n478547159",
        related_ids: [ "n516583937", "n401309937" ]
}

Using the $unwind operator on related_ids removes each document from the pipeline and introduces two new ones. One for each of the related_ids elements. At the same time, it replaces the related_ids array with one of the values. Running { $unwind: '$related_ids' } turns the above document into the following two:

{
        _id: "n478547159",
        related_ids: "n516583937"
}
{
        _id: "n478547159",
        related_ids: "n401309937"
}

In our case, we want a document for each of the elements in the tags array so that we can group on this field later. We introduce our $unwind operator:

$unwindTags = array(
        '$unwind' => '$tags'
);

and add it to our list of pipeline operators:

$result = $c->aggregate(
        array( $allWithCuisine, $justTheTags, $unwindTags )
);

When we run the script now, we get 554 documents in the following form:

…
array (
        '_id' => 'n470071537',
        'tags' => 'amenity=fast_food',
),
array (
        '_id' => 'n470071537',
        'tags' => 'cuisine=burger',
),
array (
        '_id' => 'n470071537',
        'tags' => 'name=C&Ms',
),
…

Because we are only interested in the cuisine=… tag, we use our previously defined $match operator to filter out all the documents that don't have this tag:

$result = $c->aggregate(
        array( $allWithCuisine, $justTheTags, $unwindTags, $allWithCuisine )
);

Which leaves us with 117 documents again.

$group

Now that we have extracted and massaged our data, we are ready to group the documents by their cuisine=… key. The $group operator groups all documents in the pipeline by a key, and allows for computed fields. In our case we want to group by the tags field:

$groupByTags = array(
        '$group' => array( '_id' => '$tags' )
);

Then we add it to our list of pipeline operators:

$result = $c->aggregate(
        array(
                $allWithCuisine, $justTheTags, $unwindTags, $allWithCuisine,
                $groupByTags,
        )
);

Our results includes one document for each distinct $tags value. A small excerpt:

…
array (
        '_id' => 'cuisine=kebab;turkish',
),
array (
        '_id' => 'cuisine=pizza',
),
array (
        '_id' => 'cuisine=fine_dining',
),
…

In order to also have a count for each of the distinct values in an extra count field, we need to modify the $group operator in the pipeline. I have already mentioned that you can have computed fields, and that's what we need here. A computed field attaches an expression to a field name. In this case, we want the count field to increment by 1 each time we find a document with this field—for this we use the $sum operator:

$groupByTags = array(
        '$group' => array(
                '_id' => '$tags',
                'count' => array( '$sum' => 1 )
        )
);

Each document that now comes out of the pipeline looks like:

…
array (
        '_id' => 'cuisine=turkish',
        'count' => 2,
),
array (
        '_id' => 'cuisine=japanese',
        'count' => 7,
),
array (
        '_id' => 'cuisine=italian',
        'count' => 10,
),
…

Other computed fields are also possible. If we want for example to also record which original _id field had a cuisine=… tag, we modify the group operator to add this field as well:

$groupByTags = array(
        '$group' => array(
                '_id' => '$tags',
                'count' => array( '$sum' => 1 ),
                'ids' => array( '$addToSet' => '$_id' )
        )
);

The $addToSet operator adds the original _id value as a new value to the ids array for each grouped cuisine=… tag. When we run the full script with the modified $group operator, we now get documents in the form:

array (
        '_id' => 'cuisine=friture',
        'count' => 3,
        'ids' => array (
                0 => 'n2040116467',
                1 => 'n1701471939',
                2 => 'n1701465430',
        ),
),

Because our requirement didn't really want this ids array, I have removed it from future examples.

$sort

The only thing left to do is now sort our cuisine=… tags with the most used tags first. For this we use the $sort operator. The sort operator works in the same way as the MongoCursor::sort() method and accepts the same arguments. In order to sort by the count field in descending order, we create the pipeline operator as follows:

$sort = array(
        '$sort' => array( 'count' => -1 )
);

And add it to our pipeline:

$result = $c->aggregate(
        array(
                $allWithCuisine, $justTheTags, $unwindTags, $allWithCuisine,
                $groupByTags, $sort,
        )
);

When running our script now, we get a list of all distinct cuisine=… tags ordered by their occurrence:

array (
        '_id' => 'cuisine=regional',
        'count' => 19,
),
array (
        '_id' => 'cuisine=burger',
        'count' => 15,
),
array (
        '_id' => 'cuisine=chinese',
        'count' => 14,
),
…

Conclusion

With this I conclude my introduction to the aggregation framework. You can find the final script here. The documentation is extensive so I would suggest to give it a good read.

I'm going back to preparing my PHP Benelux workshop now!

Where is the Sun?

London, UK

Thursday, January 17th 2013, 09:14 GMT

In a previous article I wrote that I am using my Raspberry PI as status screen showing the weather among things, but I wanted to make the widget that shows the current weather a bit more interesting. Instead of having the background black (for nights) and white (for days) I want to have a better approximation of the lightness of the sky. In order to be able to do this, I need to know: Where the Sun in the sky is? (In Britain, of course that would be: Where in the sky is the Sun behind the clouds.)

With PHP's date_sun_info() function you can easily calculate when the Sun rises and sets, but it's not useful to determine how far above or under the horizon the Sun is. For that, I needed to implement a little bit more maths. I found an excellent tutorial online that explains the formulas that are used to calculate the position of the Sun. The trigonometry and maths go beyond me at the moment though!

I've implemented some of those functions in a simple library, called "astro". You can find it on GitHub at https://github.com/derickr/astro. Right now, it doesn't implement a lot more than just the position of the Sun, but I am intending to implement the rest of the algorithms too.

Of course, just a C-library of some maths isn't very useful if your language of choice is PHP, so I also implemented a tiny PHP extension wrapping the astro library. It's called solarsystem and available on GitHub as well. There is only an earth_sunpos() function so far, but again, I am intending to extend on that.

In order to make use of it, you'll have to run:

git clone git://github.com/derickr/php-solarsystem.git
cd php-solarsystem
git submodule init
git submodule update
phpize
./configure
make
make install

Then you can either add extension=solarsystem.so to php.ini, or run scripts with php -dextension=solarsystem.so yourscript.php. In the tests/ directory of the Git checkout you can find a script called sun-position.php. If we examine that, we will see that for four cities (Johannesburg, London, Longyearbyen and Oslo) we calculate the position of the Sun for every 15 minutes during January 14th, 2013. The main function there is earth_sunpos() which takes a Unix timestamp, as well as the latitude and the longitude of the location for which we want to calculate the Sun's position.

The script produces CSV, that I redirect into a file:

php -dextension=solarsystem.so sun-position.php > sunpos.csv

I then opened this file in LibreOffice and made a pretty graph out of it:

For Longyearbyen (yellow line) it shows that the Sun never rises as it always stays below the horizon. It also shows that the highest point over the horizon is different for London and Oslo—mostly because they are 10° apart horizontally. For London, you can also see that sunrise happens around 08:00 and sunset around 16:20.

The position over the horizon, combined with the weather forecast allows me to calculate the likely lightness of the sky. But that will have to wait to a future blog post.

Tweaking the Logitech R400 presenter tool on Linux

London, UK

Wednesday, January 9th 2013, 09:21 GMT

Updated on April 9th, 2013, after comments by Jim Diamond

For Christmas I received a Logitech R400 presenter tool as a replacement for the php|architect one that has now fallen apart. However, to use it together with my presentation system—pres2, about which I previously wrote—I need it to provide left and right arrow keypresses. By default its left and right buttons generate Prior and Next events in X.

It took me a while to get this to work, so in short, this is how I changed it. First of all, I created the file /etc/udev/logitech-r400 as root with contents:

0x70037 f5
0x70029 f11
0x7003E f11
0x7004B left
0x7004E right

This maps the two scan codes that the Play button (lower-left) both to f11, the empty screen button to f5 and the left and right buttons to the left and right arrow keys.

The first column represents the scancode, which I obtained by first looking up with input event the device was tied to:

stat -t /dev/input/by-id/usb-Logitech_USB_Receiver-event-kbd --printf "%N\n"

Which showed the following for me:

‘/dev/input/by-id/usb-Logitech_USB_Receiver-event-kbd’ -> ‘../event16’

Then with event16 I ran:

/lib/udev/keymap -i input/event16

And pressed all four buttons (and the Play button twice). This then showed up on the screen:

scan code: 0x7004B   key code: pageup
scan code: 0x7004E   key code: pagedown
scan code: 0x7003E   key code: f5
scan code: 0x70029   key code: esc
scan code: 0x70037   key code: dot

After creating the file, to test things I ran:

sudo /lib/udev/keymap input/event16 /etc/udev/logitech-r400

Which showed:

Remapped scancode 0x70037 to 0x3f (prior: 0x34)
Remapped scancode 0x70029 to 0x57 (prior: 0x01)
Remapped scancode 0x7003e to 0x57 (prior: 0x3f)
Remapped scancode 0x7004b to 0x69 (prior: 0x68)
Remapped scancode 0x7004e to 0x6a (prior: 0x6d)

After this ran, the presenter tool now sends the key presses that I want. To make this permanent, I added as root a new file /etc/udev/rules.d/logitech.rules, with as contents:

ENV{ID_VENDOR}=="Logitech*", ATTRS{idProduct}=="c52d", RUN+="keymap $name /etc/udev/logitech-r400"

The changes will now also persist after rebooting as well.

My first possibility to test the new tool will be at PHP Benelux where I will be giving a MongoDB tutorial on January 25th. Tickets are still available.

OpenStreetMap: A Year of Edits 2012

London, UK

Tuesday, January 1st 2013, 00:04 GMT

Just like last year I've analysed all the edits to OpenStreetMap and produced a visualisation. It was a bit trickier this year as I also wanted to visualize the work of the redaction bot as well as the "redaction period" and the switch to ODbL for OpenStreetMap data.

The result is: OpenStreetMap: A Year of Edits (2012):

OpenStreetMap: A Year of Edits (2012)

This animation shows all additions and modifications of nodes (flashes) and cumulative edits (fading to purple) in 2012. Additional visualisations are the redaction bot's work (flying away yellow spheres) and the shift to ODbL (green flash). It gives a good overview of the effort that 1000s of contributors put in to make OpenStreetMap the best source of mapping data.

You can see the video in HD on Vimeo.

I have started to make the code for this available through GitHub, but the code is very much not modularized and it is likely it will only work on my set-up. Expect this to change for next year!

Mappy New Year!

Weather from the Norwegians, sunset and sunrise

London, UK

Friday, December 21st 2012, 11:12 GMT

I have recently acquired a Raspberry PI and as a hacking project I am using it as status screen in the living room. As part of this, it has a weather widget.

I use the weather data from http://yr.no ("the Norwegians") as they provide free weather data for the whole world. For each location, there is a nice XML file with the weather forecast.

The XML file has a forecast for several periods. See for example the first upcoming one:

…

This shows among others, the period (December 21st, 12:00 to 13:00), the symbol to use (number="1"), the expected precipitation and temperature. There is a description of what the symbols mean. The page is in Norwegian, but that's all right, as I can actually read that.

For most of the symbols there is both a day and a night variant of the symbol. And in some cases, there is also a "mørketid" variant - the period in which the sun never comes over the horizon.

In each XML file, the coordinates of the location for which the forecast is are also present:



  
        London
        Capital
        United Kingdom

The location information comes from http://geonames.org, which handily also includes the Olson timezone ID for the location. The timezone ID combined with the time, the latitude and the longitude are enough to calculate sunrise and sunset times with PHP's date_sun_info() function.

For example, to find out the sunset and sunrise for today in London (Europe/London, 51.50833°N, 0.12574°W):

This returns the following array:

Array
(
        [sunrise] => 1356077046
        [sunset] => 1356105238
        [transit] => 1356091142
        [civil_twilight_begin] => 1356074637
        [civil_twilight_end] => 1356107647
        [nautical_twilight_begin] => 1356072046
        [nautical_twilight_end] => 1356110239
        [astronomical_twilight_begin] => 1356069597
        [astronomical_twilight_end] => 1356112687
)

Each value is a Unix timestamp. To display when the Sun rises and sets:

Which shows (for today): Sunrise: 08:04, Sunset: 15:53. The transit value is the time when the Sun is at its highest point, and the different twilight_begin/twilight_end values represent different start and end times of the three different forms of twilight. From the above information, it is also trivial to calculate the length of day: echo $result['sunset'] - $result['sunrise']); (or today in London: 28192 seconds, which is 7h49m52s).

Near the summer and winter solstices there are regions on Earth where the Sun never rises (the afore mentioned "mørketid") or when the Sun never sets. If we take for example Kirkenes in Northern Norway (69.72°N, 30.04°E), we can calculate all the different values with:

 $value )
{
        if ( gettype( $value ) == 'boolean' ) {
                echo $key, ': always ', $value ? 'above' : 'below', "\n";
        } else {
                echo $key, ': ', date( 'H:i', $value ), "\n";
        }
}
?>

Which shows:

sunrise: always below
sunset: always below
transit: 10:58
civil_twilight_begin: 08:48
civil_twilight_end: 13:07
nautical_twilight_begin: 07:03
nautical_twilight_end: 14:53
astronomical_twilight_begin: 05:44
astronomical_twilight_end: 16:11

Indicating that the Sun never rises. During summer (July 22nd, 13:45 CEST), it shows for Kirkenes:

sunrise: always above
sunset: always above
transit: 12:06
civil_twilight_begin: always above
civil_twilight_end: always above
nautical_twilight_begin: always above
nautical_twilight_end: always above
astronomical_twilight_begin: always above
astronomical_twilight_end: always above

Which indicates that the Sun is always above the horizon.

It is important to use the correct timezone in your script, otherwise you get weird results:

Which then shows:

Sunrise: 14:55, Transit: 19:51, Sunset: 00:48

Which is clearly incorrect.

Now we have figured out whether it's day, night or "mørketid" we can show the correct symbol. With "Partly Cloudy" (symbol $nr = 5) we select the symbol as follows:

if ( $result['sunrise'] === false || $result['sunset'] === false )
{
        $symbol = sprintf( "%02dm", $nr );
}
else if ( time() > $result['sunrise'] && time() < $result['sunset'] ) )
{
        $symbol = sprintf( "%02dd", $nr );
}
else
{
        $symbol = sprintf( "%02dn", $nr );
}

Some last notes

If I would have paid more attention, I would have seen in the XML file:

which means I wouldn't have had to calculate the sunrise and sunset.

I would also have seen the var="mf/01n.26" part of:

which means I wouldn't have to do the ugly trick to create the correct symbol in the first place… Of course, this article wouldn't have been as nearly as interesting then.

Happy Southern Solstice!

Weather forecast from yr.no, delivered by the Norwegian Meteorological Institute and the NRK (Weather forecast) — http://www.yr.no/place/United_Kingdom/England/London/

Weather symbols from yr.no (Images) — http://om.yr.no/forklaring/symbol/

Read Preferences wth the MongoDB PHP driver

London, UK

Tuesday, December 18th 2012, 09:41 GMT

Read Preferences are a new Replica Set and Sharding feature implemented by most MongoDB drivers that are supported by 10gen. This functionality requires MongoDB 2.2. In short, Read Preferences allow you to configure from which nodes you prefer the driver reads data from. In a Replica Set environment it is the driver that does the selection of the preferred node, and in a Sharded environment it is the mongos process that routes queries according to the defined Read Preferences.

There are a few different Read Preference types. There is primary (the default) which will make sure the driver (or mongos) only reads from primary nodes. This is the default and guarantees consistent reads. The other types can all possibly returned outdated documents as replication is asynchronous in MongoDB. Other types include primaryPreferred, which reads from a primary, unless no primary is available; secondary, which runs queries on secondaries only; secondaryPreferred, which reads from secondaries first, and if none are available, from the primary node; and the last one is nearest which does not prefer a specific type of node (primary or secondary) but solely selects one of the "nearest" nodes - within 15ms of the closest node.

The PHP driver checks ping times to nodes in a Replica Set about every 5 seconds and keeps this information with the connection manager, about which I wrote in an earlier article.

As an example, let us assume that we have a Replica Set with four nodes and that the driver has established the following ping times:

Primary node a: 2ms
Secondary node b: 23ms
Secondary node c: 1ms
Secondary node d: 7ms
Secondary node e: 3ms

The manager searches the manager for all connections and returns all connections for which a connection matches the Read Preference criteria. For primary or secondary it only includes nodes with that property, for the other three Read Preference types it will return all the connections. The set that is returned for secondaryPreferred is [ a, b, c, d, e ]. The manager then sorts this list according to the Read Preference and the ping time. As we are more interested in secondaries with secondaryPreferred, the sorted list of nodes becomes { c, e, d, b, a }. Node c is first because it has the lowest ping time (1ms). In the next step, all nodes that are more than 15ms slower than the fastest node are eliminated. In our example, we are now left with { c, e, d, a }. In the last step, we select a node from this set. With a Read Preference of secondaryPreferred we eliminate the last node from the list if there is more than one node in the list, and the last node in the list is a primary. The final list to pick a node from is now { c, e, a }. From this remaining list, a random node is picked.

With the PHP driver, you can choose a Read Preference in a few different ways. The simplest way is probably directly in the connection string:

new MongoClient( 'mongodb://a,d/?replicaSet=demoset&readPreference=secondaryPreferred' );

It is also possible to set the Read Preference on a connection, database or collection object. When a read preference is set, it is inherited by every database or connection object that is created afterwards:

setReadPreference( Mongo::RP_SECONDARY_PREFERRED );

$d = $m->demoDb;
/* The Read Preference for $d is 'secondaryPreferred' */
$d->setReadPreference( Mongo::RP_PRIMARY_PREFERRED );
/* The Read Preference for $d has been changed to 'primaryPreferred' */

$c = $d->myCollection;
/* The Read Preference for $c is 'primaryPreferred' */

$c = $m->demoDb->myCollection;
/* The Read Preference for $c is 'secondaryPreferred' */
$d->setReadPreference( Mongo::RP_NEAREST );
/* The Read Preference for $c is now 'nearest' */
?>

In future versions of the driver, we will also allow setting a Read Preference on a by-query base.

Tags

Each node in a Replica Set can be annotated with zero or more tags. These tags are for example dc=us-east, use=reporting or systemtype=big. Tags can be added to an already existing Replica Set configuration as follows:

cfg = rs.conf();
cfg.members[0].tags = { "dc" : "west", "use" : "website" };
cfg.members[1].tags = { "dc" : "east", "use" : "reporting" };
rs.reconfig(cfg);

In this example we add the tags dc and use to the first and second node in the Replica Set. In combination with Read Preferences, those tags allow you to restrict from which node you want to read. Let's go back to our previous example with five nodes, but add tags as well:

Primary node a: 2ms, tags: dc=west, use=website
Secondary node b: 23ms, tags: dc=west, use=website
Secondary node c: 1ms, tags: dc=west, use=analysis
Secondary node d: 7ms, tags: dc=east, use=analysis
Secondary node e: 3ms, tags: dc=east, use=website

When we now combine the Read Preference secondaryPreferred with the Tag Set dc=west, nodes c and d are removed from the candidate list after all possible candidates are found and before the candidate servers are sorted according to their node type (primary or secondary) and their ping time. With the nodes not having the tag dc=west set eliminated, we are left with the set [ a, b, c ]. This set then gets sorted into { c, b, a }. Node b is eliminated because it is too slow and finally node c is picked because a is primary and we prefer a secondary instead.

If none of the nodes matches the giving Tag Set, then a connection can not be found, and the query fails. In order to prevent this, it is possible to specify multiple lists of Tag Sets. For example. In this example we prefer the west data centre to be used over the east data centre. If none of the nodes destined for powering the website are available, then we are fine using any available node.

dc=west, use=website
dc=east, use=website
empty

With the PHP driver, you would configure this in the connection string like:

new MongoClient( 'mongodb://a,b,d/?replicaSet=demoset&readPreference=secondaryPreferred&readPreferenceTags=dc:west,use:website&readPreferenceTags=dc:east,use:website&readPreferenceTags=' );

As you can see, this becomes a really long string, so you can alternatively set options as an array through the second argument to the constructor:

new MongoClient(
    'mongodb://a,b,c',
    array(
        'replicaSet' => 'demoset',
        'readPreference' => 'secondaryPreferred',
        'readPreferenceTags' => array(
            'dc=west,use=website',
            'dc=east,use=website',
            ''
        )
    )
);

The MongoClient::setReadPreference, MongoDb::setReadPreference and MongoCollection::setReadPreference methods accept an array of Tag Sets as second argument:

$m = new MongoClient( 'mongodb://a,c,e/?replicaSet=demoset' );
$c = $m->demoDb->myCollection;
$c->setReadPreference(
    Mongo::RP_SECONDARY_PREFERRED,
    array( 'dc=west,use=website', 'dc=east,use=website', '' )
);

With this I conclude the introduction of Read Preferences. I will write about the new Aggregation Framework in MongoDB 2.2 in an upcoming article.

Debugging Connections with the MongoDB PHP driver

Tokyo, Japan

Tuesday, December 11th 2012, 10:13 JST

In a previous article I already mentioned that the 1.3 version of the MongoDB driver has improved logging functionality to aid with debugging connection issues. I've already briefly introduced MongoClient::getConnections(), but it provides a bit more information than I have already shown. The other improvement are changes to the MongoLog class.

Mongo::getConnections()

Mongo::getConnections() returns information about all the connections that the connection manager knows about. Besides basic information such as the connection's hash ($m->getConnections()[0]['hash']) it also returns the following information:

hash: the connection's identifier; f.e.: string(22) "whisky:27017;-;X;21093".
server: contains an element for each of the elements in the hash, but broken up.
connection: contains information about the connection and server it is connected to. It is an array with the following fields:
- last_ping: timestamp that indicates when the server was last pinged; for example: int(1352479219).
- last_ismaster: timestamp that indicates when the topology of a replicaset was last checked for; in the case of a standalone connection, this is: int(0).
- ping_ms: the time it takes to get to the server and back again in milliseconds, this is very often just 0.
- connection_type: the type of connection. The value is: 1 for a standalone connection, 2 for multiple "mongos" connections and 3 for a replica set connection.
- connection_type_desc: a string representation of the above types.
- max_bson_size: the maximum size of a document, right now, that is int(16777216) (bytes).
- tag_count: the number of tags associated with this mongod server. I will explain what tags are and how you can use them in a future article on Read Preferences.
- tags: an array containing all the tags.

As an example, this is what is returned for my replica set test setup:

getConnections() );
?>

Which returns:

Array
(
  [0] => Array
    (
      [hash] => whisky:13000;seta;X;3955
      [server] => Array                           ← 'exploded' hash
        (
          [host] => whisky
          [port] => 13000
          [repl_set_name] => seta
          [pid] => 3955
        )

      [connection] => Array
        (
          [last_ping] => 1353515082
          [last_ismaster] => 1353515082
          [ping_ms] => 0                          ← ping time
          [connection_type] => 4
          [connection_type_desc] => SECONDARY     ← secondary node
          [max_bson_size] => 16777216             ← 16MB document limit
          [tag_count] => 2
          [tags] => Array                         ← tags
            (
              [0] => dc:west
              [1] => use:accounting
            )
        )
    )

  [1] => Array
    (
      [hash] => whisky:13001;seta;X;3955
      [server] => Array
        (
          [host] => whisky
          [port] => 13001
          [repl_set_name] => seta
          [pid] => 3955
        )

      [connection] => Array
        (
          [last_ping] => 1353515082
          [last_ismaster] => 1353515082
          [ping_ms] => 0
          [connection_type] => 2
          [connection_type_desc] => PRIMARY
          [max_bson_size] => 16777216
          [tag_count] => 2
          [tags] => Array
            (
              [0] => dc:east
              [1] => use:reporting
            )
        )
    )
)

MongoLog

The MongoLog class already existed in earlier versions of the driver, but the debugging information it provided was not really useful. While we were implementing the connection handling, we also overhauled the logging through MongoLog. We tried to improve the categorisation of specific into groups, and also classify them according to importantness. There is also a new MongoLog::setCallback() static method that can be used to catch all messages that pass through MongoLog. In that case, they will no longer show up as PHP Notices.

First, we show the most basic usage where we send all of the drivers messages as PHP Notices:

This returns for example the following messages:

Notice: PARSE   INFO: Parsing localhost:27017 in - on line 7

Notice: PARSE   INFO: - Found node: localhost:27017 in - on line 7

Notice: PARSE   INFO: - Connection type: STANDALONE in - on line 7

Notice: CON     INFO: mongo_get_read_write_connection: finding a STANDALONE connection in - on line 7

Notice: CON     INFO: connection_create: creating new connection for localhost:27017 in - on line 7

Notice: CON     INFO: get_server_flags: start in - on line 7

Notice: CON     FINE: send_packet: read from header: 36 in - on line 7

Notice: CON     FINE: send_packet: data_size: 70 in - on line 7

Notice: CON     FINE: get_server_flags: setting maxBsonObjectSize to 16777216 in - on line 7

Notice: CON     FINE: is_ping: pinging localhost:27017;-;X;8830 in - on line 7

Notice: CON     FINE: send_packet: read from header: 36 in - on line 7

Notice: CON     FINE: send_packet: data_size: 17 in - on line 7

Notice: CON     WARN: is_ping: last pinged at 1353519357; time: 0ms in - on line 7

Notice: REPLSET FINE: finding candidate servers in - on line 7

Notice: REPLSET FINE: - all servers in - on line 7

Notice: REPLSET FINE: filter_connections: adding connections: in - on line 7

Notice: REPLSET FINE: - connection: type: STANDALONE, socket: 3, ping: 0, hash: localhost:27017;-;X;8830 in - on line 7

Notice: REPLSET FINE: filter_connections: done in - on line 7

Notice: REPLSET FINE: limiting by seeded/discovered servers in - on line 7

Notice: REPLSET FINE: - connection: type: STANDALONE, socket: 3, ping: 0, hash: localhost:27017;-;X;8830 in - on line 7

Notice: REPLSET FINE: limiting by seeded/discovered servers: done in - on line 7

Notice: REPLSET FINE: sorting servers by priority and ping time in - on line 7

Notice: REPLSET FINE: - connection: type: STANDALONE, socket: 3, ping: 0, hash: localhost:27017;-;X;8830 in - on line 7

Notice: REPLSET FINE: sorting servers: done in - on line 7

Notice: REPLSET FINE: selecting near servers in - on line 7

Notice: REPLSET FINE: selecting near servers: nearest is 0ms in - on line 7

Notice: REPLSET FINE: - connection: type: STANDALONE, socket: 3, ping: 0, hash: localhost:27017;-;X;8830 in - on line 7

Notice: REPLSET FINE: selecting near server: done in - on line 7

Notice: REPLSET FINE: pick server: random element 0 in - on line 7

Notice: REPLSET INFO: - connection: type: STANDALONE, socket: 3, ping: 0, hash: localhost:27017;-;X;8830 in - on line 7

Notice: CON     FINE: mongo_connection_destroy: Closing socket for localhost:27017;-;X;8830. in Unknown on line 0

Notice: CON     INFO: freeing connection localhost:27017;-;X;8830 in Unknown on line 0

As you can see, that is very verbose information—mostly useful when there is really something going wrong. We might ask you for this when submitting a bug report to the PHP driver's JIRA. In general though, you probably want to restrict the amount of information to something more manageable.

First of all, there are a few different levels that you can set through MongoLog::setLevel():

MongoLog::NONE: no messages at all
MongoLog::WARN: only show warnings, such as connection failures and misconfigurations
MongoLog::INFO: informational messages, such as which options have been found during parsing, which hosts have been found, and which sort of connection the driver thinks it is using—this is in most cases the best level to pick
MongoLog::FINE: diagnostic messages such as debugging for algorithms, and timing messages

The four constants can be used as a bit-field, so the following is possible (but not very useful):

MongoLog::setLevel( MongoLog::WARN | MongoLog::FINE );

The messages are divided into different categories. You can configure which categories are represented in the messages with the parameter to MongoLog:setModule():

MongoLog::CON: Single connection issues and operations
MongoLog::IO: Transferring queries, results and cursor related operations
MongoLog::PARSE: Connection string parsing
MongoLog::RS: Selecting which connection to use while using replica sets.
MongoLog::SERVER: Not used at all

These categories can also be combined as a bit-field. For example, to only show connection and connection selection messages, use:

MongoLog::setModule( MongoLog::CON | MongoLog::RS );

The MongoLog class already existed, but new in 1.3 is the MongoLog::setCallback method. This method allows you to set-up your own callback function to deal with the MongoDB driver's log messages. We use this in the test cases to check whether specific messages have values we are looking for. It also allows you to filter, handle, and for example log messages in various ways.

For example, in the replica set tests, we set the following:

MongoLog::setCallback( function($a, $b, $message) { if (preg_match('/connection: type: ([A-Z]+),/', $message, $m )) { @$GLOBALS['mentions'][$m[1]]++; }; } );

And to make that a bit more readable and avoiding the closure:

function logCallBack( $a, $b, $message )
{
    if ( preg_match( '/connection: type: ([A-Z]+),/', $message, $m ) )
    {
        @$GLOBALS['mentions'][$m[1]]++;
    };
}

MongoLog::setCallback( 'logCallBack' );

The callback function accepts three arguments, the first one representing the module for this message, the second the log level and the third argument is the message. In the above example, we basically only want to find out which connection types the log dumps, and we compare that against what we expect. The values of the module constants that you can compare the first two arguments against are described in the documentation.

With this I conclude this post about new debugging functionality in the driver. Still to come are posts about Read Preferences and the new Aggregation Framework in MongoDB 2.2.

Connection Handling with the MongoDB PHP driver

London, UK

Tuesday, December 4th 2012, 09:07 GMT

The 1.3 release series of the PHP MongoDB driver features a rewritten connection handling library. This is quite a large change and changes how the PHP driver deals with persistent connections and connection pooling.

Connections in the 1.2 series

The 1.2 release introduced connection pooling into the driver. This means that any time the driver needs to use a connection to run queries, it would request a connection from the pool. It would then later return the connection once it was done with it. Being done is defined as "the variable holding the connection object goes out of scope". To illustrate that, take for example the following scripts.

The simplest version:

demo->test;
$c->insert( array( 'test' => 'yes' ) );
?>
← connection is returned to the pool as $m goes out of scope

Inside a function:

demo->test;
        $c->insert( array( 'test' => 'yes' ) );
} // ← connection is returned to the pool as $m goes out of scope
?>

Now, in a few cases, this can result in lots of connections being made. One common occurence is trying to use connections deep inside a model layer-which typically happens if you use ORMs/ODMs with complex structures having a link to your connection object. A simple variant of how this happens is:

Connections in the 1.3 series

In the 1.3 series the connection management is done different. Per worker process (thread or PHP-FPM or Apache worker) the driver now manages connections in the C-part of driver separately from the Mongo* objects. This greatly reduces complexity in the driver. Let's have a look at how the driver handles connection for a basic MongoDB setups with just one node.

When a worker process starts up, the MongoDB driver instantiates a manager to manage your connections. By default of course, this manager is rather sad, as it contains no connections yet.

Upon the first request doing new MongoClient();, the driver creates a new connection, and with this connection it creates a hash to identify this connection. Parameters for this hash includes: the hostname and the port, the process ID (pid), if available the replica set name and in case of authenticated connections, the database name, the username and a hash of the password. We will come back to authenticated connections later. You can quite easily see which hash is created, by calling the MongoClient::getConnections() method:

getConnections()[0]['hash'] );
?>

Which outputs:

string(22) "whisky:27017;-;X;22835"

The - in this output denotes that the connection does not belong to a replica set and the X in this output is a place holder in case no username, database and password are given. 22835 is the process ID of the running script.

This newly created connection is then registered with the manager:

Any time a connection is needed, for either an insert, delete, update, find or command query, the driver asks the manager to find a fitting connection to perform this query. It uses the information from the arguments to new MongoClient() as well as the PID of the current process to find this connection. Because the manager has a connection list per worker process/thread, and PHP's workers all run one request at a time, there is no need to have more than one connection per host. The connection gets reused and reused until the PHP worker ends, or you close the connection forcefully with MongoClient::close().

Replica sets

In a replicaset environment, this works all a bit different. In the connection string to new MongoClient() you need to specify a few hosts, preferrably all that you are aware of, as well as an annotation that you are using a replica set, for example:

$m = new MongoClient("mongodb://whisky:13000,whisky:13001/?replicaSet=seta");

It is important that you specify the replicaSet argument, as without it the driver will assume you are connecting to three different mongos processes.

Upon instantiation, the driver will discover the replica set's topology. The following output shows that all visible data nodes of the replica set will have a connection registered in the manager after the new MongoClient() call:

getConnections() as $c )
{
	echo $c['hash'], "\n";
}
?>

Which outputs:

whisky:13001;seta;X;32315
whisky:13000;seta;X;32315

Notice that the connection string made no reference to the whisky:13000 node. The manager has now registered two connections:

The manager contains a bit more information than just the connection hash and the TCP/IP socket. It also knows which nodes are primary nodes, and how "far away" a specific node is. This script shows how to retrieve this additional information:

getConnections() as $c )
{
	echo $c['hash'], ":\n",
		" - {$c['connection']['connection_type_desc']}, ",
		"{$c['connection']['ping_ms']} ms\n";
}
?>

Which outputs:

whisky:13001;seta;X;5776:
 - SECONDARY, 1 ms
whisky:13000;seta;X;5776:
 - PRIMARY, 0 ms

The driver knows about two types of queries. Write queries, which involve insert, update, remove and commands, and read queries, which involves find and findOne. By default, the manager will always return a connection to a primary node, unless you use the new read preferences. I will be writing a separate post about read preferences, but the basic idea is that they can control from which replica set nodes data is read from. For now, we will only use the setSlaveOkay() equivalent directly in the connection string:

$m = new MongoClient("mongodb://whisky:13000,whisky:13001/?replicaSet=seta&readPreference=secondaryPreferred");

As you can see, this connection string is rather long, which is why the PHP driver also allows you to pass in an array of options as second argument:

$options = array(
        'replicaSet' => 'seta',
        'readPreference' => 'secondaryPreferred',
);
$m = new MongoClient("mongodb://whisky:13000,whisky:13001/", $options);

Per query, the driver will ask the manager for a fitting connection. For write queries, it will always pick a primary node, and for read queries it will pick a secondary node if they are available and not too "far away" (what that means, I'll again explain in my upcoming read preferences article).

Authenticated connections

If authentication is enabled on MongoDB, then the connection hash will start to include an authentication hash. This is to make sure that different scripts that use the same MongoDB server, but different user name/password/database combinations, do not inadvertently use a wrongly authenticated connection. Let's see what the hash turns into if we connect to the admin database with the admin user:

getConnections()[0]['hash'] );
?>

Which outputs:

string(64) "whisky:27017;-;admin/admin/bda5cc70cd5c23f7ffa1fda978ecbd30;8697"

The X that we previously saw, has now been replaced by a string containing the database name admin, the user name admin and the hash bda5cc70cd5c23f7ffa1fda978ecbd30. This hash is created from the user name database and hashed password.

In order for authentication to work the driver needs to have the database name in the connection string (in our case admin), otherwise, it will default to admin.

To use a database after you have established a connection, need to select the database again though. For example when you want to run a query:

$collection = $m->demoDb->collection;
$collection->findOne();

If this database matches the database name in the connection string, or when the database name in the connection string was admin, everything is well. However, if the database name accessed as property of the connection object is different then the one specified in the connection string then the driver has to create a new connection to prevent leaking of authentication between multiple requests. The following example illustrates this:

test2;
$collection = $db->collection;
var_dump( $collection->findOne() );
?>

Which outputs:

Fatal error: Uncaught exception 'MongoCursorException' with message
'whisky:27017: unauthorized db:test2 ns:test2.collection lock type:0 client:127.0.0.1'
in …/mongo-connect-5.php.txt:6

This is of course we have not actually authenticated for the test2 database. If we authenticate, then it works (even though findOne() does not actually find anything), and we can see all the created connections:

test2;
$db->authenticate('user2', 'user2' );
$collection = $db->collection;
$collection->findOne();

foreach ( $m->getConnections() as $c )
{
	echo $c['hash'], "\n";
}
?>

Which outputs:

whisky:27017;-;test/user/602b672e2fdcda7b58a042aeeb034376;26983
whisky:27017;-;test2/user2/984b6b4fd6c33f49b73f026f8b47c0de;26983

And we have two authenticated connections in our manager:

But with one snag. If you have E_DEPRECATED notices turned on, you will see:

Deprecated: Function MongoDB::authenticate() is deprecated in …/mongo-connect-6.php.txt on line 5

The driver can do things much more optimised if you create two MongoClient objects instead:

 false ) );
$mTest2 = new MongoClient( 'mongodb://user2:user2@whisky:27017/test2', array( 'connect' => false ) );

$mTest1->test->test->findOne();
$mTest2->test2->test->findOne();

foreach ( $mTest2->getConnections() as $c )
{
	echo $c['hash'], "\n";
}
?>

With this, I conclude this post on the MongoDB driver's new connection handling. In future posts I will write about the 1.3 release providing (a lot) better debugging information, read preference functionality, and the new aggregation framework that comes with MongoDB 2.2.

Sad T-rex (photo) — http://www.flickr.com/photos/ijammin/5456757507/

Whisky glass (photo) — http://www.flickr.com/photos/feastguru_kirti/5396934673

Lock (photo) — http://www.flickr.com/photos/leehaywood/4141478822

Maps (the non-digital variety)

London, UK

Thursday, November 29th 2012, 13:00 GMT

As you probably know, I am pretty much into OpenStreetMap and digital maps. I have always found (paper) maps, and different visualisations fascinating. I found myself at London GeoMob last Thursday where I encountered two types of non-digital, non-paper maps.

The first one, was a 1800s style rendering of London from Wellingtons Travel — fully hand-drawn. I found them quite beautiful, and would be happy to have one on my wall. Some pictures that I took of it are here:

I am even more excited about a second project: SplashMaps. At some point you probably have been in a situation where you were out and about walking somewhere in a city or the country and you had to find your way. In cities you can often ask people and/or get our your phone, but that doesn't cut it when you are on some remote hill in Scotland. You need a paper map (for more than one reason!). And of course, it is always raining (we're in Scotland after all!). Ordnance Survey will sell you laminated Explorer Maps for this, which have saved me quite a few times. However, they are bulky and the place that you are looking for is either always on a fold, or on the opposite site of the map!

At GeoMob a solution showed up in the form of SplashMaps—they are basically a map printed on cloth, water-proof, but you can still use a marker to mark out your route, wash it, and it is good as new again. Here is a picture of one of their prototypes of the New Forest:

The maps are specifically made with walking and cycling (in national parks) in mind, so it has all the features that are important: paths, bike repair shops, bogs and pubs (and a lot more!). They are currently running a funding round on KickStarter and I would urge you to check them out. I have backed the project and looking forwards to getting my maps for my favourite parks... I do need to think about which ones they are!

Mongo is dead, long live MongoClient

London, UK

Tuesday, November 27th 2012, 15:09 GMT

This afternoon we published version 1.3.0 of the MongoDB PHP driver. Besides a number of bug fixes since RC2 and RC3, this new release also includes a new MongoClient class. This new MongoClient class serves as a replacement for the Mongo class. The old Mongo class is now deprecated and will be removed in a future release, although we are keeping it in place for now because of backwards compatibility reasons. We have already removed it mostly from the documentation, and are working to update all our other material as well.

All of the other MongoDB drivers are making a similar change.

Now, you may be wondering why we are replacing Mongo with MongoClient across the board. The biggest reason is that the new class will have acknowledged writes on by default—or expressed in deprecated wording: MongoClient has safe mode on by default. We are moving away from the word safe though, mainly because the definition of the word has very little to do with its meaning in the MongoDB context. Your data is perfectly safe without safe mode on as well (in fact, that is what journaling is for). The safe option determined whether or not the driver would wait for the MongoDB server to acknowledge a write operation.

Together with the addition of the MongoClient class we will deprecate the safe option to methods handling writes, such as MongoCollection::insert(), MongoCollection::update(), and others. We do provide a new way of controlling how the driver waits for the server to acknowledge writes (or not). The connection string now supports the w option, which you can use as shown in the following examples:

Turning off acknowledged writes:

$m = new MongoClient('mongodb://localhost/?w=0');
Turning on acknowledged writes (the default):

$m = new MongoClient('mongodb://localhost/?w=1');
Turning on replica set acknowledged writes:

$m = new MongoClient('mongodb://localhost/?w=2');

Setting w in the connection strings makes that the default write concern for any write operation. You can of course override this per write operation as well, by specifying the w option:

demoDb->collection->insert(
        [ 'client' => 'awesome' ], // ← document
        [ 'w' => 0 ]  // ← don't acknowledge writes for this insert
);
?>

If the server acknowledges that the write operation could not succeed, for example when you have a duplicate _id key violation, then the driver will throw a MongoCursorException. The driver uses acknowledged writes by default with MongoClient, which means you are required to handle the cases where the write did not succeed and the exception is thrown:

demoDb->collection->insert( [ '_id' => 'awesome' ]);
}
catch ( MongoCursorException $e )
{
        echo $e->getCode(), ': ', $e->getMessage(), "\n";
}
?>

You can either retry the write yourself, or not, depending on the reason why the write did not succeed. In case of a duplicate key violation, you probably do not want to retry. In case a connection could not be made, you probably do want to retry (after a certain timeout).

So to repeat our major change: The new MongoClient class uses acknowledged writes by default.

Address lookups with Leaflet and Nominatim

London, UK

Tuesday, November 20th 2012, 10:52 GMT

I recently wrote a patch for joind.in to add a map of an event's location to the event detail page. With the same patch, I also replaced the location part of the event edit page with a solution that uses JQuery, Leaflet as map API, OpenStreetMap tiles and Nominatim for doing address lookups. This article forms a small tutorial on how to use this same set-up yourself.

The Basics

To start, we create a new directory for our project:

mkdir addresses
cd addresses

Then I downloaded the Leaflet and jQuery libraries and extracted them in the js directory of the project:

mkdir js
curl -L https://github.com/CloudMade/Leaflet/zipball/v0.4.5 -o leaflet.zip
unzip leaflet.zip
mv CloudMade-Leaflet-*/dist/* js
rm -rf CloudMade-Leaflet-*
rm leaflet.zip
curl http://code.jquery.com/jquery-1.8.2.min.js -o js/jquery-1.8.2.min.js

As first step, we are simply going to show a map on a web page. The map is going to be full screen, and will not have any bells and whistles. The code to embed a map is small, but we will separate it into three files for clarity: a CSS file for our styles (site.css), an HTML file for the structure (index.html) and a JS file for all our JavaScript functions (js/map.js).

Let's start with the HTML file:


  
    Leaflet and Nominatim example

This HTML file includes the Leaflet and jQuery libraries, as well as the default CSS file that Leaflet needs. We are also including our own CSS file (site.css):

body {
  margin: 0;
}
div#map {
  width: 100%;
  height: 100%;
}

In the body of the HTML file, we place a

as contained for the map, and then include a JavaScript file that is responsible for embedding them map:

var map;

function load_map() {
  map = new L.Map('map', {zoomControl: false});

  var osmUrl = 'http://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png',
    osmAttribution = 'Map data © 2012 OpenStreetMap contributors',
    osm = new L.TileLayer(osmUrl, {maxZoom: 18, attribution: osmAttribution});

  map.setView(new L.LatLng(51.538594, -0.198075), 12).addLayer(osm);
}

window.onload = load_map;

If you request the index.html page now through the browser, you will see something like:

Adding the Address Search

In order to add an address lookup form, we need to add more HTML. Our HTML will feature an input box (for the address), a submit button, and a place holder to show our results in. We add this code between the

and the

 tags:
      
  
  
  



      To style this, we add the following at the end of our CSS file:
      div#search {
  background-color: white;
  position: absolute;
  bottom: 40px;
  left: 40px;
  width: auto;
  height: auto;
  padding: 10px;
}
div#search input {
  width: 200px;
}
div#results {
  font-style: sans-serif;
  color: black;
  font-size: 75%;
}


      If we reload the page in our browser, we will see something like:
      
      Now the only thing left to do is to implement the addr_search function. In our JS file (js/map.js) we add before window.onload = load_map; the following lines (split over multiple sections in this tutorial):
      function addr_search() {
  var inp = document.getElementById("addr");

  $.getJSON('http://nominatim.openstreetmap.org/search?format=json&limit=5&q=' + inp.value, function(data) {


      This above line uses jQuery's AJAX capabilities to request a URL, parse the JSON result and issue a callback if it worked. We query Nominatim here, with as format json and limiting the result to 5 items. Nominatim also supports other parameters, which are documented here. 
      var items = [];

$.each(data, function(key, val) {
  items.push(
    "" + val.display_name +
    ''
  );
});


      For each of the items in our result, we create an 
 element which has an  containing a call to a JavaScript function (chooseAddr). This function is responsible for re-centering the map according to the picked latitude and longitude. 
          $('#results').empty();
    if (items.length != 0) {
      $('', { html: "Search results:" }).appendTo('#results');
      $('
', {
        'class': 'my-new-list',
        html: items.join('')
      }).appendTo('#results');
    } else {
      $('', { html: "No results found" }).appendTo('#results');
    }
  });
}


      This processes the results that came back from Nominatim. If there are results, we shows those including a Search results: header, and if there are no results, we show No results found.
      Then we need to add one more function, the chooseAddr function which looks like:
      function chooseAddr(lat, lng, type) {
  var location = new L.LatLng(lat, lng);
  map.panTo(location);

  if (type == 'city' || type == 'administrative') {
    map.setZoom(11);
  } else {
    map.setZoom(13);
  }
}


      We simply use the latitude and longitude from the function invocation, and in order to make things slightly nicer we zoom in a bit less if the item type is either a city or an administrative border. As each of the returned results actually includes a full bounding box, we probably can use that to zoom in better, but I will leave that for your own experiments - you'd want the panInsideBounds() method of Leaflet's Map class for that.
      In the end, if we click on the Search button, a list is presented of all our search results:
      
      And after clicking one of the links, we see the map centered on Paris:
      
      The code for this example is available on github in my osm-tools repository at https://github.com/derickr/osm-tools/tree/master/leaflet-nominatim-example



Managing Pull Requests for the MongoDB PHP driver
Mon, 29 Oct 2012 09:22:00 +0000

  
    
      Managing Pull Requests for the MongoDB PHP driver

      
      
         London, UK
        Monday, October 29th 2012, 09:22 GMT
      
      The MongoDB PHP Driver is hosted as an Open Source project through GitHub. Everybody can check-out the repository and provide patches to this. In order to make things easier to manage, even Hannes, Jeremy and me, the maintainers of the extension will only merge code into it through patches.
      To streamline the managing of those patches we are using GitHub's Pull Request (PR) system. Any time we fix a bug, or add a feature, we create a branch of our personal fork and when we are done, we submit a pull request to the master repository at http://github.com/mongodb/mongo-php-driver. Before this pull request gets merged, a few things happen. First of all, we make sure that there is a related issue in JIRA. If there is no issue, then one has to be created. Also we verify the patch against our contributing guidelines - including whether the patch can actually automatically be applied to the branch.
      Once the pull request is deemed okay, we review the code and the test cases that are required to be submitted as part of the pull request. In our case, I usually review Hannes' patches, and Hannes reviews mine. GitHub's PR system allows us to easily add comments to bits of the patch which is mighty useful. Of course, as part of the review process we need to be able to compile the code and run the test cases as well. For this purpose I have written a small bash function to help me out:
      function mongoprfetch()
{
  git checkout master
  git fetch origin pull/$1/head:pr/$1
  git checkout pr/$1
  git rebase master
}


      This function is run on the command line (in the checkout of mongodb/mongo-php-driver) with as arguments its Pull Request number:
      mongoprfetch 206


      The script will checkout master, fetch the PR as it was a branch from GitHub and rebases it against master. This allows us to test whether the PR cleanly applies to the latest code in master. Without the rebasing, the commit history would also look really complex as this figure shows:
      
      When all review comments are resolved, the code compiles and the test cases succeed, it is time to merge the patch. For this, I am using another small bash function to help me out:
      function mongoprmerge()
{
  git checkout master
  git merge --no-ff -m "Merged pull request #$1" pr/$1
  git branch -D pr/$1
  git push
}


      Because we already have rebased against the latest master branch when running mongoprfetch 206, a merge would mean that we lose the ability to see which commits belonged to the PR - as Git would simply use "fast forward":
      
      We find this undesirable and hence we specify the --no-ff option to git merge as you can see in the mongoprmerge function. This produces histories that look like:
      
      The only thing that we have not automated as part of merging pull requests is closing the associated Jira tickets as well.
      Right now we have no specific scripts to handle also the stable branch (v1.2) as we have been working hard to reimplement the connection handling stuff in the PHP driver. I will write more about the changes in the driver, as well as managing a merging to/from a stable branch in future blog posts.