Version	With CC	Without CC	With coverage_enable=0	Without Xdebug
2.1.2	06:44	00:49	not available	00:32
2.2-dev	05:37	00:48	00:46	00:32

Derick Rethans - tag: blog

MongoDB Cursors with PHP

Tue, 22 May 2012 08:15:00 +0000

MongoDB Cursors with PHP

Tuesday, May 22nd 2012, 09:15 BST

Recently I was asked to improve the MongoCursor::batchSize documentation. This began an indepth investigation in how the PHP driver for MongoDB handles pulling data that's been queried from the MongoDB server. Here are my findings.

A MongoCursor is created as soon as you run the find() method on a MongoCollection object, like in:

$m = new Mongo();
$collection = $m->demoDb->demoCollection;
$cursor = $collection->find();

Just calling find() will only create a cursor object, and does not immediately send the query to the server for processing. That is only done as soon as you start reading from the cursor for the first time. Because of this, you can call additional methods on the newly created cursor object that still influence how the query is run on the server. One of such examples is the sort() method that makes the result sort according to its arguments (in this example, by name):

$cursor->sort( array( 'name' => 1 ) );
$result = $cursor->getNext();

When you then call getNext() on $cursor the driver sends to the server the query, and requests to return a default number of documents in the first batch. The default Batch Size is 101. Let's have a look on what's get send on the wire in our simple query for all documents, sorted by name:

The Number to Return is 0, which means to use the default. So even although we only want to fetch one result (getNext() asks the cursor for the next document only), the server returns 101 documents:

The driver stores all 101 documents locally and during the next 100 calls to getNext() the driver will simply return the documents from the local memory. Once getNext() gets called for the 102th time, the driver connects back to the server to request more documents:

// skip the other 100 docs
for ($i = 0; $i < 100; $i++) { $cursor->getNext(); }
// request document 102:
$result = $cursor->getNext();

When the driver asks for more documents separately (i.e., not at the same time it is issuing a query) without a specific batch size, the server fills up 4MB of documents. On the wire, the request for Get More looks like:

and the reply like:

As you can see, the returned data is 4194378 bytes, and the Number Returned is 34673.

Setting your own batch size

You can instruct the driver to use different batch sizes, by using the batchSize() method on the $cursor. In this new example, we use the batchSize() method to request 25 documents per round trip to the server:

$cursor = $collection->find()->sort( array( 'name' => 1 ) );
$cursor->batchSize(25);
$result = $cursor->getNext();

When we run this script, we will see the following on the wire:

As expected, the Number to Return is now 25. During iteration, all query results are returned from the server to the driver in batches of 25 documents:

// retrieve another 25 documents to trigger the getMore
for ($i = 0; $i < 25; $i++) { $cursor->getNext(); }

Which creates this query:

And this reply:

As you can see, another 25 documents are returned, starting from the 25th document.

Using limit

Besides batchSize(), the driver also supports limit(). limit() is something that exclusively happens on the client side. It is just a counter that restricts how often the cursor can iterate over the data. However, the driver also uses the value passed to the function to ask for only the amount of documents that it still needs to fetch. This means that if we run this script, the driver will request 50000 documents:

$cursor = $c->find()->sort( array( 'name' => 1 ) );
$cursor->limit( 50000 );
$res = $cursor->getNext();

On the wire we'll see:

Sadly, not all 50000 documents fit in the first reply (as every reply is limited to 4MB) and the server replies that it has only returned 34678 documents:

The driver now calculates how many documents it still needs to fullfill it's limit of 50000: 50000 - 34678 = 15322. It then requests those images with a Get More query:

It is also possible to combine limit() and batchSize().

Combining limit and batchSize

If we just use limit() then the driver will always try to fetch as many documents it can to full-fill the limit. This sometimes means it will fill up all 4MB of the maximum allowed reply packet, especially if your limit is set high enough (in our example, that's more than 34678 documents). That's not often what you want, and you can use a combination of limit() and batchSize() to fix it. If we want to query at most 128 documents with at most 50 documents per batch, we can specify that as:

$cursor = $c->find()->sort( array( 'name' => 1 ) );
$cursor->limit( 128 )->batchSize( 50 );
$res = $cursor->getNext();
// retrieve the other 127 documents that we still want
for ($i = 0; $i < 127; $i++) { $cursor->getNext(); }

On the wire, we'll see the following exchange:

Initial query:

First 50 documents:

First Get More for the second batch of 50:

The second batch of 50 documents:

The second and last Get More for a batch of 28 (128 - 50 - 50 = 28):

And the last batch of 28 documents returned:

I wouldn't quite suggest you use a small batch such as 50 though as it would incur lots of round trips from and to the server. In some cases a small batch size makes sense.

Take for example a situation where you need to process 100.000s of documents and the processing time for each item is 2 seconds. In the first batch, the driver will pull 101 documents, which in total will take 202 seconds. When the driver attempts to fetch the next batch with Get More, it fails because the default cursor time-out (on the client size) is only 30 seconds. Setting your batch size to 5 in this case avoids your cursor from timing out. You can of course also change the cursor timeout. Do not use the immortal flag though, as that means something else (see NoCursorTimeout at the MongoDB Wire Protocol documentation).

One More Thing

Setting the batch size to 1, as well as setting a negative batch size has a special meaning for the MongoDB server. In both cases, it instructs the server to return up to the absolute value of the requested and then terminate the cursor, allowing no further documents to be fetched. That means that this script doesn't do what you expect it to:

$cursor = $c->find()->sort( array( 'name' => 1 ) );
$cursor->batchSize( 1 )->limit( 10 );
$cursor->getNext();
var_dump( $cursor->getNext() );

The var_dump() for the second getNext() will always return NULL. However, if you set a batch size of 2, it works just like you would expect:

$cursor = $c->find()->sort( array( 'name' => 1 ) );
$cursor->batchSize( 2 )->limit( 10 );
$cursor->getNext(); // item 1
$cursor->getNext(); // item 2
var_dump( $cursor->getNext() ); // item 3

Now let's set a batch size of -2 and see what happens on the wire. The request is:

As you can see, the driver really sends -2 to the server, indicating that negative batch sizes are handled on the server side.

And the reply is:

The Cursor ID is 0 in the reply, meaning that there is no cursor to fetch further documents.

We have now come to the end of the article on MongoDB's cursors. All the screenshots of the packets on the wire were made with WireShark, which has support for the MongoDB wire protocol. You can read more about the wire protocol at http://www.mongodb.org/display/DOCS/Mongo+Wire+Protocol; but this is often not something you should have to be concerned about. Just to sum things up:

the Batch Size controls how many documents the server sends in a reply packet.
the server will never send more documents than fit in 4MB.
the Limit option controls how many documents the driver will (try to) fetch from the server.
by default the first batch of documents without any limits specified is 101 documents.
a negative Batch Size or a Batch Size of 1, will terminate the cursor immediately after returning the next batch of documents.

Ignite London: Crowd Sourcing a Map of the World

London, UK

Tuesday, May 15th 2012, 09:47 BST

Almost two weeks ago, I gave a talk at Ignite London about OpenStreetMap, titled "Crowd Sourcing a Map of the World". Ignite's presentation style is 20 slides which automatically advance every 15 seconds. Having never done this before I actually wrote the whole talk out. The presentation that I gave slightly diverges from this but I thought it'd still be good to reproduce here. I did add some links to more information, and if you want to see the recording, you can find it at the end of this post.

1. This talk is about a project, started here in the UK with as its major goal to create a free map of the whole planet. From roads and motorways to country-side footpaths, restaurants and of course pubs. This talk is about OpenStreetMap, the free map of the world.

2. There are of course already plenty of mapping solutions available. Maybe one of the best maps can be acquired through Ordnance Survey. They can be regarded as the national authority on this subject. It's however expensive to get access to their maps, especially the very detailed maps from OS MasterMap. Additionally, it's only for the UK.

3. Besides the commercial solutions, you might wonder why we simply can't do with GoogleMaps? It's mostly freely available for use and also provides you with satellite imagery and StreetView. They even allow you in some areas to update the map through Google MapMaker.

4. But one thing Google doesn't give you access to, is the data behind the map. All you will ever see, is the rendered map tiles and perhaps some APIs to lookup locations and points of interest. Even for data that you have added yourself through MapMaker.

5. Both aspects; the cost of commercial maps, as well as the access to the data that is behind the map tiles is something that the OpenStreetMap project addresses. But which steps have to be taken to obtain this enormous amount of geographical data?

6. We start by getting our wellies and trusty GPS out. Maybe even some pen and paper. We find a location that looks rather empty on the map and travel to that area to see what's on the ground. This is step one: data gathering in the field.

7. In urban areas such as London the roads have often already been mapped and a GPS is not accurate enough to be useful. Then we just use pen and paper to record points of interest, such as shops, landmarks, restaurants and postboxes, my personal favourite.

8. In the country side, donated aerial imagery makes it possible for us to easily trace tracks and footpaths. However, you can't be sure whether the imagery is up-to-date, and you can't always see where fences, streams and local wild life create barriers.

9. In both situations, surveys are best done in groups: at mapping parties. It helps spread the workload and a larger area can be surveyed in one go. As an additional benefit, it allows us to go the pub and discuss our mapping adventures!

10. Doing a survey is important. We take photographs, video and notes with pen and paper of everything that seems to be of interest. This leaves a record that everything we map is actually existing and we can prove that nothing has been copied from other copyrighted maps.

11. After collecting the data, we enter it into the database. This includes basic information such as street names, but we also record whether a café has wheelchair access, or whether a pub has wifi. Updates to the map show up on the site close to real time.

12. Every map object has tags associated with it. Tags tell whether a line is a road, or perhaps a fence. All the tags are free form so you can generally add as much information about an object as you want. Sometimes however, this gets slightly out of hand and people tag pandas in trees and eyes on postboxes.

13. Once the data has been added to the map, we can make use of it. One of the primary uses is obviously showing the data as map tiles. But with all the extra data, we can generate maps that show all the information you're interested in-and nothing more.

14. Clockwise, we have four different visualisations of the map data: we have a cycling-specific style, a style that shows transport routes, a rendering with MapQuest's style sheets and even a water colours inspired style.

15. Having mentioned MapQuest; they were one of the first companies to make use of OpenStreetMap data. They provide, free of charge, map tiles with their own rendering style as well as an instance of Nominatim, OpenStreetMap's geolocation sister project.

16. Lots of companies have already switched to OpenStreetMap. The property search site Nestoria recently switched from using GoogleMaps to OpenStreetMap. Partly because of their costs, but also partly because "The maps are equal or better". geocaching.com, TfL's countdown website and Apple also use OpenStreetMap maps and data.

17. Although OpenStreetMap provides a free and editable map of the world, there are certain requirements for using the data as well. The most important one is that you always need to attribute the OpenStreetMap project.

18. In order to help people start using OpenStreetMap for their mapping needs, the Switch2OSM site has been launched. This website provides background information, case studies and technical information on how to use OpenStreetMap data.

19. Right now, OpenStreetMap has very good data coverage in the country, but we are not nearly finished. A lot of work still has to be done, and we rely on you to improve the data too, even if you add just a postbox.

20. In the last 5 minutes we have looked at what OpenStreetMap is, how the data is gathered and how the data is added to the map. Further more, we had a look at different use cases of the data. OpenStreetMap in the UK: Footpaths and pubs a speciality!

And then I planned showing the "Year of Edits" video, but that sadly didn't work out. I'm including it for good measure here though. (If you want it in HD, follow the link).

OpenStreetMap: A Year of Edits (2011)

The video of the talk itself is at http://vimeo.com/41626116 and is embedded here:

Crowdsourcing a Map of the World

10 years of Xdebug and Xdebug 2.2.0 released

London, UK

Tuesday, May 8th 2012, 14:00 BST

Today it has been ten years since the first release of Xdebug: version 0.7.0. I would like to celebrate this tenth anniversary with a new release: Xdebug 2.2.0. Xdebug 2.2 adds support for PHP 5.4 and provides some new features:

Colours on the command line

First of all there is now var_dump() overloading and colours on the command line. I've already written about that before, but here are the screenshots again:

Linux/Mac:

Windows:

You can find an article about this here.

Better support for closures in stack and function traces

Closures are functions that don't have a function name associated with them. Therefore, Xdebug creates a pseudo function in its stack traces. The stack trace for this example:

Looks like:

Notice: foo in /tmp/closure-stack-trace.php on line 21

Call Stack:
        0.0009     268464   1. {main}() /tmp/closure-stack-trace.php:0
        0.0026     269288   2. test1() /tmp/closure-stack-trace.php:27
        0.0032     270024   3. {closure:/tmp/closure-stack-trace.php:19-22}($a = 5, $b = 25) /tmp/closure-stack-trace.php:24
        0.0038     270496   4. trigger_error('foo') /tmp/closure-stack-trace.php:21

In the 3rd line in the call stack you see {closure:/tmp/closure-stack-trace.php:19-22}($a = 5, $b = 25) where /tmp/closure-stack-trace.php:19-22 contains the filename and line numbers on which the closure is defined at.

The size of arrays is now shown with the overloaded variable output

After this change, each array in the HTML version of the overloaded var_dump() function now looks like:

array (size=5)

Before the change, this was only:

array

Added the method call type to xdebug_get_function_stack

This changes adds another array element to xdebug_get_function_stack() to show whether it was a static or dynamic function call. For example, for:

new Error_Entry(false, $errno);

It now adds the ["type"]=> string(7) "dynamic" element to each stack element:

array(6) { ["function"]=> string(11) "__construct" ["type"]=> string(7) "dynamic" ["class"]=> string(11) "Error_Entry" ["file"]=> string(%d) "%sbug00241.php" ["line"]=> int(11)

Extra information to error printouts to tell that the error suppression operator has been ignored due to xdebug.scream

When you have the xdebug.scream option activated, Xdebug will now tell you when it had any effect on the error reporting with a big "SCREAM: Error suppression ignored for" warning prepended to the actual error message.

Changelog

You can find the full changelog for Xdebug 2.2.0 at the Xdebug website where you can also download the latest version. If you are using Windows, and don't know which binary to download, please refer to the wizard.

Support

If you find Xdebug valuable for your PHP development, perhaps you want to support its development by acquiring a "support" contract. See the buying "support" page if you feel generous.

What's next?

Now Xdebug 2.2 is out of the door, I am looking for new features to add to Xdebug. What would you like to see added to Xdebug? Please leave a comment here, or add your feature requests at http://bugs.xdebug.org.

Working at 10gen

London, UK

Thursday, April 5th 2012, 13:00 BST

As of today I am working for 10gen, the company behind MongoDB. I have been contracting with them for a few months to work on the PHP driver for MongoDB. I am now making the switch to a full time position.

I am responsible for supporting MongoDB in the PHP ecosystem through driver development, integration with open source tools, community support, and developer advocacy. For now, I will concentrate to get the driver into a better shape, and speaking about MongoDB and PHP.

I have already given a few talks on the subject, and if you would like me to speak at your user group or conference, please let me know. If you have any comments on using MongoDB with PHP, I would love to hear them too.

I am looking forwards with the awesome team at 10gen. Both with team in London, as well as with the folks in Palo Alto, New York and Dublin. 10gen is also hiring a few more people for a similar position in New York and Palo Alto. Have a look at the job posting here if you are interested.

New in Xdebug 2.2: Colours on the command line

London, UK

Wednesday, March 21st 2012, 09:43 GMT

This is the first article in a series about new features in Xdebug 2.2. Besides support for PHP 5.4, there are a few that might be of interest.

Xdebug has overloaded var_dump() with xdebug_var_dump() for a long time and the overloaded function could be configured with a few configuration settings. There is xdebug.var_display_max_data to configure how much of a string should be shown; xdebug.var_display_max_children to configure how many children in an array should be shown and xdebug.var_display_max_depth to configure how many levels "deep" the var_dump() should go on for. This functionality was available for when PHP had its html_errors setting on as is usually the case in a web environment1.

Xdebug 2.2 adds this same functionality to the non-HTML environment: the command line. Now the overloaded var_dump() and native xdebug_var_dump() functions also accept the three aforementioned settings:

derick@whisky:~$ php \ -dxdebug.var_display_max_data=8 \ -dxdebug.var_display_max_children=4 \ -r 'var_dump( "a longer string", array( 1, 2, 3, 4, 5, 6, 7 ) );'

outputs:

string(15) "a longer"... array(7) { [0] => int(1) [1] => int(2) [2] => int(3) [3] => int(4) (more elements)... }

Now, to be fair. This is all a side effect; and merely an add-on to a patch by Michael Maclean. He wrote a patch that adds colours to the output on the command line by using ANSI escape codes. This patch also made the overloaded var_dump() listen to the limiting settings for variable display. After his patch, the following was the behaviour on the command line as long as stdout is a tty and xdebug.cli_color is set to 1.

I have extended this so that the settings regarding data display also work without xdebug.cli_color set to 1. Further more, in Xdebug 2.2.0RC1, setting xdebug.cli_color to 2 forces the colours from being shown, even if stdout is not a tty.

Initially, the colour coding of errors and var_dump() output, would only work on a Unix system where ANSI escape codes are commonly supported. After the release of Xdebug 2.2.0RC1, Chris Jones submitted a bug report suggesting that this functionality could also be available on the Windows console. I wasn't aware that Windows could do this anymore since they dropped ANSI.SYS but apparently there is a tool, ANSICON, that reimplements this. From the next release, Xdebug 2.2.0RC2, Xdebug will now also check whether the ANSICON environment variable is set, just like Xdebug would check whether stdout is a tty on a Unix platform. As a result, the equivalent console output as shown before looks like the following on Windows (providing ANSICON is installed):

Now the only thing left is adding complete documentation for this feature ;-)

As always, if you think Xdebug is a valuable tool, have a look at http://xdebug.org/buy-support.php.

1
: Except for in PHP 5.3, where this sadly was turned off by default.

London at Night

Sat, 17 Mar 2012 11:45:00 +0000

London at Night

London, UK

Saturday, March 17th 2012, 11:45 GMT

A few days ago I finally wandered into London with a tripod and my camera. I've been meaning to do this for months and take long exposure shots of various places in London, as well as seeing whether I can make cool timelapses. I've only taken a few pictures near the Thames and the London Eye so far, with one timelapse of the London Eye. I am however planning to return for more. This photography trip also coincided with the conjunction of Jupiter and Venus.

Here are some of the images:

Jupiter and Venus next to Big Ben

A bus streaming past

Houses of parliament

And at last item, a timelapse of the London Eye:

The London Eye, spinning at 75 times its normal speed.

This timelapse was recorded with 802 images. Each image is taken 3 seconds after the previous one resulting in a speed-up of 75 times. Lighting has been manually monitored during the taking of the images.

The rest of the images are on flickr.

Foursquare and OpenStreetMap

Montreal, Canada

Thursday, March 1st 2012, 11:43 EST

Yesterday, FourSquare announced that they are "joining the OpenStreetMap movement", meaning that the now use OpenStreetMap data for their map tiles. With their own map styles and using MapBox for rendering their styles. This is amazing news I think.

Of course, as with many changes, some people are happy but there are others that complain that the map data just isn't up to standards yet for their area. That's a fair point as OpenStreetMap does not have all roads and points-of-interest (POIs) of every city and town in the whole world. The geographical data has mostly been contributed by people on the ground with GPS devices through so called mapping parties. Both in places like London as well as in slums such as Kibera. For a look at the progress over such a weekend, check this video or have a look at "OpenStreetMap: A Year of Edits" for an animation showing all the edits in 2011.

Foursquare is all about spamming twitter, I mean, checking into venues to acquire points and badges. OpenStreetMap is all about having a database of geographical data, and that is not only roads! I've written on OpenStreetMap before.

A venue is a POI in the OpenStreetMap jargon. It makes a whole lot of sense to keep only one database of these POIs/venues. The venues are added (and fixed) by foursquare users in the first place so getting that data into OpenStreetMap would be awesome.

I think this is a great opportunity for fourquare to contribute the venue data to improve OpenStreetMap. I think in this case, there needs to be some form of review process before they can be added to OpenStreetMap, or perhaps consolidated with already existing POI data. This is going be an interesting challenge, and luckily foursquare already is toying with how to improve OpenStreetMap's data.

Besides venues, OpenStreetMap also contains roads and in places those roads are missing or don't have names. Perhaps foursquare can even sponsor mapping parties in the "empty" areas. In the end, a free-to-use open geographic database is what is beneficial to everybody.

For more information on OpenStreetMap, have a look at its wiki or drop by on IRC.

To GMT or not to GMT

Montreal, Canada

Wednesday, February 29th 2012, 11:18 EST

A leap day with a post on Date/Time issues seems fitting...

Earlier today, on twitter, @skoop asked: "dear #lazyweb, when I use DateTimeZone('GMT'), why does format('e') output UTC?" What he means is that:

$date = new DateTime('now', new DateTimeZone('GMT')); echo $date->format(DateTime::RFC2822 . ' e' );

which shows:

Wed, 29 Feb 2012 16:26:23 +0000 UTC

As you can see that has UTC and not GMT as you might expect.

If you look closely at the documentation for the "Other" group of timezones, it lists with the GMT timezone as warning: "Please do not use any of the timezones listed here (besides UTC), they only exist for backward compatible reasons." If you use GMT as timezone identifier in the constructor to DateTimeZone, PHP will instead use the correct UTC in output. When you create a DateTimeZone object like this, you will always get a "type 3" DateTimeZone object:

$date = new DateTime('now', new DateTimeZone('GMT')); var_dump($date);

which shows:

object(DateTime)#1 (3) { ["date"]=> string(19) "2012-02-29 16:30:51" ["timezone_type"]=> int(3) ["timezone"]=> string(3) "UTC" }

Now apparently some systems *cough*Silverlight*cough* require GMT to be used. GMT is not a timezone, but just a timezone abbreviation meant for output only. Read more about that in the article "Leap Seconds and What To Do With Them". However, if it is necessary you can create a DateTime object with a different timezone type. In this case you want a "type 2" timezone associated with the DateTime object. You do that by simply forcing that timezone abbreviation when instantiating a DateTime object:

$date = new DateTime('today GMT'); var_dump( $date );

which shows:

object(DateTime)#1 (3) { ["date"]=> string(19) "2012-02-29 16:32:16" ["timezone_type"]=> int(2) ["timezone"]=> string(3) "GMT" }

Things like this also work:

$date = new DateTime( "GMT" ); $date->setDate( 2012, 2, 19 ); var_dump( $date );

And of course, this is not limited to GMT only:

$date = new DateTime( "EST" ); $date->setDate( 2012, 2, 19 ); var_dump( $date );

which shows:

object(DateTime)#1 (3) { ["date"]=> string(19) "2012-02-19 16:37:58" ["timezone_type"]=> int(2) ["timezone"]=> string(3) "EST" }

As a reminder of the three different types of timezones that can be attached to DateTime objects:

A UTC offset, such as in new DateTime( "2012-02-29 -0500" );

A timezone abbreviation, such as in new DateTime( "2012-02-29 EST" );

A timezone identifier, such as in new DateTime( "2012-02-29 America/Montreal" );

Please also be aware that only DateTime objects with "type 3" timezones attached to them will calculate correctly over Daylight Saving Time boundaries.

If you want to learn more about Dates and Times, and how to use them with PHP, please get a copy of my book "php|architect's Guide to Date and Time Programming".

Random Bugs and Testing RCs

London, UK

Friday, February 24th 2012, 13:00 GMT

At the PHP UK Conference Rasmus mentioned that he wants more people contributing to PHP. There are plenty of ways how you can do that.

The first one is testing release candidates RCs of PHP releases. You can do a very basic test by running "make test" after compiling PHP, but it's even a lot more important to test your own code with the RCs. This often catches more things as the PHP Development Team doesn't quite know how everybody uses PHP. It's a little bit late in the game now for PHP 5.4, but head over to http://qa.php.net/rc.php for a quick intro and a link to where to download the RC8. PHP 5.4.0 is not released yet, so testing the RC is still very valuable. Hurry up though, as this is most likely the last RC for 5.4.0! Also, you are not allowed to complain about PHP 5.4 breaking stuff unless you've tested RCs :-)

The second thing that Rasmus brought up is a new feature on http://bugs.php.net It's modelled like the "random cute cat picture" features that sites like http://cuteoverload.com use for giving you a random cat picture. Except that we don't show cute cats, but a random PHP bug. If you have a few spare minutes go hit http://bugs.php.net/random until there is either an unconfirmed bug report that needs triage, or perhaps you can just fix it because you're a C wizzard. This is hopefully a better time waster than random cute cats.

Oh, yeah, go test the PHP 5.4.0 RC! 🐱

OpenStreetMap: A Year of Edits

London, UK

Sunday, January 1st 2012, 00:00 GMT

In the past weeks I've been working on some visualisations related to additions and changes being made to OpenStreetMap. To start of the new year, I hereby present: OpenStreetMap: A Year of Edits (2011):

OpenStreetMap: A Year of Edits (2011)

You can see the video in HD on Vimeo. Happy New Year!

How Did He Do That?

This animation is all made with Open Source software. I haven't had the time to clean up the code to make it releasable yet, but I am intending to do that in the near feature and add it to my OpenStreetMap tools git repository on github.

The first thing I did, was download all the hourly replication files from http://planet.openstreetmap.org/hour-replicate for the year.

With a script, I scanned over all those XML files and for every three hours I created an image showing the edits of those last three hours (in white), as well as all the other edits in the background (in purple). With 4 frames per day, this gives me 1460 frames.

Each of the frames is an equirectangular projection which I then map onto a sphere with a faded background of "earthmap10k" of http://planetpixelemporium.com/earth.html (I downloaded it years ago when it was still free to download). For the 3D rendering of the Earth with the edits layed on top of it I used POV-Ray.

I generated two different loops so that I can make two full rotations at different angles. This gives me in total 2882 frames which is about two minutes. The POV-Ray generated images I post-processed by overlaying the progress bar in the lower left and added the fading and merging effects. Then I used mencoder to create a MPEG video out of it. Finally I added the sound track again by using mencoder and uploaded it to Vimeo.

Twig extension

London, UK

Monday, November 21st 2011, 09:20 GMT

A while ago, Fabien asked me to have a look at porting one of Twig's slowest methods, TwigTemplate::getAttribute(), into a PHP extension. It is a complex method that does a lot of different checks and look-ups. Fabien's benchmarks showed that this method was responsible for quite a large amount of time. On top of that, it didn't seem that it could be optimised any further as PHP code itself. Seeing that it was very likely that this method would be a lot faster when written in C, SensioLabs decided to sponser me for the development of a port of this method into a PHP extension.

Over the next few months I've worked on re-implementing just this method as a C-function in twig-ext: twig_template_get_attributes(). From initial benchmarks it looks like this function as implemented in the extension gives about a 15% speed increase while rendering templates. Twig's compiled templates directly contain a call to this function (as opposed to marshal it through the original TwigTemplate::getAttribute() method) for additional performance. This probably means that you'll have to remove your compiled templates cache while upgrading.

The extension at http://github.com/derickr/twig-ext has now been merged, via http://github.com/derickr/Twig, into http://github.com/fabpot/Twig. The plan is that this new improvement is going to be part of Twig 1.4.

If you are also interested in having some of your PHP code ported into a PHP extension in C, please feel free to contact me.

XFCE and moving windows around with the keyboard

London, UK

Thursday, November 17th 2011, 09:31 GMT

During a recent apt-get update && apt-get dist-upgrade I suddenly found myself with the horrors that are Gnome 3 and the Gnome shell, and of course, at the least opportune moment when I was actually planning on heading into an office (more about that later). I've always been a big Sawfish fan, and was annoyed when Gnome replaced that with Metacity, but I always managed to get Sawfish back up again. No more luck now, and without a Debian package maintainer I decided to try out XFCE.

I found XFCE to be as easy to use with the keyboard as Sawfish but with one big missing issue: I couldn't change the size and position of windows at the keyboard anymore. I hardly use a mouse as it hurts too much, and always relied on those keyboard shortcuts to position my windows so I was quite a bit at a loss. After some searching I ran into a post on nabble that contained a script that almost did what I want. It uses a bunch of standard X tools (xprop and xwininfo) to find some information about the active window and its properties. Then depending on some parameters it would use wmctrl to move and/or resize the window. For example: ./bin/window-geometry-control.sh -m left would move the window 1 pixel to the left.

That's all good, but way too slow. I basically just want to bump my windows to the top, right, bottom and left sides of my screen. So besides the simple move and resize, I added what Sawfish originally also had: move left/right/top and bottom; and well as resize-to left/right/top and bottom. Now with the command ./bin/window-geometry-control -b left I can move the active window all the way to the left, and with ./bin/window-geometry-control -s bottom I can resize my window from its current position all the way to the bottom of the screen. To each of the four directions and two methods I assigned keyboard shortcuts in xfce4-settings-manager, Keyboard, Application Shortcuts.

The screenshot above shows those assigned keyboard shortcuts. I've put the modified script on github. As you can see I hardcoded the width and height of my own screen in it. Feel free to make this dynamic and send a pull request.

Multiple PHP versions set-up

London, UK

Monday, November 7th 2011, 09:11 GMT

For many of my projects (both hobby and commercial) I need to support many different PHP configurations. Not only just different PHP versions, but also debug builds, ZTS builds and 32-bit builds. In order to be able to test and build extensions against all those different PHP configurations I have adopted a simple method that I'm sharing with you here.

The major part of it is that I use a different installation prefix for every configuration of PHP that I make. Right now, ls /usr/local/php shows:

5.1dev 5.3.2 5.3.8dev 5.3dev-32bit 5.3dev-zts 5.4dev trunk 5.2dev 5.3.7 5.3dev 5.3dev-nodebug 5.4.0beta1 5.4dev-zts

There are two types of PHP installs that I make: "dev" versions from SVN branches, and "release" versions build from SVN tags. From the list above you see I have the SVN versions of 5.1, 5.2, 5.3.8, 5.3 (in various forms) and 5.4 (both normal, and ZTS).

I have a script to compile PHP which whatever combination I want: version, debug (default) or non-debug, normal (default) or ZTS; and 64-bit (default) or 32-bit. The configure options are nicely hardcoded :-)

The scripts accept arguments in a specific order:

version ["debug"|"nodebug" [, "no-zts"|"zts" [, ""|"32bit" ] ] ]

For a simple 5.3dev build I run:

./build 5.3dev

This compiles PHP 5.3 from SVN, in debug mode, without ZTS and for the 64-bit architecture. Something more exotic variants could be:

./build 5.3.8 debug zts ./build 5.4dev nodebug no-zts 32bit

Each invocation of the script will configure and build PHP, and then install into /usr/local/php/{{variant}}.

The script also has a hard coded location where the PHP sources are. In my case, that's a sparse checkout into /home/derick/dev/php/php-src.

With the help of a small .bashrc snippet:

function pe () { version=$1 shift if [ "$#" == "0" ]; then export PATH=/usr/local/php/${version}/bin:/usr/local/bin:/usr/bin:/bin else PATH=/usr/local/php/${version}/bin:$PATH $@ fi }

I can now easily switch between PHP versions by typing on the shell:

pe 5.3dev

or:

pe 5.4dev-zts

And each version will have a totally separated environment for me to install PEAR packages and PECL extensions in, and do my own extension development. Of course, each separated environment also comes with its own php.ini file (in /usr/local/php/{{variant}}/lib/php.ini).

This set-up makes my life a whole lot easier, and I hope it is useful for you as well. Thanks for listening!

Xdebug's Code Coverage speedup

London, UK

Friday, September 23rd 2011, 09:23 BST

Besides a debugging and development aid, Xdebug also implements the back-end code to provide code coverage information for use with PHPUnit. Code coverage tells you how much of your code base is actually being tested by your unit tests. It's a very useful feature, but sadly, it slows down PHP's execution quite a lot. One part of this slowdown is the overhead to record the information internally, but another part is because I have to overload lots of opcodes. (Opcodes are PHP's internal execution units, similar to assembler instructions) They are always overloaded even if code coverage is not used, because it's only safe to overload them for the whole request.

The upcoming Xdebug 2.2 has a few changes to improve code coverage performance. First of all, the speed with which information about code coverage is recorded has been improved by contributions by Taavi Burns. And secondly, Xdebug 2.2 features a new setting, xdebug.coverage_enable with which it is possible to disable the overloading of code coverage specific opcodes. By default that setting will still be on.

Just to show how those improvements have effect on the speed, I've run a benchmark (running Apache Zeta Components Graph's tests) to compare Xdebug 2.1.2 against Xdebug 2.2-dev.

The results are as follows:

Version

With CC

Without CC

With coverage_enable=0

Without Xdebug

2.1.2

06:44

00:49

not available

00:32

2.2-dev

05:37

00:48

00:46

00:32

Another benchmark, run by Ross McFarlane gives: @derickr Based on 3 runs each, execution of 254 tests with 1070 assertions has dropped from 19s to 12s. Nice work!.

In Xdebug 2.2 I would also like to introduce modes. Each mode will set a default configuration for Xdebug's setting to be the most optimal for that specific tasks. Examples of tasks could be: "profiling", "debugging" or "tracing". Let me know (in the comments) whether you think that's a good idea.

Remote Debugging PHP with a Firewall in the Way

London, UK

Thursday, August 25th 2011, 21:42 BST

The PHP debugging extension Xdebug has "remote" debugging capabilities for single-step debugging PHP applications. This works by setting your favourite IDE into listening mode and instructing Xdebug (with one of the handy browser extensions for example) to initiate debugging. When Xdebug is instructed it tries to make a connection to the IP address and port number that are configured in php.ini. On this IP address and port Xdebug expects to find a listening IDE. This IP address can just be 127.0.0.1 in case your IDE and web server/CLI script run on the same machine. But as Xdebug supports remote debugging, PHP/Xdebug could also be running on a totally different machine than your IDE. In that case, you need to pick the IP address of the machine on which the IDE is running as the configuration value for xdebug.remote_host. Alternatively you can set xdebug.remote_connect_back to true, so that Xdebug may simply pick up your client machine's IP address from the HTTP headers.

There could however be a firewall in the way that prevents Xdebug connecting directly to your IDE's IP address. That can be because the network you are on employs NAT in which case you could try (to convince) to get your system administrator to install a DBGp proxy. In some cases, he might not be willing to do so, or perhaps, there is simply a black box that doesn't allow either a proxy to be run on it, or port forwarding to be set-up on. In this case, there is no way Xdebug can connect to your IDE's IP address and port. Or is there?

Breaking the Firewall Down

There is another solution, at least, if you have SSH access to the development machine where Xdebug is running on. In this case, you can simply ssh in and set-up a tunnel:

ssh -R 9000:localhost:9000 username@dev.example.com

This command opens up port 9000, configured with the first 9000 in the command, for listening on the machine where you ssh-ed into. Every connection that is made to this port is then forwarded to localhost:9000, which in this case is port 9000 on the machine from where you ran the ssh command from. When you set Xdebug's xdebug.remote_host setting to localhost and xdebug.remote_port to 9000 you know have instructed Xdebug to connect to your SSH-tunneled connection which is forwarded to your local port 9000. And that is exactly where your IDE is listening for incoming debugging connections. Result: Firewall circumvented.

If you are so unfortunate to be using Windows on the client side then you don't have the luxury of being able to use the simple SSH client. However, you can set-up tunnels with putty as well. After configuring your normal session, go to the Connection, SSH, Tunnels section of the configuration and fill in Source port and Destination, and select Remote, just like in this screen shot:

Then click the Add button and you will see the following on your screen:

Don't forget to go back to the Session tab and press save. You're now ready to login and when you do so an SSH tunnel will be created just like in the Unix case above.

You can check whether it worked by running: netstat -a -n | grep 9000 on the SSH prompt after logging in. You should then see:

derick@xdebug:~$ netstat -a -n | grep 9000 tcp 0 0 0.0.0.0:9000 0.0.0.0:* LISTEN tcp6 0 0 :::9000 :::* LISTEN

The tcp6 portion might not be there, but that's all right. There, even with Windows: Firewall circumvented.