Derick Rethans - tag: php

OpenStreetMap: A Year of Edits

Sun, 01 Jan 2012 00:00:00 +0000

OpenStreetMap: A Year of Edits

Sunday, January 1st 2012, 00:00 GMT

In the past weeks I've been working on some visualisations related to additions and changes being made to OpenStreetMap. To start of the new year, I hereby present: OpenStreetMap: A Year of Edits (2011):

OpenStreetMap: A Year of Edits (2011)

You can see the video in HD on Vimeo. Happy New Year!

How Did He Do That?

This animation is all made with Open Source software. I haven't had the time to clean up the code to make it releasable yet, but I am intending to do that in the near feature and add it to my OpenStreetMap tools git repository on github.

The first thing I did, was download all the hourly replication files from http://planet.openstreetmap.org/hour-replicate for the year.

With a script, I scanned over all those XML files and for every three hours I created an image showing the edits of those last three hours (in white), as well as all the other edits in the background (in purple). With 4 frames per day, this gives me 1460 frames.

Each of the frames is an equirectangular projection which I then map onto a sphere with a faded background of "earthmap10k" of http://planetpixelemporium.com/earth.html (I downloaded it years ago when it was still free to download). For the 3D rendering of the Earth with the edits layed on top of it I used POV-Ray.

I generated two different loops so that I can make two full rotations at different angles. This gives me in total 2882 frames which is about two minutes. The POV-Ray generated images I post-processed by overlaying the progress bar in the lower left and added the fading and merging effects. Then I used mencoder to create a MPEG video out of it. Finally I added the sound track again by using mencoder and uploaded it to Vimeo.

Twig extension

London, UK

Monday, November 21st 2011, 09:20 GMT

A while ago, Fabien asked me to have a look at porting one of Twig's slowest methods, TwigTemplate::getAttribute(), into a PHP extension. It is a complex method that does a lot of different checks and look-ups. Fabien's benchmarks showed that this method was responsible for quite a large amount of time. On top of that, it didn't seem that it could be optimised any further as PHP code itself. Seeing that it was very likely that this method would be a lot faster when written in C, SensioLabs decided to sponser me for the development of a port of this method into a PHP extension.

Over the next few months I've worked on re-implementing just this method as a C-function in twig-ext: twig_template_get_attributes(). From initial benchmarks it looks like this function as implemented in the extension gives about a 15% speed increase while rendering templates. Twig's compiled templates directly contain a call to this function (as opposed to marshal it through the original TwigTemplate::getAttribute() method) for additional performance. This probably means that you'll have to remove your compiled templates cache while upgrading.

The extension at http://github.com/derickr/twig-ext has now been merged, via http://github.com/derickr/Twig, into http://github.com/fabpot/Twig. The plan is that this new improvement is going to be part of Twig 1.4.

If you are also interested in having some of your PHP code ported into a PHP extension in C, please feel free to contact me.

Multiple PHP versions set-up

London, UK

Monday, November 7th 2011, 09:11 GMT

For many of my projects (both hobby and commercial) I need to support many different PHP configurations. Not only just different PHP versions, but also debug builds, ZTS builds and 32-bit builds. In order to be able to test and build extensions against all those different PHP configurations I have adopted a simple method that I'm sharing with you here.

The major part of it is that I use a different installation prefix for every configuration of PHP that I make. Right now, ls /usr/local/php shows:

5.1dev  5.3.2  5.3.8dev  5.3dev-32bit    5.3dev-zts  5.4dev      trunk
5.2dev  5.3.7  5.3dev    5.3dev-nodebug  5.4.0beta1  5.4dev-zts

There are two types of PHP installs that I make: "dev" versions from SVN branches, and "release" versions build from SVN tags. From the list above you see I have the SVN versions of 5.1, 5.2, 5.3.8, 5.3 (in various forms) and 5.4 (both normal, and ZTS).

I have a script to compile PHP which whatever combination I want: version, debug (default) or non-debug, normal (default) or ZTS; and 64-bit (default) or 32-bit. The configure options are nicely hardcoded :-)

The scripts accept arguments in a specific order:

version ["debug"|"nodebug" [, "no-zts"|"zts" [, ""|"32bit" ] ] ]

For a simple 5.3dev build I run:

./build 5.3dev

This compiles PHP 5.3 from SVN, in debug mode, without ZTS and for the 64-bit architecture. Something more exotic variants could be:

./build 5.3.8 debug zts
./build 5.4dev nodebug no-zts 32bit

Each invocation of the script will configure and build PHP, and then install into /usr/local/php/{{variant}}.

The script also has a hard coded location where the PHP sources are. In my case, that's a sparse checkout into /home/derick/dev/php/php-src.

With the help of a small .bashrc snippet:

function pe () {
        version=$1
        shift

        if [ "$#" == "0" ]; then
                export PATH=/usr/local/php/${version}/bin:/usr/local/bin:/usr/bin:/bin
        else
                PATH=/usr/local/php/${version}/bin:$PATH $@
        fi
}

I can now easily switch between PHP versions by typing on the shell:

pe 5.3dev

or:

pe 5.4dev-zts

And each version will have a totally separated environment for me to install PEAR packages and PECL extensions in, and do my own extension development. Of course, each separated environment also comes with its own php.ini file (in /usr/local/php/{{variant}}/lib/php.ini).

This set-up makes my life a whole lot easier, and I hope it is useful for you as well. Thanks for listening!

Xdebug's Code Coverage speedup

London, UK

Friday, September 23rd 2011, 09:23 BST

Besides a debugging and development aid, Xdebug also implements the back-end code to provide code coverage information for use with PHPUnit. Code coverage tells you how much of your code base is actually being tested by your unit tests. It's a very useful feature, but sadly, it slows down PHP's execution quite a lot. One part of this slowdown is the overhead to record the information internally, but another part is because I have to overload lots of opcodes. (Opcodes are PHP's internal execution units, similar to assembler instructions) They are always overloaded even if code coverage is not used, because it's only safe to overload them for the whole request.

The upcoming Xdebug 2.2 has a few changes to improve code coverage performance. First of all, the speed with which information about code coverage is recorded has been improved by contributions by Taavi Burns. And secondly, Xdebug 2.2 features a new setting, xdebug.coverage_enable with which it is possible to disable the overloading of code coverage specific opcodes. By default that setting will still be on.

Just to show how those improvements have effect on the speed, I've run a benchmark (running Apache Zeta Components Graph's tests) to compare Xdebug 2.1.2 against Xdebug 2.2-dev.

The results are as follows:

Version	With CC	Without CC	With coverage_enable=0	Without Xdebug
2.1.2	06:44	00:49	not available	00:32
2.2-dev	05:37	00:48	00:46	00:32

Another benchmark, run by Ross McFarlane gives: @derickr Based on 3 runs each, execution of 254 tests with 1070 assertions has dropped from 19s to 12s. Nice work!.

In Xdebug 2.2 I would also like to introduce modes. Each mode will set a default configuration for Xdebug's setting to be the most optimal for that specific tasks. Examples of tasks could be: "profiling", "debugging" or "tracing". Let me know (in the comments) whether you think that's a good idea.

Remote Debugging PHP with a Firewall in the Way

London, UK

Thursday, August 25th 2011, 21:42 BST

The PHP debugging extension Xdebug has "remote" debugging capabilities for single-step debugging PHP applications. This works by setting your favourite IDE into listening mode and instructing Xdebug (with one of the handy browser extensions for example) to initiate debugging. When Xdebug is instructed it tries to make a connection to the IP address and port number that are configured in php.ini. On this IP address and port Xdebug expects to find a listening IDE. This IP address can just be 127.0.0.1 in case your IDE and web server/CLI script run on the same machine. But as Xdebug supports remote debugging, PHP/Xdebug could also be running on a totally different machine than your IDE. In that case, you need to pick the IP address of the machine on which the IDE is running as the configuration value for xdebug.remote_host. Alternatively you can set xdebug.remote_connect_back to true, so that Xdebug may simply pick up your client machine's IP address from the HTTP headers.

There could however be a firewall in the way that prevents Xdebug connecting directly to your IDE's IP address. That can be because the network you are on employs NAT in which case you could try (to convince) to get your system administrator to install a DBGp proxy. In some cases, he might not be willing to do so, or perhaps, there is simply a black box that doesn't allow either a proxy to be run on it, or port forwarding to be set-up on. In this case, there is no way Xdebug can connect to your IDE's IP address and port. Or is there?

Breaking the Firewall Down

There is another solution, at least, if you have SSH access to the development machine where Xdebug is running on. In this case, you can simply ssh in and set-up a tunnel:

ssh -R 9000:localhost:9000 username@dev.example.com

This command opens up port 9000, configured with the first 9000 in the command, for listening on the machine where you ssh-ed into. Every connection that is made to this port is then forwarded to localhost:9000, which in this case is port 9000 on the machine from where you ran the ssh command from. When you set Xdebug's xdebug.remote_host setting to localhost and xdebug.remote_port to 9000 you know have instructed Xdebug to connect to your SSH-tunneled connection which is forwarded to your local port 9000. And that is exactly where your IDE is listening for incoming debugging connections. Result: Firewall circumvented.

If you are so unfortunate to be using Windows on the client side then you don't have the luxury of being able to use the simple SSH client. However, you can set-up tunnels with putty as well. After configuring your normal session, go to the Connection, SSH, Tunnels section of the configuration and fill in Source port and Destination, and select Remote, just like in this screen shot:

Then click the Add button and you will see the following on your screen:

Don't forget to go back to the Session tab and press save. You're now ready to login and when you do so an SSH tunnel will be created just like in the Unix case above.

You can check whether it worked by running: netstat -a -n | grep 9000 on the SSH prompt after logging in. You should then see:

derick@xdebug:~$ netstat -a -n | grep 9000
tcp        0      0 0.0.0.0:9000            0.0.0.0:*               LISTEN
tcp6       0      0 :::9000                 :::*                    LISTEN

The tcp6 portion might not be there, but that's all right. There, even with Windows: Firewall circumvented.

Valgrinding shared modules

London, UK

Monday, August 8th 2011, 09:56 BST

Over the past year I've been writing various commercial (more about that later) and open source PHP extensions (such as QuickHash, more about that later too). Most of the time they are shared extensions that are not part of PHP. While testing whether I correctly free all memory with Valgrind, I ran into the issue where I couldn't see the stack frames of where the memory leaks occurred in the extensions, and once I even ran into a Valgrind bug. The reason why Valgrind could not show the function names belonging to the stack frames is because PHP had already unloaded the shared extensions from memory.

I often found myself compiling the extensions into PHP statically so that there was nothing to unload for PHP, but that was becoming annoying. So instead I added a patch to PHP that prevents the shutdown sequence from actually unloading the modules. You can trigger this behaviour by setting the ZEND_DONT_UNLOAD_MODULES environment variable before running your script:

# export ZEND_DONT_UNLOAD_MODULES=1
# valgrind --leak-check=full --show-reachable=yes php -r 'echo "Hello World\n";'

Without ZEND_DONT_UNLOAD_MODULES exported, my Valgrind output shows a block like:

# unset ZEND_DONT_UNLOAD_MODULES
# valgrind --leak-check=full --show-reachable=yes php -r 'echo "Hello World\n";'
...
==25829== 8 bytes in 1 blocks are indirectly lost in loss record 2 of 21
==25829==    at 0x4C25E84: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==25829==    by 0xCE440DC: ???
==25829==    by 0xCE44316: ???
==25829==    by 0xCE44368: ???
==25829==    by 0xCBEE55F: ???
==25829==    by 0xCBD3F87: ???
==25829==    by 0x949A85: zend_activate_modules (zend_API.c:2285)
==25829==    by 0x8B5EBC: php_request_startup (main.c:1491)
==25829==    by 0xA83D60: do_cli (php_cli.c:954)
==25829==    by 0xA84F7B: main (php_cli.c:1356)
...

And with ZEND_DONT_UNLOAD_MODULES exported, it shows instead:

# export ZEND_DONT_UNLOAD_MODULES=1
# valgrind --leak-check=full --show-reachable=yes php -r 'echo "Hello World\n";'
...
==25824== 8 bytes in 1 blocks are still reachable in loss record 2 of 30
==25824==    at 0x4C25E84: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==25824==    by 0xCE440DC: event_base_priority_init (in /usr/lib/libevent-1.4.so.2.1.3)
==25824==    by 0xCE44316: event_base_new (in /usr/lib/libevent-1.4.so.2.1.3)
==25824==    by 0xCE44368: event_init (in /usr/lib/libevent-1.4.so.2.1.3)
==25824==    by 0xCBEE55F: zm_activate_http_request_pool (http_request_pool_api.c:58)
==25824==    by 0xCBD3F87: zm_activate_http (http.c:373)
==25824==    by 0x949A85: zend_activate_modules (zend_API.c:2285)
==25824==    by 0x8B5EBC: php_request_startup (main.c:1491)
==25824==    by 0xA83D60: do_cli (php_cli.c:954)
==25824==    by 0xA84F7B: main (php_cli.c:1356)
...

As you can see all the symbols are now nicely resolved. This patch is part of the upcoming PHP 5.4, but applies to PHP 5.2 and PHP 5.3 as well.

Xdebug on github

Wed, 22 Jun 2011 20:47:00 +0000

Xdebug on github

London, UK

Wednesday, June 22nd 2011, 21:47 BST

I've just opened my Xdebug repository on github. Importing my old SVN repository was a bit of a task. With the help of Ole Marius Smestad I managed to cook up a bash script to import and fix-up the repository. In the meanwhile I've learned more about git than I ever wanted to know, but still not enough. I maintain that, although git is very powerful, it is also a lot less intuitive than SVN (or even CVS).

In any case, Xdebug is now on github, as well as the Xdebug website (http://xdebug.org), my proof-of-concept debugging client GtkDBGp and the specification of the protocol that Xdebug uses: DBGp. The issue tracker is still at the old location—http://bugs.xdebug.org—and I am not intending to change that. I will keep the SVN repository active for now.

Spatial Indexes: Solr

Tue, 14 Jun 2011 08:04:00 +0000

Spatial Indexes: Solr

London, UK

Tuesday, June 14th 2011, 09:04 BST

In two previous articles I introduced the spherical Earth model, using SQLite as a geographical data storage and using MySQL as a geographical data storage. In this article we're going to have a look at importing the data into something else than a relational database: the search platform Solr. (Yes, I know I've skipped PostgreSQL, but I'll come back to that).

Solr

Solr is "the popular, blazing fast open source enterprise search platform from the Apache Lucene project." Since version 3.1, Solr has support for spatial search, including geospatial search. It can store coordinates in two different field types: solr.PointType for n-dimensional points, and solr.LatLonType for a two-dimensional point for geospatial search. The main difference is that with solr.PointType, calculations are done according to the flat Earth model, and solr.LatLonType does calculations for a spherical Earth model—which is just what we need.

In this example we will use the default Solr configuration, unless noted otherwise. First we download Solr 3.1 and then untar it with tar -xvzf apache-solr-3.1.0.tgz. Then we edit example/solr/conf/solrconfig.xml to comment out the section on Query Elevation Component so that it looks like:

See here for the reason.

Setting Up Solr

Before we can start using Solr, we need to define a schema. This schema is quite analogous to a database schema, except that Solr only has one table. By default Solr comes with a schema defining lots of fields in example/solr/conf/schema.xml. I've changed the whole section to look like:

I've defined specific fields for the type of source (1=node, 2=way/area), the name of the amenity (name of the pub, hospital, etc), what sort of amenity it is (cafe, bank, etc), the address and postcode, the phone number, the type of cuisine as well as the location. For the location I've also defined two subtypes: location_0_coordinate and location_1_coordinate. Solr requires this for multi-dimensional types such as solr.LatLonType.

The types for each field are also defined in schema.xml. text is for a string that is analysed and tokenized (broken down in smaller parts) so that you can search in only parts of the text, string is for a string of characters that is not analysed or tokenized, and location contains our latitude/longitude point. Each of those types are associated with a specific Java class with an entry such as:

The subFieldSuffix in this line configures the suffix in the fields location_0_coordinate and location_1_coordinate.

Now we have changed the default schema, we can start Solr with: cd example && java -jar start.jar.

On the PHP side, we need to do a bit of set-up as well. I've been using the Solr PECL extension to talk to Solr which can be installed with pecl install solr. Please do not forget to add the extension to php.ini with extension=solr.so. Check whether it is installed correctly with: php --ri solr.

Importing

For our import, we need to change the script we used previously quite substantially. Instead of just using London, I've also downloaded the whole of the UK from Geofabrik and extracted all amenities from the dataset with Osmosis:

wget http://download.geofabrik.de/osm/europe/great_britain.osm.bz2

./osmosis-0.39/bin/osmosis -v 5 \
--read-xml file=great_britain.osm.bz2 \
--tf reject-relations \
--tf accept-nodes amenity=* \
--tf reject-ways \
outPipe.0=POI \
\
--read-xml file=great_britain.osm.bz2 \
--tf reject-relations \
--tf accept-ways amenity=* \
--used-node outPipe.0=area \
\
--merge inPipe.0=POI inPipe.1=area \
--write-xml file=great_britain_amenity.osm

This is going to take quite some time, and will result in a 500MB XML file.

The main logic of the importing script remains the same, but instead of adding each item to a database, we now build a Solr import document and add it to the search index. I've also split the address and postcode into two separate fields, and added the amenity field to store all the different amenities. For now, I'm filtering out post boxes, parking areas and grave yards. The script, parsepoi-solr.php is available here.

After downloading, and renaming the downloaded file to parsepoi-solr.php we can run the script with:

php -dmemory_limit=1G parsepoi-solr.php great_britain_amenity.osm

When done, this should have imported more than 140.000 items into Solr. You can verify this, by going to http://localhost:8983/solr/select/?q=*:*&rows=0

Querying

As you can see, you can query Solr through its HTTP interface quite easily. In fact, all queries to Solr are done over HTTP. The Solr extension however abstracts this away from you for at least the import. I couldn't find any functionality in the extension to do spatial queries, so we'll do it manually.

If we look at the URL above, we can divide it into different parts:

http://localhost:8983/solr/: The base URL for this Solr instance.
select/?: The action for running search queries.
q=*:*: q is the search query parameter, and its value, *:* means all fields (the first *) and all possible values (the second *). In this case that means return everything.
rows=0: Do not return any rows, so that we just get a count.

We get back the result as an XML file. If we want JSON instead, we can simply append &wt=json. In case you want a CSV file, append &wt=csv.

In a first example, all we want to do is to return all cafes within 100 meter. We construct the search query as follows:

http://localhost:8983/solr/select/?: The base URL with search action.
q=amenity:cafe: We search in the field amenity and look for the value cafe.
fq={!geofilt}: We set a query filter to geofilt_. This filter restricts the result set according to the location (pt) and the maximum distance from this location (d) in km.
sfield=location: The field to use for location information.
pt=51.5375,-0.1934: The point that we center our search around.
d=0.1: The maximum distance in kilometers.
wt=csv: The format to return, in our case, CSV.

Together this makes the full GET request: http://localhost:8983/solr/select/?q=amenity:cafe&fq={!geofilt}&sfield=location&pt=51.5375,-0.1934&d=0.1&wt=csv

And the result is:

id,phone,cuisine,location,address,location_1_coordinate,name,amenity,type,location_0_coordinate,postcode
w62088838,,,51.53750834,-0.19329616,75 Kilburn High Road,-0.19329616,Costa,cafe,2,51.53750834,
w78337118,,,51.53828298,-0.19410346,101 Kilburn High Road,-0.19410346,Caffè Nero,cafe,2,51.53828298,NW6 6JE
w105467205,,,51.537555925,-0.192445525,274 Belsize Road,-0.192445525,Belsize Cafe,cafe,2,51.537555925,NW6 4BT
w105467209,+44 207 3724002,,51.537624071429,-0.19228221428571,270 Belsize Road,-0.19228221428571,Lord Jim,cafe,2,51.537624071429,NW6 4BT
w107710475,+44 20 76245736,,51.53695276,-0.19263662,2 Kilburn Bridge,-0.19263662,Famished Cafe,cafe,2,51.53695276,NW6 6HT
w107710482,,,51.53717868,-0.19288912,Kilburn Bridge,-0.19288912,Mike's,cafe,2,51.53717868,NW6 6HT
w107710490,+44 20 76246942,,51.53732946,-0.1930577,12 Kilburn Bridge,-0.1930577,La Dolce Vita,cafe,2,51.53732946,NW6 6HT

Which fields are returned can be configured. In this case, we are only interested in the location and name of each amenity, so we restrict the number of returned fields with: fl=id,location,name. The result now becomes:

id,location,name
w62088838,51.53750834,-0.19329616,Costa
w78337118,51.53828298,-0.19410346,Caffè Nero
w105467205,51.537555925,-0.192445525,Belsize Cafe
w105467209,51.537624071429,-0.19228221428571,Lord Jim
w107710475,51.53695276,-0.19263662,Famished Cafe
w107710482,51.53717868,-0.19288912,Mike's
w107710490,51.53732946,-0.1930577,La Dolce Vita

Sadly, the results do not come back ordered by distance from our starting point. In order to do that, we need to add one more query parameter: sort=geodist() asc. The full query is now: http://localhost:8983/solr/select/?q=amenity:cafe&fq={!geofilt}&sfield=location&pt=51.5375,-0.1934&d=0.10&wt=csv&fl=id,location,name&sort=geodist()+asc

And the result:

id,location,name
w62088838,51.53750834,-0.19329616,Costa
w107710490,51.53732946,-0.1930577,La Dolce Vita
w107710482,51.53717868,-0.19288912,Mike's
w105467205,51.537555925,-0.192445525,Belsize Cafe
w105467209,51.537624071429,-0.19228221428571,Lord Jim
w107710475,51.53695276,-0.19263662,Famished Cafe
w78337118,51.53828298,-0.19410346,Caffè Nero

It is however not possible to return the distance from our starting point directly with Solr. With the exception being that if you use only the geodist() function as query argument. In this case, make sure to include the special field score in the fl= argument as well. An URL showing this is: http://localhost:8983/solr/select/?q={!func}geodist()&fq={!geofilt}&sfield=location&pt=51.5375,-0.1934&d=0.10&wt=csv&fl=id,location,name,score&sort=geodist()+asc Of course, in this case you can not filter for specific amenities.

Conclusion

In currently released versions of Solr (3.1 at the time of writing), it is not possible to return the distance to the starting point for the search as part of the result set. The Solr developers have already implemented the capabilities to add the results of function queries to result documents for version 4.0. You can verify this by downloading a nightly build and running the query: http://localhost:8983/solr/select/?q=amenity:cafe&fq={!geofilt}&sfield=location&pt=51.5375,-0.1934&d=0.10&fl=id,location,name,score,geodist()&sort=geodist()+asc In this query I've added geodist() to the fl= parameter. The result now includes an extra field called geodist():



  
  
        
          w62088838
          Costa
          51.53750834,-0.19329616
          4.352648
          0.0072415726934058865
        
        
          w107710490
          La Dolce Vita
          51.53732946,-0.1930577
          4.352648
          0.030333097323281075

I could not manage to alias it to a different field, or get it to work with the CSV format (wt=csv).

If the return format is xml (the default) or json, then the result includes, besides the number of found items, the total search time for this query. Look for this information in the QTime field. In all of the examples here, the QTime has been less than 5, meaning 5 milliseconds. Solr is extremely fast, even with huge amounts of data. I will try to import the amenities of the whole "planet" at some point, and report back with some benchmarking information.

In the next installment of this series on storing geospatial data, I will be looking at using MongoDB as data store for geographical information.

Translating Twitter, part 2

Tue, 31 May 2011 08:04:00 +0000

Translating Twitter, part 2

London, UK

Tuesday, May 31st 2011, 09:04 BST

A while ago I wrote in an article about translating tweets in my client Haunt. For the translating itself I was using the Google Translate API, which has sadly be deprecated. Evil after all I suppose.

I've now rewritten my translation code to use the Bing Translation APIs instead. You need to register an API key (see http://www.bing.com/developers/appids.aspx) to be able to use the APIs. The APIs that I am using are fairly simple though.

For a simple translation, requesting http://api.microsofttranslator.com/V2/Http.svc/Translate?appId=[yourappid]&to=en&text=[yourtext] is all you need to do. It will auto-detect the language for you as well.

However, it does not return the detected language, so I had to resort to using two requests in order to reimplement the same functionality that I had before with the Google APIs. I also found that it was easier to use the Http and not the Ajax variant of the API. It requires using SimpleXML to get to the data, but at least you do not have to fight with the BOM (Byte-order mark) and quoting.

The full code looks like:

with as output:

string(2) "nl"
string(34) "A high tree catches a lot of wind."

What is OpenStreetMap?

Thu, 12 May 2011 08:04:00 +0000

What is OpenStreetMap?

London, UK

Thursday, May 12th 2011, 09:04 BST

With people more and more complaining that Google Maps gets it wrong, I often reply that the complainers have a look at OpenStreetMap instead. But there are a few misunderstandings on what OpenStreetMap actually is.

OpenStreetMap has its main site at http://openstreetmap.org. This shows a rendering of the map that we like to call a "slippy map". This is an online map, that can interactively be zoomed and paned. The slippy map on the site is nothing more than a display of map tiles, static images rendered from OpenStreetMap data as an example of what you can do with it. The real power of OpenStreetMap is not this default rendering, but the possibility to actually access the data behind this map rendering. And this is where it differs from Google maps, Bing Maps and Yahoo! Maps and various others. OpenStreetMap is the only mapping service that allows you to do something more than just look at pretty map tiles. OpenStreetMap is a database project, with as its main purpose to have an exhaustive database of every street, city, road, building etc on the planet, and not a map display project.

Now what does that actually mean? For one thing, it means that the sample map tiles (that are displayed in the slippy map) are not meant to be used heavily from applications. It's all right to show a map tile of your neighbourhood on your web site, but it's not okay to write a mobile application that allows you to scrape map tiles of large areas. Have a look at the general tile usage policy for some background. The only reason why there is a sample rendering is to aid mappers with improving the data that is the core of OpenStreetMap.

Obviously, the OpenStreetMap project wants its data to be used. But if you're not allowed to use the map tiles en-masse, then how can you use the data?

First of all, you can query the OpenStreetMap database in various ways. In most cases, you're going to get an XML file with descriptions of nodes (points of interest, facilities such as toilets, benches, addresses), ways (roads, water ways, transport routes) and areas (buildings, lakes). There is some more info on the (very simple schema) here. This data you can parse and import (as I've shown in a previous article) and do all kinds of cool tricks with, such as finding out the closest pub. This functionality is not available on the main OpenStreetMap site, because "OpenStreetMap does not aim to create and host every webservice possible, but to provide the data so that others can" make awesome map-related applications.

Secondly there is a growing number of web sites that uses the OpenStreetMap data to render specialized maps. There is OpenCycleMap that renders information related to using your bicycle and Öpvnkarte for public transport. There is even a tool that uses the postcode data in OpenStreetMap. It's quite possible to render your own maps as well, but that I will get back to in a later article.

MapQuest is a good example of a company that uses OpenStreetMap data. They have made their own rendering with its own styles for a visually different map. Their site implements searching for pubs/bars and restaurants for example and MapQuest are also happy with you using their map tiles for your applications. Recently, they have also announced a simple API to generate a static map based on OpenStreetMap with their rendering style.

The map above is generated by just requesting http://open.mapquestapi.com/staticmap/v3/getmap?size=320,260&zoom=12¢er=51.51,-0.15

Another major difference with established mapping providers is the ability to edit the map yourself. You can add your own business, houses, addresses, postcodes, your favourite restaurants with either on-line or off-line editors. My list of recent edits is here.

There is also a whole plethora of special renderings of the map data to assist with debugging issues with the map. There is the "no name" rendering which shows roads without name, ITO World's osm analysis that shows deviations of road names from another source in the UK, keepright and OSM Inspector that have some automated checks and a tool to overlay postcodes for checking.

Obviously, there are a few things missing that would make OpenStreetMap an even better resource for mapping related applications and web-applications. Feel free to suggest things that you find missing or lacking in the comments of this article. I'd be happy to hear why you would think that OpenStreetMap is not a good fit for your applications and usage.

Spatial Indexes: MySQL

Tue, 12 Apr 2011 08:04:00 +0000

Spatial Indexes: MySQL

London, UK

Tuesday, April 12th 2011, 09:04 BST

In two previous articles I introduced The spherical Earth model and importing data into SQLite for querying geographical data. In this article we're going to have a look at importing the data into MySQL and finding out how to best store and query spatial data in the databases.

MySQL

MySQL has some support for Spatial Extensions, but it's not particularly useful. For example, there is no way to query anything within the radius around a specific point. Their community pages list a method of implementing it, but it only calculates for a flat Earth model.

Instead, we'll have to implement our own algorithms again. But first of all, let us import the data into MySQL. First we create the MySQL database:

derick@whisky:~$ mysqladmin -u root -p create poi
mysql> CREATE TABLE poi(id int, type int, lat float, lon float, name char(255), address char(255), cuisine char(64), phone char(18));

And then we take the script from the previous article on importing data, and change the third line from:

$d = ezcDbFactory::create( 'sqlite://' . dirname( __FILE__ ) . '/pois.sqlite' );

to:

$d = ezcDbFactory::create( 'mysql://root:root@localhost/poi' );

Of course, substitute the username and password (root, root) and the database name (poi) to one that suits yourself. We then run again:

php parseoi.php.txt great_britain_pubs.osm

And check that our import is complete:

mysql> SELECT count(*) from poi\G
*************************** 1. row ***************************
count(*): 28147

Now that we have all the POIs imported, we can query the database. Because MySQL doesn't have working spatial extensions, nor the register-function capabilities from SQLite such as we saw in the previous article, we have to come up with a new solution. The most obvious one is writing a stored procedure for the task, another (non-explored option) would be to write a user defined function. Writing the stored procedure is simple enough; we just have to convert the distance() function to SQL. On the MySQL command line you can enter:

delimiter //

CREATE FUNCTION distance (latA double, lonA double, latB double, LonB double)
    RETURNS double DETERMINISTIC
BEGIN
    SET @RlatA = radians(latA);
    SET @RlonA = radians(lonA);
    SET @RlatB = radians(latB);
    SET @RlonB = radians(LonB);
    SET @deltaLat = @RlatA - @RlatB;
    SET @deltaLon = @RlonA - @RlonB;
    SET @d = SIN(@deltaLat/2) * SIN(@deltaLat/2) +
        COS(@RlatA) * COS(@RlatB) * SIN(@deltaLon/2)*SIN(@deltaLon/2);
    RETURN 2 * ASIN(SQRT(@d)) * 6371.01;
END//

Fetching all the pubs within a 250 meter radius around 51.5375°N, 0.1933°W is than as easy as running:

mysql> SELECT name, address, phone
       FROM poi
       WHERE DISTANCE(lat, lon, 51.5375, -0.1933) < 0.25;

With the result being:

| name            | address                        | phone           |
+-----------------+--------------------------------+-----------------+
| Mrs Betsy Smith | Kilburn High Road 77 , NW6 6HY | +44 20 76245793 |
| The Cock Tavern | Kilburn High Road 125          | NULL            |
| The Old Bell    | NULL                           | NULL            |
| The Westbury    | Kilburn High Road 34 , NW6 5UA | +44 20 76257500 |
+-----------------+--------------------------------+-----------------+
4 rows in set (0.72 sec)

If we want to also include the distance from the centre point in the result, we need to modify the query to:

mysql> SELECT name, DISTANCE(lat, lon, 51.5375, -0.1933) AS dist
       FROM poi
       HAVING dist < 0.25
       ORDER BY dist;

with as result:

| name            | dist                |
+-----------------+---------------------+
| Mrs Betsy Smith | 0.00825473345748987 |
| The Cock Tavern |     0.2420193460511 |
| The Old Bell    |   0.103123484090313 |
| The Westbury    |   0.150294300836645 |
+-----------------+---------------------+
4 rows in set (0.72 sec)

In both cases, the query takes 0.72 seconds. This is not overly fast, and the main reason for this is that the distance() function has to be called for every row in the table. An index can not be created on this either. However, what we can do is to filter out the results roughly first, by calculating a bounding box of latitude/longitude pairs around the centre point. Calculating the latitude boundaries can be done by:

d = (distInKm / 6371.01 * 2π) * 360
lat1 = centreLat - d
lat2 = centreLat + d

In our example that becomes:

d = (0.250 / 6371.01 * 2π) * 360 = 0.0022483
lat1 = 51.5375 - 0.0022483 = 51.5352.. -> 51.5351
lat2 = 51.5375 + 0.0022483 = 51.5397.. -> 51.5398

Or in PHP:

We round slightly up and down to combat inaccuracies in the calculations—it is a rough estimate after all. After adding the index on the lat column, and the index on the lon column, we reissue the query:

mysql> CREATE index poi_lat ON poi(lat);
mysql> CREATE index poi_lon ON poi(lon);

mysql> SELECT name, DISTANCE(lat, lon, 51.5375, -0.1933) AS dist
       FROM poi
       WHERE lat BETWEEN 51.5351 AND 51.5398
       HAVING dist < 0.25;

Which returns the same result as before, but faster:

| name            | dist                |
+-----------------+---------------------+
| The Westbury    |   0.150294300836645 |
| The Old Bell    |   0.103123484090313 |
| Mrs Betsy Smith | 0.00825473345748987 |
| The Cock Tavern |     0.2420193460511 |
+-----------------+---------------------+
4 rows in set (0.01 sec)

We can pre-filter the result set even more, by also limiting on the longitude boundary. This involves a few more calculations than for the latitude boundaries. The distance in degrees longitude that belongs to a distance in km depends on the latitude of the location. So first we need to calculate the latitude boundaries as we have done above, and with that information calculate the longitude boundaries.

In the first step (red), we calculate the northern and southern boundaries of the circle. We then calculate the western and eastern boundaries for the northern boundary (green), and southern boundary (blue).

In PHP this algorithm becomes:

If we use both functions we calculate as boundaries:

Which returns the query to execute:

SELECT name, DISTANCE(lat, lon, 51.5375, -0.1933) AS dist
FROM poi
WHERE lat BETWEEN 51.535251699514 AND 51.539748300486
  AND lon BETWEEN -0.19691479627963 AND -0.18968520372037
  HAVING dist < 0.25
  ORDER BY dist;

If we run this query, we get as result:

| name            | dist                |
+-----------------+---------------------+
| Mrs Betsy Smith | 0.00825473345748987 |
| The Old Bell    |   0.103123484090313 |
| The Westbury    |   0.150294300836645 |
| The Cock Tavern |     0.2420193460511 |
+-----------------+---------------------+
4 rows in set (0.00 sec)

As you can see, we are getting the result a lot faster now—0.00 sec vs 0.72 sec. Please do note, that if we would just have used the boundaries without using the HAVING dist < 0.25 clause, we would have gotten one extra result that is slightly too far away (0.281 km):

| name            | dist                |
+-----------------+---------------------+
| Mrs Betsy Smith | 0.00825473345748987 |
| The Old Bell    |   0.103123484090313 |
| The Westbury    |   0.150294300836645 |
| The Cock Tavern |     0.2420193460511 |
| Bar Hemia       |   0.281138534935208 |
+-----------------+---------------------+

Conclusion

In this article we saw how we can use a MySQL stored procedure to find our pubs and bars within a certain distance from a central location. In the next part, we will be looking on how to solve the same problem with PostgreSQL.

Spatial Indexes: Fetching Data/SQLite

Thu, 31 Mar 2011 08:17:00 +0000

Spatial Indexes: Fetching Data/SQLite

London, UK

Thursday, March 31st 2011, 09:17 BST

In a previous article I introduced 'The Flat Earth Model' and the 'The spherical Earth model'. In this article we're going to have a look at fetching a data set and importing them into a SQLite database to query from PHP. What better data set is there to import than all of the UK's pubs?

Getting the Data

In order to get a suitable data set we are going to use the data from the OpenStreetMap project. This project is mainly concerned with making an open map of the entire globe, but it also contains a vast database of points-of-interest (POI). OpenStreetMap contributes seems to be a big fan of pubs, and hence they are mapped really well. POIs can either be stored as a node (a single point with a geographical location) such as The Long Acre or as a closed way (an ordered collection of nodes where the first and last node are the same), such as Brondes Age.

I will demonstrate two methods to fetch the pubs in an XML format containing nodes and ways. The first method is with XAPI. XAPI is an interface to the OpenStreetMap database to allow users to query and filter items. In order to fetch data through it, you specify a bounding box and a predicate. A bounding box specifies the most Eastern, Southern, Western and Northern coordinates. The UK has roughly as bounding box 9.05°W, 48.77°N, 2.19°E, 58.88°N, or short: -9.05, 48.77, 2.19, 58.88. Fetching the data can simply be done by querying the server with wget:

wget -O pubs.xml 'http://xapi.openstreetmap.org/api/0.6/*[amenity=pub][bbox=-9.05,48.77,2.19,58.88]'

This is going to take a long time; and will most likely just not work or time-out. The current XAPI server, written in an obscure programming language called MUMPS, is extremely unreliable and is really slow. A new version of XAPI build in Java is on the way, but right now it's limited to 10 square degrees.

Luckily, there is an alternative in the form of parsing (an extract of) the planet file. The planet file is an enormous database dump of OpenStreetMap's data. The people at Geofabrik have extracts for specific parts of the world at http://download.geofabrik.de/osm/. From them I downloaded the europe/great_britain.osm.pbf file (274 MB) for my example. With a tool called Osmosis we can then filter out all pubs into a similar formatted XML file. I am only giving the command to do our task at hand, but a detailed usage guide for Osmosis is available too. The command runs in about four minutes and looks like:

./osmosis-0.38/bin/osmosis -v 5 \
\
--read-pbf file=great_britain.osm.pbf \
--tf reject-relations \
--tf accept-nodes amenity=pub,bar \
--tf reject-ways \
outPipe.0=POI \
\
--read-pbf file=great_britain.osm.pbf \
--tf reject-relations \
--tf accept-ways amenity=pub,bar \
--used-node \
outPipe.0=area \
\
--merge inPipe.0=POI inPipe.1=area \
--write-xml file=great_britain_pubs.osm

Importing the Data

The resulting XML file has two important elements: nodes and ways. The XML for The Long Acre looks like:

Important here are the latitude (51.51°N) and longitude (0.12°W) and of course, the name of the pub in the tag element.

And the XML for Brondes Age (a way) is a bit more complex, and looks like:

There is no latitude and longitude, but instead there are references to nodes (the nd elements). There is also a large collection of descriptive tags, such as addr:housenumber and contact:phone. In order to calculate the latitude and longitude of Brondes Age we can either take the correct approach by calculating the centroid or we can simply take the average latitude and longitude of all the nodes. To make things simple, I will take the simple approach.

In order to find the latitude and longitude of all the nodes, we simply scan through the file again and find all the nodes that correspond to the ref attribute of each nd element. We should disregard the last one of each way though, as it is the same as the first one.

SQLite

To start, we will use a very simple RDBMS: SQLite. We will still have to define a database schema. We can simply do that with:

derick@whisky:~$ sqlite pois.sqlite
sqlite> CREATE TABLE poi(id int, type int, lat float, lon float, name char, address char, cuisine char, phone char);

The importing of data then can be done by this "simple" script (after adjusting the path to eZ Components/ Zeta Components) with:

php parsepoi.php.txt great_britain_pubs.osm

After running the import script, there should be about 28000 POIs in the database.

Querying the Data

Once we have imported the POIs into our SQLite database, we are ready to query them. SQLite does not have a very extensive set of functions so we can not do the calculation in the query directly. Just to iterate from the previous article in this series, the formula for calculating the distance in a spherical Earth model is:

One solution would be to query the database, and then use the distance() function to filter out unwanted elements, like:

createSelectQuery();
$q->select( '*' )->from( 'poi' );
$s = $q->prepare();
$s->execute();

foreach ( $s as $res )
{
  $e = distance( $lat, $lon, $res['lat'], $res['lon'] );
  if ( $e < $wantedD )
  {
    echo sprintf( '%.3f,%.3f %-40s %.2f km away',
      $res['lat'], $res['lon'], $res['name'],
      $e ), "\n";
  }
}
?>

This will show all pubs in a 250 meter radius around 51.53°N, 0.19°W:

derick@whisky:/home/httpd/html/test/maps$ php fetch-sqlite-simple.php
51.538,-0.193 Mrs Betsy Smith                          0.01 km away
51.539,-0.195 The Cock Tavern                          0.24 km away
51.537,-0.192 The Old Bell                             0.10 km away
51.537,-0.192 The Westbury                             0.15 km away

Of course, this is not very efficient as all items are selected from the database, and then filtered out depending on their calculated distance. Luckily, SQLite supports user defined functions written in PHP. This would mean that we can increase performance a bit by letting PHP's internals call the distance function for every row:

sqliteCreateFunction( 'dist', 'distance' );

// Centre point
$lat = 51.5375;
$lon = -0.1933;

// Distance (in km)
$wantedD = 0.25;

$q = $d->createSelectQuery();

// Use the user defined dist() function as additional column
$q->select( "*, dist($lat, $lon, lat, lon) as e" )
  ->from( 'poi' )
  ->where( "e < $wantedD" );

$s = $q->prepare();
$s->execute();

foreach ( $s as $res )
{
  echo sprintf( '%.3f,%.3f %-40s %.2f km away',
    $res['lat'], $res['lon'], $res['name'],
    $res['e'] ), "\n";
}
?>

The result is naturally the same as before.

Conclusion

In this installment we have seen how to retrieve information from OpenStreetMap's database and import them into SQLite for querying. In the next installment we will have a look at how to import and query with MySQL and PostgreSQL.

Xdebug 2.1.1

Mon, 28 Mar 2011 08:40:00 +0000

Xdebug 2.1.1

London, UK

Monday, March 28th 2011, 09:40 BST

I've just released Xdebug 2.1.1. It is a bug fix release that primarily addresses issues in the debugging and profiling functionality of Xdebug.

Among the bug fixes is a fix that solves an issue with newer versions of the KCacheGrind profiling GUI. The newer versions are more strict with input which causes an issue with Xdebug profiler's files. Xdebug 2.1.1 solves this.

As an experiment I set up a pledge system to allow myself to work on this (and related issues) during my "work time". Twenty-four pledges were made, in about two weeks; matching the goal. Sebastian, Jan, Michael, Pale Purple, Jeff, Christoph, Karel, Yannick, Jake, Venakis, Brian, Simon, Kenneth, DM Baker, "gizmola", Ladislav, Volker, React, Michal and three anonymous supporters—thanks for your support!

Another main issue that was fixed was that line numbers (in function traces, profile files and while debugging) for "magic" function calls (such as __get(), __call() etc) were always recorded as 0. This made it impossible to set breakpoints for example.

You can find Xdebug 2.1.1's source and Windows binaries at the download page. A full ChangeLog is available as well. Through twitter you can kept up-to-date about Xdebug's development efforts and support is available through the mailinglist. If you think that Xdebug is a valuable tool, you can donate via PayPal.

Sponsoring Xdebug

Tue, 22 Mar 2011 18:30:00 +0000

Sponsoring Xdebug

London, UK

Tuesday, March 22nd 2011, 18:30 GMT

Over the past 7 years I've spend countless hours making Xdebug an awesome development and debugging tool. I love working on it; it is a good way to get familiar with PHP's "interesting" internals, and it also helps PHP developers finding (potential) issues within their code base faster.

With the 2.0 release announcement I asked users of Xdebug to send me a postcard if they liked it; and since the 2.1 announcement I mention that it is possible to give donations through PayPal. I have received quite few postcards and donations.

Recently, an issue with KCacheGrind and Xdebug's profiler functionality became known. As an experiment Sebastian suggested to see whether it would be a good idea to set-up a pledge-like system to arrange some funding so that I can dedicate my "work time" to working on Xdebug issues and features. I set-up a campaign on Pledgie for this purpose. Both Sebastian and I expected Pledgie to work a bit different than it actually did. Instead of holding on to the pledged money until the issue was implemented, it transfered the pledges directly to my PayPal account.

Twenty-four pledges were made, in about two weeks; matching the goal. Thanks Sebastian, Jan, Michael, Pale Purple, Jeff, Christoph, Karel, Yannick, Jake, Venakis, Brian, Simon, Kenneth, DM Baker, "gizmola", Ladislav, Volker, React, Michal and three anonymous supporters!

Last night I committed several patches to the Xdebug repository, including the one that sparked the idea to set-up a pledge to fund Xdebug's profiler files. These patches, and a few others, now form part of the upcoming Xdebug 2.1.1 release. I've published the source package for the first (and most likely only) release candidate: Xdebug 2.1.1RC1. Windows binaries will follow shortly. You can also install/upgrade the release candidate through PECL: pecl install xdebug-beta or pecl upgrade xdebug-beta.

The Pledgie campaign has now been closed, but I am intending to set-up new ones for more of the elaborate features—most likely starting with one related to Xdebug's code coverage functionality. In the meanwhile, please test Xdebug 2.1.1RC1, and let me know your comments about sponsoring specific Xdebug features. I plan on releasing to write up some more thoughts about that soon. I expect Xdebug 2.1.1 to come out before the end of the month. If you think Xdebug is useful, feel free to donate as well!

Spatial Indexes: Calculating Distance

Wed, 09 Mar 2011 12:17:00 +0000

Spatial Indexes: Calculating Distance

Montréal, Canada

Wednesday, March 9th 2011, 07:17 EST

During my "Geolocation and Mapping with PHP" talk that I've given a few times I briefly touch on the subject of indexes on data-sets of spatial data. This isn't as simple as just solving Pythagoras theorem and this article is meant to clarify this.

The flat Earth model

Pythagoras theorem can be used to calculate the distance between two points quite easily; you take the square root of the square of the absolute vertical distance plus the square of the absolute horizontal distance; in short:

d = √(|x1 - x2|² + |y1 - y2|²)

or in PHP:

$d = sqrt(pow(abs($x1 - $x2), 2) + pow(abs($y1 - $y2), 2));

If you take for example London's coordinates (51.50°N, 0.13°W) and Amsterdam's coordinates (52.37°N, 4.90°E) we can calculate the distance with:

d = √(|-0.13 - 4.90|² + |51.50 - 52.37|²)
d = √(5.03² + 0.87²)
d = √(26.0578)
d ≅ 5.10

And on a map:

But what does a difference of "5.10°" actually mean? How far is this in useful units, such as meters?

If we show the whole map of which the above is an extract, we come to:

The distance around the equator, and through the poles is roughly the same, 40.000km (please be aware that the blue line only shows half of it, the other half is going through the anti-meridian at 180°W/E). 5.03° in East-West difference is then about 40 000 ✕ (5.03/360) = 559 km and the North-South difference about 20 000 ✕ (0.87/180) = 97 km. Using those numbers within the Pythagoras theorem we end up with a distance of √(559² + 97²) = 567 km. Although the calculation is correct, the answer is still wrong. The real distance is closer to 360 km.

The spherical Earth model

If we look at the Earth in its original (mostly) spherical 1 shape, then it's clear that 10° longitude (East/West) at 60°N is going to be less of a distance than 10°E/W at the equator. It's actually fairly easy to calculate how much 1° longitude is at 60°N by using cos(60) * 1/360 * 6371 * 2π 2. Or in PHP:

This returns 55.597 km per ° for 60°N. The same distance in degrees on the Equator gives 6371 * 2 * M_PI / 360 = 111.195.

The following diagram shows ones more that latitudinal degrees always correspond with the same distance in kilometer, whereas longitudinal degrees differ.

In the diagram the line A is a line from 0°N, 90°W to 10°N, 90°W. It has the same length as line B, from 30°N, 90°W to 40°N, 90°W: a 36th of the circumference of the Earth through the poles. Line C, from 0°N, 30°W to 0°N, 20°W has the same length. Line D however, from 50°N, 30°W to 50°N, 20°W is shorter by a factor of cos(50°) ≅ 0.64.

If we look again at the distance between London and Amsterdam, ignore the differences in latitude and instead pick the average, we see:

Which is a bit shorter than the expected 360km, but that's because we conveniently forgot about the difference in latitude.

Sadly, we can't use Pythagoras's theorem to calculate the real distance with the latitude difference taken account as well. This is because the theorem is meant for Euclidean geometry, and a sphere does not follow the rules of this geometry. Instead we need to use a formula that is called the great-circle distance formula. The main concept behind it is that a circle is drawn across the whole sphere that connects both the start (point P) as well as the end (point V). Then with that circle the distance can be calculated. The following diagram shows this:

I will spare you how the function is derived, but the distance calculation ends up as being:

If we punch in our original numbers form London (51.50°N, 0.13°W) and Amsterdam (52.37°N, 4.90°E), we calculate the following:

Which gives us the expected result of 358.07 km.

Conclusion

I hope that the above clarified the difference between 2D spatial indexing with the flat Earth model and spatial indexing of geo-located data (the spherical Earth model). In future articles I will go into specific implementations of spatial indexing by traditional databases such as SQLite, MySQL and PostGreSQL; NoSQL databases such as MongoDB and CouchDB; and Solr.

1
The Earth is not really a sphere, but an approximation of it. However, doing the same calculations for an ellipsoid can (as far as I know) only be done by approximation. The difference would hardly matter for finding the closest pub.
2
In this article, I've used an average radius of the Earth of 6371 km.