Derick Rethans - tag: php

Collecting Garbage: PHP's take on variables

Tue, 31 Aug 2010 08:23:00 +0000

Collecting Garbage: PHP's take on variables

Tuesday, August 31st 2010, 09:23 BST

This is the first part of three-parts column that was originally published in the April 2009 issues of php|architect.

In this three part column I will explain the merits of the new Garbage Collection (also known as GC) mechanism that is part of PHP 5.3. Before we start with the intricate details of PHP's new GC engine I will explain why it is actually needed. This, combined with an introduction how PHP deals with variables in general is explained in this first part of the column. The second part will cover the solution and some notes on the GC mechanism itself, and the third part covers some implications of the GC mechanism, as well as some benchmarks. But now first on to the introduction.

PHP stores variables in containers called a "zval". A zval container contains besides the variable's type and value, also two additional bits of information. The first one is called "is_ref" and contains a boolean value whether this variable is part of a "reference set". With this bit PHP's engine knows how to differentiate between normal variables, and references. However, PHP has user-land references—as created by the & operator, but also an internal reference counting mechanism to optimize memory usage. The second piece of additional information, called "refcount", contains how many variables names—also called symbols—point to this one zval container. All symbols are stored in a symbol table, of which there is one per scope. There is a scope for the main script (ie, the one requested through the browser), as well as for every function or method.

A zval container is created when a new variable is created with a constant value, such as:

$a = "new string";

In this case the new symbol name "a" is created in the current scope, and a new variable container is created with type "string", value "new string". The "is_ref" bit is by default set to "false" because no user-land reference has been created. The "refcount" is set to "1" as there is only one symbol that makes use if this variable container. Also, if the "refcount" is "1", "is_ref" is always "false". If you have Xdebug installed you can display this information by calling:

xdebug_debug_zval('a');

which displays:

a: (refcount=1, is_ref=0)='new string'

Assigning this variable to another variable name, increases the refcount:

$a = "new string";
$b = $a;
xdebug_debug_zval( 'a' );

which displays:

a: (refcount=2, is_ref=0)='new string'

The refcount is "2" here, because the same variable container is linked with both "a" and "b". PHP is smart enough not to copy the actual variable container when it is not necessary. Variable containers get destroyed when the "refcount" reaches zero. The "refcount" gets decreased by one for each symbol linked to the variable container leaves the scope (f.e. if the function ends) or when unset() is called on a symbol. The following example shows that:

$a = "new string";
$c = $b = $a;
xdebug_debug_zval( 'a' );
unset( $b, $c );
xdebug_debug_zval( 'a' );

which displays:

a: (refcount=3, is_ref=0)='new string'
a: (refcount=1, is_ref=0)='new string'

If we now call "unset( $a );" the variable container, including the type and value will be removed from memory.

Things get a tad more complex with compound types such as arrays and objects. Instead of a scalar value, arrays and objects store their properties in a symbol table of their own. This means that the following example creates three zval containers:

$a = array( 'meaning' => 'life', 'number' => 42 );
xdebug_debug_zval( 'a' );

which displays (after formatting):

a: (refcount=1, is_ref=0)=array (
        'meaning' => (refcount=1, is_ref=0)='life',
        'number' => (refcount=1, is_ref=0)=42
)

Graphically, it looks like:

You can see the three zval containers here: "a", "meaning" and "number". Similar rules apply for increasing and decreasing "refcounts". Below we add another element to the array, and set it's value to the contains of an already existing element:

$a = array( 'meaning' => 'life', 'number' => 42 );
$a['life'] = $a['meaning'];
xdebug_debug_zval( 'a' );

which displays (after formatting):

a: (refcount=1, is_ref=0)=array (
        'meaning' => (refcount=2, is_ref=0)='life',
        'number' => (refcount=1, is_ref=0)=42,
        'life' => (refcount=2, is_ref=0)='life'
)

Graphically, it looks like:

From the above we see that both the old and new array element now point to a zval container which "refcount" is "2". Of course there are now two zval containers, but they are the same one. The xdebug_debug_zval() function does not show this, but could, by also displaying the memory pointer. Removing an element from the array is like removing a symbol from a scope. By doing so, the "refcount" of a container that an array element points to is decreased. Again when the "refcount" reaches zero, the variable container is removed from memory. Again an example to show this:

$a = array( 'meaning' => 'life', 'number' => 42 );
$a['life'] = $a['meaning'];
unset( $a['meaning'], $a['number'] );
xdebug_debug_zval( 'a' );

which displays (after formatting):

a: (refcount=1, is_ref=0)=array (
        'life' => (refcount=1, is_ref=0)='life'
)

Now, things get interesting if we add the array itself as an element of the array, which we do in the next example—in which I also sneaked in an reference operator as otherwise PHP would create a copy here:

$a = array( 'one' );
$a[] =& $a;
xdebug_debug_zval( 'a' );

which displays (after formatting):

a: (refcount=2, is_ref=1)=array (
        0 => (refcount=1, is_ref=0)='one',
        1 => (refcount=2, is_ref=1)=...
)

Graphically, it looks like:

You can see that the array variable ("a") as well as the second element ("1") now point to a variable container that has a "refcount" of "2". The "..." in the display above shows that there is recursion involved, which of course in this case it means that the "..." points back to the original array.

Just like before, unsetting a variable removes the symbol, and the reference count of the variable container it points to is decreased by one. So if we unset variable $a after running the above code, the reference count of the variable container that $a and element "1" point to gets decreased by one, from "2" to "1". This can be represented like:

(refcount=1, is_ref=1)=array (
        0 => (refcount=1, is_ref=0)='one',
        1 => (refcount=1, is_ref=1)=...
)

Graphically, it looks like:

Although there is no symbol in any scope pointing to this structure anymore, it can not be cleaned up either because the array element "1" still points to this same array. Because there is no external symbol pointing to it, there is no way for a user to clean up this structure anymore, and thus you get a memory leak. Fortunately, PHP will clean up this data structure at the end of the request, but before then this is taking up valuable space in memory. The mentioned situation happens often if you're implementing parsing algorithms or other things where you have a child point back at a "parent" element. The same situation can also happen with objects of course, where it actually happens easier as objects are always implicitly used by reference.

This might not be a problem if this only happens once or twice, but if there is thousands, or even millions of these memory losses, this obviously starts being a problem. Especially in long running scripts, such as daemons where the request basically never ends, or in large sets of unit tests. The latter caused problems for us while running the unit tests for the Template component of the eZ Components library. In some cases it would require over 2 GiB of memory, which our test server didn't quite have.

With that we conclude this introduction, for more information on how PHP deals with variables I can point you at June 2005 issue of php|architect. That article is also available on-line as PDF (http://derickrethans.nl/files/phparch-php-variables-article.pdf). In the next installment, we're going to discuss the solution to the memory leak problem with circular references.

64-bit integers in MongoDB

Mon, 09 Aug 2010 13:23:00 +0000

64-bit integers in MongoDB

London, UK

Monday, August 9th 2010, 14:23 BST

The current project that I'm working on relies heavily on MongoDB, a bridge between key-value stores and traditional RDBMS systems. Users in this project are identified by their Facebook UserID, which is a "64-bit int datatype". Unfortunately, the MongoDB PHP Driver only had support for 32-bit integers causing problems for newer users of Facebook. For those users, their nice long UserID was truncated to only 32 bits which didn't quite make the application work.

MongoDB stores documents internally in something called BSON (Binary JSON). BSON has two integer types, a 32-bit signed integer type called INT and a 64-bit signed integer type called LONG. The documentation of the MongoDB PHP driver on types says (or used to say, depending on when you're reading this) that only the 32-bit signed integer type is supported because "PHP does not support 8 byte integers". That's not quite true. PHP's integer type supports 64-bit on the platforms where the C-data type long is 64 bits. That is generally on every 64-bit platform (where PHP is compiled for 64 bits); except on Windows, where the C-data type long is always only 32 bits.

Whenever a PHP integer is sent to MongoDB, the driver would use the 32 least significant bits to store the number as part of the document. The example here shows what happens (on a 64-bit platform):

selectCollection('test', 'inttest');
$c->remove(array());

$c->insert(array('number' => 1234567890123456));

$r = $c->findOne();
echo $r['number'], "\n";
?>

shows:

int(1015724736)

In binary:

1234567890123456 = 100011000101101010100111100100010101011101011000000
      1015724736 =                      111100100010101011101011000000

Truncating data is obviously not a very good idea. In order to address this issue we could just allow for the native PHP integer type to be used when storing data from PHP into MongoDB. But instead of changing how the MongoDB driver works by default I've added the new setting mongo.native_long — simply because otherwise we might be breaking applications. With the mongo.native_long setting enabled, we see the following result instead of the outcome of the script above:

insert(array('number' => 1234567890123456));

$r = $c->findOne();
var_dump($r['number']);
?>

This script shows:

int(1234567890123456)

On 64-bit platforms, the mongo.native_long setting allows for 64-bit integers to be stored in MongoDB. The MongoDB data type that is used in this case is the BSON LONG, instead of the BSON INT that is used if this setting is turned off. The setting also changes the way how BSON LONGs behave when they are read back from MongoDB. Without mongo.native_long enabled, the driver would convert every BSON LONG to a PHP double which results in the loss of precision. You can see that in the following example:

insert(array('number' => 12345678901234567));

ini_set('mongo.native_long', 0);
$r = $c->findOne();
var_dump($r['number']);
?>

This script shows:

float(1.2345678901235E+16)

On 32-bit platforms, the mongo.native_log setting changes nothing for storing integers in MongoDB: the integer is stored as a BSON INT as before. However, when the setting is enabled and a BSON LONG is read from MongoDB a MongoCursorException is thrown alerting you that the data could not be read back without losing precision:

MongoCursorException: Can not natively represent the long 1234567890123456 on this platform

If the setting is not enabled, a BSON LONG is converted to a PHP float in order to avoid breaking backwards compatibility with the current behaviour.

Although the mongo.native_long settings allows for 64-bit support on 64-bit platforms, it doesn't provide much for 32-bit platforms except preventing loss of precision while reading BSON LONGs—and then by just throwing an exception.

As part of making 64-bit integers work reliably with MongoDB, I've also added two new classes: MongoInt32 and MongoInt64. These two classes are simple wrappers around a string representation of a number. They are instantiated like this:

You can use those objects in normal insert and update queries just like normal numbers:

selectCollection('test', 'inttest');
$c->remove(array());

$c->insert(array(
        'int32' => new MongoInt32("1234567890"),
        'int64' => new MongoInt64("12345678901234567"),
));

$r = $c->findOne();
var_dump($r['int32']);
var_dump($r['int64']);
?>

Which shows:

int(1234567890)
float(1.2345678901235E+16)

As you can see, nothing is changed with how values are returned. A BSON INT is still returned as an integer, and a BSON LONG as a double. If we turn on the mongo.native_long setting then the BSON LONG that was stored through the MongoInt64 class is returned as a PHP integer on 64-bit platforms, and a MongoCursorException is thrown on 32-bit platforms.

In order to received 64-bit integers back from MongoDB on 32-bit platforms, I've added another setting: mongo.long_as_object. This will (on any platform) return a BSON LONG as stored in MongoDB as a MongoInt64 object. The following script demonstrates that:

selectCollection('test', 'inttest');
$c->remove(array());

$c->insert(array(
        'int64' => new MongoInt64("12345678901234567"),
));

ini_set('mongo.long_as_object', 1);
$r = $c->findOne();
var_dump($r['int64']);
echo $r['int64'], "\n";
echo $r['int64']->value, "\n";
?>

This script outputs:

object(MongoInt64)#7 (1) {
  ["value"]=>
  string(17) "12345678901234567"
}
12345678901234567
12345678901234567

The MongoInt32 and MongoInt64 classes implement __toString() so that their values can be echoed. You can only get their values out as strings. Please be aware that MongoDB is type-sensitive, and will not treat a number contained in a string the same way as a number that's just a number. This script shows this (on a 64-bit platform):

selectCollection('test', 'inttest');
$c->remove(array());

$nr = "12345678901234567";
$c->insert(array('int64' => new MongoInt64($nr)));

$r = $c->findOne(array('int64' => $nr)); // $nr is a string here
var_dump($r['int64']);
$r = $c->findOne(array('int64' => (int) $nr));
var_dump($r['int64']);
?>

Which shows:

NULL
int(12345678901234567)

The following tables summarises all the different conversions regarding integers, depending on which settings are enabled:

PHP to MongoDB on 32-bit systems:

From PHP	Stored in Mongo
From PHP	native_long=0	native_long=1
1234567	INT(1234567)	INT(1234567)
123456789012	FLOAT(123456789012)	FLOAT(123456789012)
MongoInt32("1234567")	INT(1234567)	INT(1234567)
MongoInt64("123456789012")	LONG(123456789012)	LONG(123456789012)

PHP to MongoDB on 64-bit systems:

From PHP	Stored in Mongo
From PHP	native_long=0	native_long=1
1234567	INT(1234567)	LONG(1234567)
123456789012	garbage	LONG(123456789012)
MongoInt32("1234567")	INT(1234567)	INT(1234567)
MongoInt64("123456789012")	LONG(123456789012)	LONG(123456789012)

Mongo to PHP on 32-bit systems:

Stored in Mongo	Returned to PHP as
	long_as_object=0		long_as_object=1
	native_long=0	native_long=1	long_as_object=1
INT(1234567)	int(1234567)	int(1234567)	int(1234567)
LONG(123456789012)	float(123456789012)	MongoCursorException	MongoInt64("123456789012")

Mongo to PHP on 64-bit systems:

Stored in Mongo	Returned to PHP as
	long_as_object=0		long_as_object=1
	native_long=0	native_long=1	long_as_object=1
INT(1234567)	int(1234567)	int(1234567)	int(1234567)
LONG(123456789012)	float(123456789012)	int(123456789012)	MongoInt64("123456789012")

Conclusion

Getting 64-bit support right with MongoDB can be tricky as we've seen. My recommendations would be to use just mongo.native_long=1 if you only deal with 64-bit platforms for your code. In this case, every integer number that you put into MongoDB will also come out as an integer number; with 64-bit integers supported.

If you however have to deal with 32-bit platforms (remember, that includes a 64-bit Windows build of PHP) then you can not reliably use just PHP's integer types and you have to use the MongoInt64 class. This comes with the restriction that you have to deal with numbers in strings for initialisation. You also need to be aware that the MongoDB shell regards all numbers as floating point numbers, and that it can not represent 64-bit integer values. Instead they will show up as floating point numbers. Do not attempt to modify those numbers on the shell, as that could change the type.

So with the script:

selectCollection('test', 'inttest');
$c->remove(array());

$c->insert(array('int64' => new MongoInt64("123456789012345678")));

The MongoDB shell mongo behaves like:

$ mongo
MongoDB shell version: 1.4.4
url: test
connecting to: test
type "help" for help
> use test
switched to db test
> db.inttest.find()
{ "_id" : ObjectId("4c5ea6d59a14ce1319000000"), "int64" : { "floatApprox" : 123456789012345680, "top" : 28744523, "bottom" : 2788225870 } }

Of course, when fetching data through a driver that supports 64-bit integers you get the proper result:

ini_set('mongo.long_as_object', 1);
$r = $c->findOne();
var_dump($r['int64']);
?>

Which shows:

object(MongoInt64)#7 (1) {
  ["value"]=>
  string(18) "123456789012345678"
}

The new functionality as outlined in this article is part of the 1.0.9 mongo release that's available through PECL with pecl install mongo. Good luck with your 64-bit integers!

VMWare on Debian Unstable

Sat, 07 Aug 2010 15:08:00 +0000

VMWare on Debian Unstable

London, UK

Saturday, August 7th 2010, 16:08 BST

In the past week I've been doing some work on the MongoDB PHP Driver, and while running the test cases I noticed that there are some issues with it when you run out of disk space. It's sadly hard to reproduce so I set out to create a VMWare image for the MongoDB developers to reproduce this issue.

Xdebug 2.1

Tue, 29 Jun 2010 14:17:00 +0000

Xdebug 2.1

London, UK

Tuesday, June 29th 2010, 15:17 BST

I've just released Xdebug 2.1. This new release contains all the bug fixes and new features that have been developed over the past three years. It provides full PHP 5.3 support, but it no longer supports PHP versions below 5.1.

A description of all the new features follows:

Error Collection

I've added xdebug_start_error_collection(), xdebug_stop_error_collection() and xdebug_get_collected_errors(), which allow you to collect all notices, warnings and error messages that Xdebug generates from PHP's error_reporting functionality so that you can output them at a later point in your script by hand.

Header Setting Interception

All functions that set HTTP headers such as with header() and setcookie() are now intercepted by Xdebug. The intercepted headers are stored internally in an array that can be retrieved by calling the xdebug_get_headers() function. This is very useful in cases where you need to test certain functionality that sets headers somewhere deep in code. This function is also used in eZ Components' test suite to test whether the correct HTTP headers are set in the MvcTools component.

Variable Assignment Tracing

The new setting xdebug.collect_assignments allows you to record changes to variables in scripts to trace files. I've already written more about it in Variable tracing with Xdebug.

"Scream" Support

The scream PECL extension disables the @ (shut-up) operator to actually see all notices, warnings and errors that PHP generates. The scream extension's functionality have been duplicated as Xdebug's xdebug.scream php.ini setting. Why disabling the @-operator is a good thing, I've already outlined in Five reasons why the shut-op operator (@) should be avoided.

Additions for Stack Traces

All HTML containers generated by Xdebug (error messages, xdebug_var_dump() output) now contain a CSS class for easier styling.

The new setting xdebug.file_link_format allows you to turn any file/line link in Xdebug's output to be decorated to a link with a specific format. For example you can set it to txmt://open/?file://%f&line=%l for opening files directly in Textmate, or gvim://%f@%l with some shell script for gvim and Firefox.

Remote Debugging

The new setting xdebug.remote_connect_back, contributed by Lucas Nealan and Brian Shire, allows Xdebug to try to make a debugging connection to the IP address from which the browser request came from. This setting is an additional solution for debugging with multiple users.

The old gdb and php3 remote debugging engines have been removed.

Overloaded var_dump

The newly introduced setting xdebug.overload_var_dump can be used to turn off Xdebug's default behavior of overriding PHP's var_dump() function with the xdebug_var_dump() function that uses pretty HTML for formatting a variable structure. It does not stop xdebug_var_dump() from working however.

Donations

You can probably imagine that writing and support Xdebug is a time-consuming effort. It's a tool that saves a lot of work during developement of applications, as well while debugging. If you think Xdebug is valuable for you, perhaps you would like to consider a donation. All donations are greatly appreciated and support future development and availablity of Xdebug.

If you want to sponsor a (new) feature in Xdebug, feel free to contact me as well.

Download and Support

The new version of Xdebug can be downloaded through the download page. A list with all changes can be found on the updates page. Support information can be found on the support page, and issues and feature requests can be filed in the bug tracker. Before asking for support, please have a look at the FAQ first.

Yahoo! Search's new front page

Thu, 06 May 2010 10:10:00 +0000

Yahoo! Search's new front page

London, UK

Thursday, May 6th 2010, 11:10 BST

Yahoo! Search recently changed their front page to include three of "Earth's Biggest Puzzles". As a long time user I find this really obnoxious especially because there doesn't seem to be any way to turn this off. So I decided to do something about it and I wrote a Greasemonkey script to get rid of it again. You can find the script here.

Find my Xdebug download wizard

Mon, 03 May 2010 09:00:00 +0000

Find my Xdebug download wizard

London, UK

Monday, May 3rd 2010, 10:00 BST

Installing Xdebug seems to be a problem for some users. The main trouble is for Windows users that don't know which file to download. Over the past few days I've worked on a wizard that analyses phpinfo()'s output and for Windows users suggests:

which file to download
where to put the file
which php.ini file to modify, and how

For Unix users, it explains:

which source tarball to download
how to compile
which php.ini file to modify, and how

I hope this makes it a bit easier to get going with Xdebug. Next up (hopefully this week) is release candidate 2 for Xdebug 2.1.0.

PHP and Ordnance Survey Mapping

Tue, 27 Apr 2010 13:27:00 +0000

PHP and Ordnance Survey Mapping

London, UK

Tuesday, April 27th 2010, 14:27 BST

About a month ago, Ordnance Survey opened up some of their data for public consumption under a brand called OpenData. The data is licenced under a "Creative Commons Attribute"-like license. One of the data sets they provide is Code-Point Open and provides a dataset "that contains postcode units, each of which have a precise geographical location". I've always been a bit of a map-geek, and have always geotagged my pictures and some of my tweets. So I was wondering what cool thing I could do with this newly released data. I decided to map all the postcodes onto the UK map where more postcodes for a specific place would create a "lighter" colour. Each postcode has on average about 15 addresses, so in more densely populated areas you have more "postcodes-per-area". Doing this wasn't very difficult and it resulted in the following map:

You can very clearly see the more densely populated areas such as London.

This generated postcode-density map I wanted to overlay on a real map, such as OpenStreetMap provides. When I tried to align the image that I generated with the OSM map, I ended up with this:

Obviously, the maps don't align. In order to fix this, I ended up learning a lot more about map projections.

The Code-Point Open data uses the UK's National Grid to store location data in. The National Grid consists of 100*100 km squares that are then further subdivided into smaller squares creating grid references such as TQ3012780512, or the numerical version 530128 180512. TQ translates into the "hundred thousands" of the Eastings and Northings according to the grid that you can see here. In this case, it specifies a point 530 128 meters East and 180 512 meters North of the origin. (If you work it out, you'll end up in London).

OpenStreetMap uses a Mercator projection to visualize maps. The Mercator projection is a cylindrical map projection, and it distorts the size and shape of large objects, as the scale increases from the Equator to the poles. Therefore it only works from about 85°N to 85°S (why it is 85° only be came clear after doing all the maths for it). Google Maps uses the same projection.

There were a few challenges plotting the Code-Point Open data on an OpenStreetMap map:

The National Grid coordinates need to be converted to Latitude/Longitude pairs
Latitude/Longitude pairs need to be mapped to pixels to align with the Mercator projection of OpenStreetMap maps.

For converting National Grid references to Latitude/Longitude pairs I found some JavaScript code. I converted this code to PHP and to verify whether my code was working properly, I used this site.

Converting Latitude/Longitude pairs to the pixel coordinates of OSM required a bit of maths. OSM provides maps in different zoom levels, from 2 to 17. Each zoom level having twice the amount of pixels horizontally and vertically. Zoom level 2 has 4 times 4 tiles, where each tile is 256*256 pixels. At zoom level two, the whole world fits into 1024*1024 pixels. The number of pixels for each axis for a specific zoom-level can be calculated by:

pow(2^zoom) * 256

For a zoom level of 7 that makes 16384 pixels in each direction. To convert a longitude (in the range -180° to 180°) we can simply apply:

x = ((lon + 180) / 360) * (pow(2^zoom) * 256)

For a Longitude of 0.003117° E at zoom-level 13 that turns out to be pixel 1048594.

Longitude conversions are more difficult due to the Mercator projection itself. To convert we apply:

y = ((atanh(sin(deg2rad(-lat))) / π) + 1) * (pow(2^(zoom-1)))

For a Latitude of 51.502817° N at zoom-level 13 that turns out to be pixel 697399.

Creating an image for the whole world at zoom level 13 is impractical as it is 2097152*2097152 pixels and downloading all the 67 million tiles for this zoom level is probably not liked by the OSM people either. So instead we take a cut out for a specific area only. For the UK (61.37°N -9.49° W, 49.76°N +3.63°E) at zoom level 7 we end up with a 1536x2048 map with the North-Western pixel being 15360,9216. On this map, we can draw the latitudes and longitudes as well as the National Grid lines (full image is on flickr):

The only thing left to do now, is to map the postcode density information to the map. I picked zoom level 6 for this, and the result is (after cropping it to 640 pixels width):

As you can see, this is perfect fit to the outlines of the country. But unfortunately, when we look very closely at the plotted map data, for example for the NW10 3 postcodes, we notice that the mapping is slightly off. The blue dots are what we plotted, and the red dots are what the locations should have been:

The reason for this is that when we converted the National Grid locations to Latitude/Longitude pairs to plot on the OSM maps, I forgot to take into account the different Datums that are used in the projections. The Earth is not a perfect sphere, and an approximation of the ellipsoid of the whole Earth is not necessarily the best fitting for a specific area such as the UK. Therefore, the National Grid uses the OSGB36 Datum which fits more closely to the UK, where as OpenStreetMap uses the WGS84 Datum that is also used by GPS. The Ordnance Survey Ireland has a more thorough explanation on their site. As you can see above, using the wrong Datum can mean locations can be off. In our example about 100 meters. Converting between different Datums is possible, albeit processor intensive.

After I figured out all the maths for this, the only problem that remains that implementing those algorithms in PHP is show—calculating all the positions from the 1.6 million postcode locations takes up to 10 minutes. This is why I am not presenting any code yet. I am planning to implement all the necessary calculations in a PHP extension to speed up the calculations

VLD released

Mon, 12 Apr 2010 20:57:00 +0000

VLD released

London, UK

Monday, April 12th 2010, 21:57 BST

I just released a new version of VLD through PECL. This release features the path analytics code that I've outlined in an earlier article. There are also a few fixes for PHP's current trunk, as well as the possibility to format output slightly like a CSV file. That feature was used to make this documentation of opcodes.

The path analysis algorithm that VLD now has, can possibly also be used to get path/branch coverage working within Xdebug's code-coverage framework. This is however not a minor task.

Xdebug 2.1.0 RC1 released

Tue, 06 Apr 2010 13:41:00 +0000

Xdebug 2.1.0 RC1 released

Schiphol, Netherlands

Tuesday, April 6th 2010, 15:41 CEST

As earlier announced in my April Fools' announcement and on Xdebug's twitter feed, I'm now releasing Xdebug 2.1.0RC1. This is the first Release Candidate and it addresses a number of issues found in the latest beta. As I usually do in release announcements, I will also be introducing a new feature in Xdebug 2.1 in this one.

In an article that I wrote a few months ago, I illustrated a few different reasons on why the shut-op operator (@) should be avoided. Although there are a few cases where it simple can not be avoided if you want to write notice-free code, the main reason why this operator should be avoided, is because it makes debugging quite a bit more difficult if errors are hidden.

Xdebug is a debugging aid to make debugging easier. So from version 2.1.0 there is a new configuration setting — xdebug.scream — that simply disables the effects of the @ operator. That means that all errors will be shown, of course, still listening to the error_reporting and display_errors settings. This functionality is not a novel thing, and the idea was lifted from the PECL scream package. However, I think this functionality fits in very well with all the other things that Xdebug has to offer.

Just to mention it once more, I have no intention of making Xdebug closed source. I spend a considerable amount of time in adding new features and addressing issues, so I would appreciate it that if consider a donation in case Xdebug has saved you time and effort in debugging your application. If you have comments or suggestions to improve Xdebug, please write to the mailinglist.

First release of the D-Bus extension

Tue, 06 Apr 2010 09:33:00 +0000

First release of the D-Bus extension

London, UK

Tuesday, April 6th 2010, 10:33 BST

A few days ago I made the first beta release of the D-Bus extension that I have been working on for a while. D-Bus is a message bus system, a simple way for applications to talk to one another. I started working on this because my cellphone, an OpenMoko FreeRunner. This cellphone (or rather, a mobile Linux computing device) implements freesmartphone.org APIs to talk to all the hardware that is available on the device. This includes GSM and SIM card interfaces, but also GPS, Audio and PIM services.

However, many other applications on the Linux desktop speak D-Bus. This includes system services such as the notification daemon, the screen saver and hardware plug-in detection as well as desktop applications such as Pidgin and Empathy.

I've given a presentation on the extension at the PHP London conference earlier this year, and will also be speaking on this subject as part of the "PHP Inside" and "PHP on the D-Bus" talks at Tek-X (Chicago, May 18-21), International PHP Conference Spring (Berlin, May 30-June 2) and the Dutch PHP Conference (June 10-12).

Xdebug no longer Open Source

Thu, 01 Apr 2010 09:19:00 +0000

Xdebug no longer Open Source

London, UK

Thursday, April 1st 2010, 10:19 BST

This was obviously an April Fools' joke, Xdebug will continue to stay Open Source and free of charge.

Later today I will be releasing the first Release Candidate of Xdebug 2.1. With regret I have to announce that from this release, I will no longer provide Xdebug as an open source debugger extension. I've noticed that there are other projects making money of Xdebug's success while I only receive donations from a group of awesome supporters.

Xdebug will continue to be available for benefactors on a limited number of platforms that I have access to (Linux 32/64bit and Windows). Please refer to the FAQ for further information.

Storing Date/Times in Databases

Mon, 29 Mar 2010 15:47:00 +0000

Storing Date/Times in Databases

London, UK

Monday, March 29th 2010, 16:47 BST

After my talk during ConFoo on Advanced Date/Time Handling I received a question about whether the UTC-offset, together with the date/time in years, months, days, hours, minutes and seconds, was enough for storing a date/time in a database and still being able to do calculations with this. The answer to this question was no, but it lead to an even more interesting discussion about what would be enough to store an accurate date/time in a database.

Firstly let me explain why storing a UTC offset is not adequate at all for doing any sort of calculations. Right now (March 25th, 2010), Montreal is on the same time as Santiago (in Chile) with both a UTC offset of -4 hours. This means that for the same date/time (2010-03-25 19:03) the same timestamp is generated (1269558180). However if we want to add eight months (2010-11-25 19:03) to the current time for each of those locations then the timestamps should be different — 1290729780 for Montreal and 1290722580 for Santiago. It turns out that neither of them is exactly a full 24 hour difference from 2010-03-25 19:03 as is shown the following script:

format( 'U' );
$santiagoNow = date_create( "{$s} America/Santiago" )->format( 'U' );
$s = '2010-11-25 19:03';
$montrealThen = date_create( "{$s} America/Montreal" )->format( 'U' );
$santiagoThen = date_create( "{$s} America/Santiago" )->format( 'U' );

$hour = 60 * 60;
$day = 24 * $hour;

echo "Montreal:\n",
    $montrealNow, " - ",
    $montrealThen, "\n",
    ( ( $montrealThen - $montrealNow ) % $day ) / $hour, "\n\n";

echo "Santiago:\n",
    $santiagoNow, " - ",
    $santiagoThen, "\n",
    ( ( $santiagoThen - $santiagoNow ) % $day ) / $hour, "\n\n";

?>

which gives the output:

Montreal:
1269558180 - 1290729780
1

Santiago:
1269558180 - 1290722580
23

From this we see that Montreal has an extra hour and Santiago has an hour less. This is because Montreal changed from Daylight Savings Time to normal time and Santiago, being on the southern hemisphere, moved from normal time to Daylight Savings time.

Now if we had only the timestamp (1269558180) and the UTC-offset, the only thing we could do would be to advance a specific number of days. In this case, there are 245 days, which makes (245 * 24 * 60 * 60 = ) 21168000 seconds. If we add this to the original timestamp, we end up at 1290726180 which corresponds to 2010-11-25 18:03 Montreal time, or 2010-11-25 20:03 Santiago time. Neither of them being the correct 2010-11-25 19:03. Because from the UTC offset of 4 hours we don't know whether we're in Montreal or Santiago, we can conclude that with just this UTC offset and the original timestamp we can't calculate the timestamp of something that's 8 months in the future. Storing the UTC-offset can not change this fact either.

In order to to proper calculations, you need to keep information about the timezone itself. In PHP timezones are identified with identifiers such as America/Montreal. If we store those alongside the timestamps, we can do the proper calculations. The following example demonstrates that:

setTimeZone( new DateTimeZone( $tzid ) );
$d->modify( '+8 months' );

echo $d->format( 'Y-m-d H:i' ), "\n";
?>

which gives the output: 2010-11-25 19:03.

This seems to work, but unfortunately, even the approach of storing the timestamp and timezone identifier is not going to work correctly under certain circumstances. In order to find out why, we have to go back to January this year.

Imagine that in January this year—January 15th to be precise—we run the following script to determine the timestamp of a date/time two months in the future:

format( 'U' );

$d = new DateTime( "@$ts" );
$d->setTimeZone( new DateTimeZone( $tz ) );
$d->modify( '+2 months' );

echo $d->format( 'U Y-m-d H:i' ), "\n";
?>

This returns the timestamp 1268664060 for the date/time 2010-03-15 10:41:00.

Now skip forwards to the current date and time. If we used this calculated timestamp and the timezone identifier America/Santiago today to generate a date/time string, we would however get a different output. The following example shows this:

setTimeZone( new DateTimeZone( $tz ) );

echo $d->format( 'Y-m-d H:i:s' ), "\n";
?>

which gives the output: 2010-03-15 11:41:00 (and not 2010-03-15 10:41:00).

The difference in output is not a bug, but is caused because some countries change Daylight Savings Time rules quite frequently. In this case Chile decided (on March 4th) that instead of going forwards to DST at midnight March 14th, they will delay that to April 4th. When we ran the code on January 15th, the rules still thought that March 15th would already be on normal time again, outside of DST. But running the code now, with the updated rule set, we find that DST is still in effect until April 4th. The following example shows the transitions from/to DST for Santiago in 2010:

getTransitions(
    strtotime( '2010-01-01 00:00 UTC' ),
    strtotime( '2010-12-31 00:00 UTC' )
);

foreach ( $transitions as $t )
{
    echo $t['time'], ' ', $t['offset'] / 3600, ' ', $t['abbr'], "\n";
}
?>

If we run this script with PHP 5.3.1 or 5.3.2—which still have the old incorrect rule set—the output is:

2010-01-01T00:00:00+0000 -3 CLST
2010-03-14T03:00:00+0000 -4 CLT
2010-10-10T04:00:00+0000 -3 CLST

In the SVN repository, the rule set has been updated, so snapshots of PHP 5.3-dev have the correct rules and the script will show:

2010-01-01T00:00:00+0000 -3 CLST
2010-04-04T03:00:00+0000 -4 CLT
2010-10-10T04:00:00+0000 -3 CLST

This leads to the conclusion that storing timestamps and timezone identifiers is not good enough either, unless you want an exact point in time, as opposed to the more expected date/time in a location. So how should you store the latter then? Basically, in the same way that DateTime objects are serialized in PHP 5.3. Let us imagine again, that the following code is run on January 15th again:

modify( '+2 months' );
$s = $d->format( 'Y-m-d H:i:s' );
$ts = $d->format( 'U' );
echo "$s (ts=$ts)\n";
?>

which gives the output 2010-03-15 10:41:00 (ts=1268664060). The next example is run with a recent rule set (such as in PHP SVN) today:

format( 'Y-m-d H:i:s' );
$ts = $d->format( 'U' );
echo "$s (ts=$ts)\n";
?>

which gives the output 2010-03-15 10:41:00 (ts=1268660460). As you can see, now the date/time itself is correct, although there is a different timestamp. PHP's DateTime serialisation does something similar:

setTimeZone( new DateTimeZone( 'America/Santiago' ) );
var_dump( $d );
?>

which gives the output:

object(DateTime)#1 (3) {
  ["date"]=>
  string(19) "2010-03-29 11:32:08"
  ["timezone_type"]=>
  int(3)
  ["timezone"]=>
  string(16) "America/Santiago"
}

For things to work correctly, you need to have an up-to-date rule set. PHP versions are not released as often as the timezonedb (sometimes more than 20 times a year) and to address this issue you can install the pecl extension timezonedb with pecl install timezonedb.

To store information in a database, I would use a char column-type to store the whole America/Santiago timezone identifier and another char column-type to store the date/time (in the yyyy-mm-dd hh:ii:ss format). Alternatively, you can pick a 'datetime' column-type, as long as that type ignores timezones altogether. For MySQL that is the DATETIME column-type, and for PostgreSQL the TIMESTAMP or TIMESTAMP WITHOUT TIME ZONE column-types.

Databases rarely handle timezones, daylight savings time and rule changes correctly, so avoid the database specific functionality all together. Using either a char or a timezone-less 'datetime' column-type would still allow you to sort and by using a 'datetime' column-type you can even do calculations.

Happy summer time!

Available for PHP Extension Writing

Mon, 15 Mar 2010 12:28:00 +0000

Available for PHP Extension Writing

London, UK

Monday, March 15th 2010, 12:28 GMT

Slightly more than a month ago I left my job as team-lead of eZ Components at eZ Systems behind, to focus on something new. During the past month I've been contemplating about what to do next and realized that I do not want to take on a new full-time position right away. Instead I will be available to work on (custom) PHP extensions and internals related issues. Extensions are a great way around PHP's limitations and performance issues.

As first project I am working on a "QuickHash" extension for StumbleUpon. This extension circumvents PHP's hefty memory (and performance) overhead by providing more specific data structures. The extension currently implements integer sets and integer to integer hashes. I am now adding integer to string hashes and string to integer hashes. The QuickHash extension will be released under the PHP Licence and I will dedicate another post to it later.

If, like StumbleUpon, you are also interested in having work done on PHP or a specific extensions feel free to contact me. I'd be happy to discuss things with you.

Xdebug 2.1.0beta3 released

Sat, 27 Feb 2010 23:57:00 +0000

Xdebug 2.1.0beta3 released

London, UK

Saturday, February 27th 2010, 23:57 GMT

I've just released Xdebug 2.1.0beta3 which includes a few crash bugs as well as the issue that headers sent from PHP scripts are not actually set.

You can find the full changelog here and get the latest version from the download page.

More source analysis with VLD

Fri, 19 Feb 2010 11:27:00 +0000

More source analysis with VLD

London, UK

Friday, February 19th 2010, 11:27 GMT

VLD is a tool that I started working on years ago to visualise the opcode arrays in PHP. Opcode arrays are what PHP's compiler generates from your source code and can be compared to assembler code that is generated by a C compiler. Instead of it being directly executed by the CPU, it is instead executed by PHP's interpreter.

Over the years I've been adding some functionality, also aided by Ilia and some others, to show more information. For example Ilia has added a more verbose dumping format for opcodes (through the vld.verbosity setting) whereas I have added routines to find out which ops in oparrays can never be reached. A very simple example of the latter is shown here:

If we run the above through VLD with php -dvld.active=1 test.php, you'll see the following output (I removed the part about the script body itself):

Function test:
filename:       /tmp/test1.php
function name:  test
number of ops:  9
compiled vars:  none
line     # *  op           fetch  ext  return  operands
---------------------------------------------------------
   2     0  >   EXT_NOP
   4     1      EXT_STMT
         2      ECHO                           'Hello%21%0A'
   5     3      EXT_STMT
         4    > RETURN                         true
   7     5*     EXT_STMT
         6*     ECHO                           'This+will+not+be+executed.%0A'
   8     7*     EXT_STMT
         8*   > RETURN                         null

End of function test.

Every opcode that has a * after the number (like in 5*) is code that can not be reached, and can possibly be eliminated from the oparrays in an optimiser.

The dead code analysis routines have also made their way into Xdebug which uses them for the code coverage functionality to highlight dead code. This mostly makes sense if you are running your code coverage together with unit tests such as you can do with PHPUnit.

Recently I've been working on some new functionality to visualise all the code paths that make up each function. These new routines sit on top of the routines that do dead code analysis. Every branch instruction (such as if, but also for and foreach) is analysed and a list of branches is created. Each branch contains information about the line on which the branch starts, the starting and ending opcode numbers that belong to the branch, as well as to which other branches this branch can jump to. There can be either no linked branches (when for example a return or throw statement is found), one linked branch (for an unconditional jump) or two linked branches (on a branch instruction). However, you need to be aware that internally, PHP's opcode don't always reflect the source code exactly.

Once all the branches and their links are found, another algorithm runs to figure out which paths can be created out of all the branches. It is best to illustrate this with an example. So let us look at the following script:

In this script we have a for-loop with a nested if construct. When we run this script through VLD (with php -dvld.verbosity=0 -dvld.dump_paths=1 -dvld.active=1 test2.php) we get the following output (again, only the test() function and with some white space modifications):

Function test:
filename:       /tmp/test2.php
function name:  test
number of ops:  22
compiled vars:  !0 = $i
line     # *  op             fetch  ext  return  operands
-----------------------------------------------------------
   2     0  >   EXT_NOP
   4     1      EXT_STMT
         2      ASSIGN                             !0, 0
         3  >   IS_SMALLER                 ~1      !0, 10
         4      EXT_STMT
         5    > JMPZNZ                  9          ~1, ->18
         6  >   POST_INC                   ~2      !0
         7      FREE                               ~2
         8    > JMP                                ->3
   6     9  >   EXT_STMT
        10      IS_SMALLER                 ~3      !0, 5
   7    11    > JMPZ                               ~3, ->15
   8    12  >   EXT_STMT
        13      ECHO                               '-'
   9    14    > JMP                                ->17
  12    15  >   EXT_STMT
        16      ECHO                               '%2B'
  14    17  > > JMP                                ->6
  15    18  >   EXT_STMT
        19      ECHO                               '%0A'
  16    20      EXT_STMT
        21    > RETURN                             null

branch: #  0; line:  2- 4; sop:  0; eop:  2; out1:   3
branch: #  3; line:  4- 4; sop:  3; eop:  5; out1:  18; out2:   9
branch: #  6; line:  4- 4; sop:  6; eop:  8; out1:   3
branch: #  9; line:  6- 7; sop:  9; eop: 11; out1:  12; out2:  15
branch: # 12; line:  8- 9; sop: 12; eop: 14; out1:  17
branch: # 15; line: 12-14; sop: 15; eop: 16; out1:  17
branch: # 17; line: 14-14; sop: 17; eop: 17; out1:   6
branch: # 18; line: 15-16; sop: 18; eop: 21
path #1: 0, 3, 18,
path #2: 0, 3, 9, 12, 17, 6, 3, 18,
path #3: 0, 3, 9, 15, 17, 6, 3, 18,
End of function test.

This dump consists of a few different parts. First of all we can see some basic information containing the name, the number of ops (22) and the compiled variables. The second part is a dump of all the opcodes that make up this function. The last part contains information about all the branches and the possible paths. This information is a bit hard to visualize in its textual form, so I've also added some code that dumps this information to a file format that the GraphViz tool "dot" can use to create a pretty graph. For this we re-run the previous PHP invocation as php -dvld.dump_paths=1 -dvld.verbosity=0 -dvld.save_paths=1 -dvld.active=1 test2.php. This creates the file /tmp/paths.dot that "dot" can use. If we run dot -Tpng /tmp/paths.dot > /tmp/paths.png we end up with the following picture:

If we put this graph next to the code, we can explain how this works. Every branch is named by the number of the first opcode in that branch:

op #1 is the assignment of $i in line 4.
op #3 is the loop test in line 4. If the condition doesn't match, we jump to op #18 on line 16 that echos the newline.
op #9 is the if condition on line 6.
op #12 is when the if condition returns true and
op #15 is when the if condition returns false.
op #17 sits behind both op #12 and op #15 and makes sure there is a jump to the counting expression in #op 6.
op #6 is the post increment operation on line 4 which will then again be followed by op #3 to check whether the end of the loop has been reached.

This is of course a very simple example, but it also works for (multiple) classes and functions in a file. You just need to make sure to tell VLD that you don't want the code executed as the output could be very large. You can use the vld.execute=0 php.ini setting for that.

I hope this new functionality can spread some light on how loops etc. work in PHP. In order to play with the code, you need to check-out VLD from my SVN with svn co svn://svn.xdebug.org/svn/php/vld/trunk vld. You can also view the code on-line at http://svn.xdebug.org/cgi-bin/viewvc.cgi/vld/trunk/?root=php. Look out for a new release coming soon!