Collecting Garbage: PHP's take on variables

This is the first part of three-parts column that was originally published in the April 2009 issues of php|architect.

In this three part column I will explain the merits of the new Garbage Collection (also known as GC) mechanism that is part of PHP 5.3. Before we start with the intricate details of PHP's new GC engine I will explain why it is actually needed. This, combined with an introduction how PHP deals with variables in general is explained in this first part of the column. The second part will cover the solution and some notes on the GC mechanism itself, and the third part covers some implications of the GC mechanism, as well as some benchmarks. But now first on to the introduction.

PHP stores variables in containers called a "zval". A zval container contains besides the variable's type and value, also two additional bits of information. The first one is called "is_ref" and contains a boolean value whether this variable is part of a "reference set". With this bit PHP's engine knows how to differentiate between normal variables, and references. However, PHP has user-land references—as created by the & operator, but also an internal reference counting mechanism to optimize memory usage. The second piece of additional information, called "refcount", contains how many variables names—also called symbols—point to this one zval container. All symbols are stored in a symbol table, of which there is one per scope. There is a scope for the main script (ie, the one requested through the browser), as well as for every function or method.

A zval container is created when a new variable is created with a constant value, such as:

$a = "new string";

In this case the new symbol name "a" is created in the current scope, and a new variable container is created with type "string", value "new string". The "is_ref" bit is by default set to "false" because no user-land reference has been created. The "refcount" is set to "1" as there is only one symbol that makes use if this variable container. Also, if the "refcount" is "1", "is_ref" is always "false". If you have Xdebug installed you can display this information by calling:

xdebug_debug_zval('a');

which displays:

a: (refcount=1, is_ref=0)='new string'

Assigning this variable to another variable name, increases the refcount:

$a = "new string";
$b = $a;
xdebug_debug_zval( 'a' );

which displays:

a: (refcount=2, is_ref=0)='new string'

The refcount is "2" here, because the same variable container is linked with both "a" and "b". PHP is smart enough not to copy the actual variable container when it is not necessary. Variable containers get destroyed when the "refcount" reaches zero. The "refcount" gets decreased by one for each symbol linked to the variable container leaves the scope (f.e. if the function ends) or when unset() is called on a symbol. The following example shows that:

$a = "new string";
$c = $b = $a;
xdebug_debug_zval( 'a' );
unset( $b, $c );
xdebug_debug_zval( 'a' );

which displays:

a: (refcount=3, is_ref=0)='new string'
a: (refcount=1, is_ref=0)='new string'

If we now call "unset( $a );" the variable container, including the type and value will be removed from memory.

Things get a tad more complex with compound types such as arrays and objects. Instead of a scalar value, arrays and objects store their properties in a symbol table of their own. This means that the following example creates three zval containers:

$a = array( 'meaning' => 'life', 'number' => 42 );
xdebug_debug_zval( 'a' );

which displays (after formatting):

a: (refcount=1, is_ref=0)=array (
        'meaning' => (refcount=1, is_ref=0)='life',
        'number' => (refcount=1, is_ref=0)=42
)

Graphically, it looks like:

You can see the three zval containers here: "a", "meaning" and "number". Similar rules apply for increasing and decreasing "refcounts". Below we add another element to the array, and set it's value to the contains of an already existing element:

$a = array( 'meaning' => 'life', 'number' => 42 );
$a['life'] = $a['meaning'];
xdebug_debug_zval( 'a' );

which displays (after formatting):

a: (refcount=1, is_ref=0)=array (
        'meaning' => (refcount=2, is_ref=0)='life',
        'number' => (refcount=1, is_ref=0)=42,
        'life' => (refcount=2, is_ref=0)='life'
)

Graphically, it looks like:

From the above we see that both the old and new array element now point to a zval container which "refcount" is "2". Of course there are now two zval containers, but they are the same one. The xdebug_debug_zval() function does not show this, but could, by also displaying the memory pointer. Removing an element from the array is like removing a symbol from a scope. By doing so, the "refcount" of a container that an array element points to is decreased. Again when the "refcount" reaches zero, the variable container is removed from memory. Again an example to show this:

$a = array( 'meaning' => 'life', 'number' => 42 );
$a['life'] = $a['meaning'];
unset( $a['meaning'], $a['number'] );
xdebug_debug_zval( 'a' );

which displays (after formatting):

a: (refcount=1, is_ref=0)=array (
        'life' => (refcount=1, is_ref=0)='life'
)

Now, things get interesting if we add the array itself as an element of the array, which we do in the next example—in which I also sneaked in an reference operator as otherwise PHP would create a copy here:

$a = array( 'one' );
$a[] =& $a;
xdebug_debug_zval( 'a' );

which displays (after formatting):

a: (refcount=2, is_ref=1)=array (
        0 => (refcount=1, is_ref=0)='one',
        1 => (refcount=2, is_ref=1)=...
)

Graphically, it looks like:

You can see that the array variable ("a") as well as the second element ("1") now point to a variable container that has a "refcount" of "2". The "..." in the display above shows that there is recursion involved, which of course in this case it means that the "..." points back to the original array.

Just like before, unsetting a variable removes the symbol, and the reference count of the variable container it points to is decreased by one. So if we unset variable $a after running the above code, the reference count of the variable container that $a and element "1" point to gets decreased by one, from "2" to "1". This can be represented like:

(refcount=1, is_ref=1)=array (
        0 => (refcount=1, is_ref=0)='one',
        1 => (refcount=1, is_ref=1)=...
)

Graphically, it looks like:

Although there is no symbol in any scope pointing to this structure anymore, it can not be cleaned up either because the array element "1" still points to this same array. Because there is no external symbol pointing to it, there is no way for a user to clean up this structure anymore, and thus you get a memory leak. Fortunately, PHP will clean up this data structure at the end of the request, but before then this is taking up valuable space in memory. The mentioned situation happens often if you're implementing parsing algorithms or other things where you have a child point back at a "parent" element. The same situation can also happen with objects of course, where it actually happens easier as objects are always implicitly used by reference.

This might not be a problem if this only happens once or twice, but if there is thousands, or even millions of these memory losses, this obviously starts being a problem. Especially in long running scripts, such as daemons where the request basically never ends, or in large sets of unit tests. The latter caused problems for us while running the unit tests for the Template component of the eZ Components library. In some cases it would require over 2 GiB of memory, which our test server didn't quite have.

With that we conclude this introduction, for more information on how PHP deals with variables I can point you at June 2005 issue of php|architect. That article is also available on-line as PDF (http://derickrethans.nl/files/phparch-php-variables-article.pdf). In the next installment, we're going to discuss the solution to the memory leak problem with circular references.

64-bit integers in MongoDB

The current project that I'm working on relies heavily on MongoDB, a bridge between key-value stores and traditional RDBMS systems. Users in this project are identified by their Facebook UserID, which is a "64-bit int datatype". Unfortunately, the MongoDB PHP Driver only had support for 32-bit integers causing problems for newer users of Facebook. For those users, their nice long UserID was truncated to only 32 bits which didn't quite make the application work.

MongoDB stores documents internally in something called BSON (Binary JSON). BSON has two integer types, a 32-bit signed integer type called INT and a 64-bit signed integer type called LONG. The documentation of the MongoDB PHP driver on types says (or used to say, depending on when you're reading this) that only the 32-bit signed integer type is supported because "PHP does not support 8 byte integers". That's not quite true. PHP's integer type supports 64-bit on the platforms where the C-data type long is 64 bits. That is generally on every 64-bit platform (where PHP is compiled for 64 bits); except on Windows, where the C-data type long is always only 32 bits.

Whenever a PHP integer is sent to MongoDB, the driver would use the 32 least significant bits to store the number as part of the document. The example here shows what happens (on a 64-bit platform):

<?php
$m = new Mongo();
$c = $m->selectCollection('test', 'inttest');
$c->remove(array());

$c->insert(array('number' => 1234567890123456));

$r = $c->findOne();
echo $r['number'], "\n";
?>

shows:

int(1015724736)

In binary:

1234567890123456 = 100011000101101010100111100100010101011101011000000
      1015724736 =                      111100100010101011101011000000

Truncating data is obviously not a very good idea. In order to address this issue we could just allow for the native PHP integer type to be used when storing data from PHP into MongoDB. But instead of changing how the MongoDB driver works by default I've added the new setting mongo.native_long — simply because otherwise we might be breaking applications. With the mongo.native_long setting enabled, we see the following result instead of the outcome of the script above:

<?php
ini_set('mongo.native_long', 1);
$c->insert(array('number' => 1234567890123456));

$r = $c->findOne();
var_dump($r['number']);
?>

This script shows:

int(1234567890123456)

On 64-bit platforms, the mongo.native_long setting allows for 64-bit integers to be stored in MongoDB. The MongoDB data type that is used in this case is the BSON LONG, instead of the BSON INT that is used if this setting is turned off. The setting also changes the way how BSON LONGs behave when they are read back from MongoDB. Without mongo.native_long enabled, the driver would convert every BSON LONG to a PHP double which results in the loss of precision. You can see that in the following example:

<?php
ini_set('mongo.native_long', 1);
$c->insert(array('number' => 12345678901234567));

ini_set('mongo.native_long', 0);
$r = $c->findOne();
var_dump($r['number']);
?>

This script shows:

float(1.2345678901235E+16)

On 32-bit platforms, the mongo.native_log setting changes nothing for storing integers in MongoDB: the integer is stored as a BSON INT as before. However, when the setting is enabled and a BSON LONG is read from MongoDB a MongoCursorException is thrown alerting you that the data could not be read back without losing precision:

MongoCursorException: Can not natively represent the long 1234567890123456 on this platform

If the setting is not enabled, a BSON LONG is converted to a PHP float in order to avoid breaking backwards compatibility with the current behaviour.

Although the mongo.native_long settings allows for 64-bit support on 64-bit platforms, it doesn't provide much for 32-bit platforms except preventing loss of precision while reading BSON LONGs—and then by just throwing an exception.

As part of making 64-bit integers work reliably with MongoDB, I've also added two new classes: MongoInt32 and MongoInt64. These two classes are simple wrappers around a string representation of a number. They are instantiated like this:

<?php
$int32 = new MongoInt32("32091231");
$int64 = new MongoInt64("1234567980123456");
?>

You can use those objects in normal insert and update queries just like normal numbers:

<?php
$m = new Mongo();
$c = $m->selectCollection('test', 'inttest');
$c->remove(array());

$c->insert(array(
        'int32' => new MongoInt32("1234567890"),
        'int64' => new MongoInt64("12345678901234567"),
));

$r = $c->findOne();
var_dump($r['int32']);
var_dump($r['int64']);
?>

Which shows:

int(1234567890)
float(1.2345678901235E+16)

As you can see, nothing is changed with how values are returned. A BSON INT is still returned as an integer, and a BSON LONG as a double. If we turn on the mongo.native_long setting then the BSON LONG that was stored through the MongoInt64 class is returned as a PHP integer on 64-bit platforms, and a MongoCursorException is thrown on 32-bit platforms.

In order to received 64-bit integers back from MongoDB on 32-bit platforms, I've added another setting: mongo.long_as_object. This will (on any platform) return a BSON LONG as stored in MongoDB as a MongoInt64 object. The following script demonstrates that:

<?php
$m = new Mongo();
$c = $m->selectCollection('test', 'inttest');
$c->remove(array());

$c->insert(array(
        'int64' => new MongoInt64("12345678901234567"),
));

ini_set('mongo.long_as_object', 1);
$r = $c->findOne();
var_dump($r['int64']);
echo $r['int64'], "\n";
echo $r['int64']->value, "\n";
?>

This script outputs:

object(MongoInt64)#7 (1) {
  ["value"]=>
  string(17) "12345678901234567"
}
12345678901234567
12345678901234567

The MongoInt32 and MongoInt64 classes implement __toString() so that their values can be echoed. You can only get their values out as strings. Please be aware that MongoDB is type-sensitive, and will not treat a number contained in a string the same way as a number that's just a number. This script shows this (on a 64-bit platform):

<?php
ini_set('mongo.native_long', 1);

$m = new Mongo();
$c = $m->selectCollection('test', 'inttest');
$c->remove(array());

$nr = "12345678901234567";
$c->insert(array('int64' => new MongoInt64($nr)));

$r = $c->findOne(array('int64' => $nr)); // $nr is a string here
var_dump($r['int64']);
$r = $c->findOne(array('int64' => (int) $nr));
var_dump($r['int64']);
?>

Which shows:

NULL
int(12345678901234567)

The following tables summarises all the different conversions regarding integers, depending on which settings are enabled:

PHP to MongoDB on 32-bit systems:

PHP to MongoDB on 64-bit systems:

Mongo to PHP on 32-bit systems:

Mongo to PHP on 64-bit systems:

Conclusion

Getting 64-bit support right with MongoDB can be tricky as we've seen. My recommendations would be to use just mongo.native_long=1 if you only deal with 64-bit platforms for your code. In this case, every integer number that you put into MongoDB will also come out as an integer number; with 64-bit integers supported.

If you however have to deal with 32-bit platforms (remember, that includes a 64-bit Windows build of PHP) then you can not reliably use just PHP's integer types and you have to use the MongoInt64 class. This comes with the restriction that you have to deal with numbers in strings for initialisation. You also need to be aware that the MongoDB shell regards all numbers as floating point numbers, and that it can not represent 64-bit integer values. Instead they will show up as floating point numbers. Do not attempt to modify those numbers on the shell, as that could change the type.

So with the script:

<?php
$m = new Mongo();
$c = $m->selectCollection('test', 'inttest');
$c->remove(array());

$c->insert(array('int64' => new MongoInt64("123456789012345678")));

The MongoDB shell mongo behaves like:

$ mongo
MongoDB shell version: 1.4.4
url: test
connecting to: test
type "help" for help
> use test
switched to db test
> db.inttest.find()
{ "_id" : ObjectId("4c5ea6d59a14ce1319000000"), "int64" : { "floatApprox" : 123456789012345680, "top" : 28744523, "bottom" : 2788225870 } }

Of course, when fetching data through a driver that supports 64-bit integers you get the proper result:

ini_set('mongo.long_as_object', 1);
$r = $c->findOne();
var_dump($r['int64']);
?>

Which shows:

object(MongoInt64)#7 (1) {
  ["value"]=>
  string(18) "123456789012345678"
}

The new functionality as outlined in this article is part of the 1.0.9 mongo release that's available through PECL with pecl install mongo. Good luck with your 64-bit integers!

VMWare on Debian Unstable

In the past week I've been doing some work on the MongoDB PHP Driver, and while running the test cases I noticed that there are some issues with it when you run out of disk space. It's sadly hard to reproduce so I set out to create a VMWare image for the MongoDB developers to reproduce this issue.

cd /tmp
wget http://derickrethans.nl/files/vmware-server.2.0.1_x64-modules-2.6.30.4-fix.patch
wget http://derickrethans.nl/files/vmware-7.0-2.6.32.patch
mkdir tmp-vmware-modules
cd tmp-vmware-modules
cp /usr/local/lib/vmware/modules/source/*.tar .
for i in *tar; do tar xvf $i; done
for i in *tar; do mv $i old-$i; done
patch -p0 < /tmp/vmware-server.2.0.1_x64-modules-2.6.30.4-fix.patch
patch -p1 < /tmp/vmware-7.0-2.6.32.patch
for i in *only; do j=`echo $i | sed 's/-only//'`; tar cvf $j.tar $i; done
cp *.tar /usr/local/lib/vmware/modules/source
vmware-config.pl

webAccessOpts="-client -Xmx64m -XX:MinHeapFreeRatio=30 -XX:MaxHeapFreeRatio=30 -Djava.net.preferIPv4Stack=true -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=$CATALINA_HOME/common/endorsed -classpath $CATALINA_HOME/bin/bootstrap.jar:$CATALINA_HOME/bin/commons-logging-api.jar -Dcatalina.base=$CATALINA_HOME -Dcatalina.home=$CATALINA_HOME -Djava.io.tmpdir=$CATALINA_HOME/temp org.apache.catalina.startup.Bootstrap"

xkeymap.keycode.108 = 0x138 # Alt_R
xkeymap.keycode.106 = 0x135 # KP_Divide
xkeymap.keycode.104 = 0x11c # KP_Enter
xkeymap.keycode.111 = 0x148 # Up
xkeymap.keycode.116 = 0x150 # Down
xkeymap.keycode.113 = 0x14b # Left
xkeymap.keycode.114 = 0x14d # Right
xkeymap.keycode.105 = 0x11d # Control_R
xkeymap.keycode.118 = 0x152 # Insert
xkeymap.keycode.119 = 0x153 # Delete
xkeymap.keycode.110 = 0x147 # Home
xkeymap.keycode.115 = 0x14f # End
xkeymap.keycode.112 = 0x149 # Prior
xkeymap.keycode.117 = 0x151 # Next
xkeymap.keycode.78 = 0x46 # Scroll_Lock
xkeymap.keycode.127 = 0x100 # Pause
xkeymap.keycode.133 = 0x15b # Meta_L
xkeymap.keycode.134 = 0x15c # Meta_R
xkeymap.keycode.135 = 0x15d # Menu
xkeymap.keycode.107 = 0x137 # Print Scrn

Snowdon Hike

This weekend I escaped London to enjoy two of my other hobbies: hiking and photography. This time I headed to Wales to climb up the 1085 meter high mountain Snowdon; Wales' highest peak. As opposed to staying in Llanberis, the town closest to many of the routes up Snowdon, I stayed in the little quiet town Caernarfon. Or rather, in the B&B Tal Menai not too far away from it. If you get there by bus though, make sure you don't get off on the one that's just a stop too far, because there is no way you can walk along the road. We ended up backtracking over a cycle path.

Caernarfon has a pretty nice castle and also a bit of town wall still. It's otherwise not very interesting but they do have a non-floating floating restaurant. As it's on the Menai Strait there are some pretty sunsets however.

There are several tracks up to Snowdon, also from different starting points. Our starting point was the Pen-y-Pass parking place at 359 meters above sea-leavel. From here on we went up the Pyg track to the summit. Unfortunately, the Welsh weather was doing it's normal thing: clouds. Which means we saw nothing of the view once we got to the summit at 1085 meters high. Basically, the view was the one that you can see here on the left.

After tea and some snacks, we headed back down. And as soon as we did the clouds cleared up showing us the view that was denied us on the way upwards. We took the Miners track back to the parking place to take the bus back to Caernarfon for a very welcomed proper pub meal.

I think I would like to go back once more, to try some of the other tracks. The Crib Goch one is supposed to be a really nice ridge walk; unfortunately it wasn't marked on the rendered OpenStreetmap at that point (it is now of course).

To see all the pictures, please visit my flickr set "Snowdon".

Xdebug 2.1

I've just released Xdebug 2.1. This new release contains all the bug fixes and new features that have been developed over the past three years. It provides full PHP 5.3 support, but it no longer supports PHP versions below 5.1.

A description of all the new features follows:

Error Collection

I've added xdebug_start_error_collection(), xdebug_stop_error_collection() and xdebug_get_collected_errors(), which allow you to collect all notices, warnings and error messages that Xdebug generates from PHP's error_reporting functionality so that you can output them at a later point in your script by hand.

Header Setting Interception

All functions that set HTTP headers such as with header() and setcookie() are now intercepted by Xdebug. The intercepted headers are stored internally in an array that can be retrieved by calling the xdebug_get_headers() function. This is very useful in cases where you need to test certain functionality that sets headers somewhere deep in code. This function is also used in eZ Components' test suite to test whether the correct HTTP headers are set in the MvcTools component.

Variable Assignment Tracing

The new setting xdebug.collect_assignments allows you to record changes to variables in scripts to trace files. I've already written more about it in Variable tracing with Xdebug.

"Scream" Support

The scream PECL extension disables the @ (shut-up) operator to actually see all notices, warnings and errors that PHP generates. The scream extension's functionality have been duplicated as Xdebug's xdebug.scream php.ini setting. Why disabling the @-operator is a good thing, I've already outlined in Five reasons why the shut-op operator (@) should be avoided.

Additions for Stack Traces

All HTML containers generated by Xdebug (error messages, xdebug_var_dump() output) now contain a CSS class for easier styling.

The new setting xdebug.file_link_format allows you to turn any file/line link in Xdebug's output to be decorated to a link with a specific format. For example you can set it to txmt://open/?file://%f&line=%l for opening files directly in Textmate, or gvim://%f@%l with some shell script for gvim and Firefox.

Remote Debugging

The new setting xdebug.remote_connect_back, contributed by Lucas Nealan and Brian Shire, allows Xdebug to try to make a debugging connection to the IP address from which the browser request came from. This setting is an additional solution for debugging with multiple users.

The old gdb and php3 remote debugging engines have been removed.

Overloaded var_dump

The newly introduced setting xdebug.overload_var_dump can be used to turn off Xdebug's default behavior of overriding PHP's var_dump() function with the xdebug_var_dump() function that uses pretty HTML for formatting a variable structure. It does not stop xdebug_var_dump() from working however.

Donations

You can probably imagine that writing and support Xdebug is a time-consuming effort. It's a tool that saves a lot of work during developement of applications, as well while debugging. If you think Xdebug is valuable for you, perhaps you would like to consider a donation. All donations are greatly appreciated and support future development and availablity of Xdebug.

If you want to sponsor a (new) feature in Xdebug, feel free to contact me as well.

Download and Support

The new version of Xdebug can be downloaded through the download page. A list with all changes can be found on the updates page. Support information can be found on the support page, and issues and feature requests can be filed in the bug tracker. Before asking for support, please have a look at the FAQ first.