Adressing Flats

I recently moved into a new area in London, and that means, a new area to map. That means drawing in all the buildings, but also adding shops and addresses. Unlike my old area, the new neighbourhood has lots of buildings with flats. In OpenStreetMap, you can tag addresses on flats with the addr:flats tag. In most cases, the addresses of the flats are formed by a flat number, building name (addr:housename) and street name (addr:street). Only occasionally the address consists of flat number, house number, and street name.

When I started tagging a nearby collection of flat blocks, and the flat number/house name/street name on their entrance nodes, I noticed that it rendered as:

As you can see, it ignored the addr:flats tag, and instead just rendered the addr:housename tag over and over again. Not particularly useful.

So I thought, why not "fix" the rendering. In the past, I already added a basic rendering for natural=tree_row to the "default" rendering style. So I had another look on how to make addr:flats render and show up instead of the house name 20 times.

OpenStreetMap's style sheet is developed through the Git repository at git@github.com:gravitystorm/openstreetmap-carto.git. I cloned the repository and searched after the places where I needed to make modifications to add the flat numbers. The style sheets first require a "style" file to tell which data to import from the main database (or .xml file). Only fields that are copied into the rendering database are available for use in the style sheets. OpenStreetMap Carto stores these definitions in the project.mml file. But that's a nasty JSON file, so instead, you add the definition into the project.yaml file. There is a script to convert the YAML file into the project.mml file which the carto tool can use to generate a Mapnik stylesheet. Mapnik is then responsible for rendering the map.

In project.yaml, just above the - id: "housenumbers" line, I added the following section:

- id: "flatnumbers"
  name: "flatnumbers"
  class: ""
  geometry: "point"
  <<: *extents
  Datasource:
    <<: *osm2pgsql
    table: |-
      (SELECT
          way,
          "addr:flats",
          way_area/NULLIF(!pixel_width!::real*!pixel_height!::real,0) AS way_pixels
        FROM planet_osm_polygon
        WHERE "addr:flats" IS NOT NULL
          AND building IS NOT NULL
      UNION ALL
      SELECT
          way,
          "addr:flats",
          NULL AS way_pixels
        FROM planet_osm_point
        WHERE "addr:flats" IS NOT NULL
        ORDER BY way_pixels DESC NULLS LAST
      ) AS flatnumbers
  properties:
    minzoom: 17
  advanced: {}

It basically instructs the stylesheet to run an extra query to obtain information on addr:flats and make that information available to the actual styles.

It requires an extra column in the database (addr:flats), the import "style" needs to include this data while importing from the data source into the rendering database. The file containing the import definitions is openstreetmap-carto.style. I added the following line in between the definitions for access and addr:housenumber:

node,way   addr:flats   text         linear

Adding an extra column into the "style" means that it is necessary to reimport the data from the main database (or XML file) into the rendering database. My original source is the openstreetmap.osm file, which I re-imported with:

/home/derick/install/osm2pgsql/osm2pgsql \
        -S openstreetmap-carto.style --slim -d gis -C 2400 openstreetmap.osm

To render the flat numbers, and attach a style to them, I had to add a section to the addressing.mss file which is a sub-stylesheet that deals with address information. This stylesheet is linked from the project file. I added the following lines in between the #interpolation and #housenumbers sections:

#flatnumbers {
  [zoom >= 17] {
        text-name: "[addr:flats]";
        text-placement: interior;
        text-min-distance: 1;
        text-wrap-width: 0;
        text-face-name: @book-fonts;
        text-fill: #666;
        text-size: 8;
  }
}

I picked the same style as for standard house numbers, but one size smaller.

With all the sections added to the right files I had to regenerate the project.mml file from project.yaml with:

./scripts/yaml2mml.py < project.yaml > project.mml

And generate a Mapnik stylesheet from the project.mml file with:

carto project.mml > mapnik.osm

With all that in place, I re-rendered the area. The flat numbers are now visible on the map:

I have packaged up all the changes, and prepared a pull request for openstreetmap-carto. With some luck, the changes will make it to the OpenStreetMap website at some point in the future.

Shortlink

This article has a short URL available: http://drck.me/flats-bt2

Comments

Nice bit of work. In reading it I've noticed that 'reimporting the database' is often mention in the context of adding a column. I dont think this is necessary:

ALTER TABLE planet_x ADD "addr:flats" VARCHAR(256) NULL;

assuming data has been imported with hstore;

UPDATE planet_x
SET "addr_flats" = tags->'addr:flats'
WHERE tags?'addr:flats'

This costs roughly a table scan per table.

Obviously you cant do anything if the tags column wasn't imported.

These ought to be capable of being wrapped up as a reversible database update (i.e., ALTER TABLE planet_x DROP "addr:flats" takes us back to where we started), which is useful if doing stuff on the live system.

Twenty Years of PHP

I know I am a day late, but PHP turned 20 yesterday! Congratulations!

Ben Ramsey wrote about his 20 years of PHP yesterday, and invited others to do the same. So here I go.

Back in 1999 during university, a few friends and I got annoyed by Schumacher's antics, and started the "Anti Schumacher Club". In order to have a membership part of our site (which thankfully can no longer be found on the Internet), we used PHP. Oh yeah, we also used it to scrape websites to find e-mail addresses to send our propaganda to.

My first patch to PHP was in June 2000 when I added support for Flash objects to getimagesize(). After an idiotic idea to implement MySQL subselects in PHP itself, I continued with my PHP career by starting to maintain the mcrypt extension later in 2000. It needed support for a new API version. Both getimagesize() and mcrypt support is something that I needed while working at the company that I started with the same guys as the Anti Schumacher Club. Throughout 2000 and later, I started to contribute little bits and bobs to PHP. These included committing Ilia's first extension, adding the 'u' modifier to printf/sprintf which prints unsigned longs, etc.

In April 2002, I started working on Xdebug. First by adding the max_nesting_level feature and functions to provide memory usage, and then later I added stack traces and support for the remote debugging. First by mimicking the PHP 3 handler, and then later by adding a GDB like debug handler, and DBGp. The PHP 3 and GDB handlers are of course a distant memory, but the DBGp protocol is still being used by IDEs to interact with Xdebug. Ilia implemented a basic profiler as well during 2002. I also "release mastered" PHP 4.1.1, various 4.2 releases during this time, and all the 4.4.x releases later.

The DBGp protocol is something that I developed with the guys from ActiveState in late 2003. Their Komodo editor still implements DBGp in the most comprehensive way, although other IDEs seem to be much more popular now. I implemented the same protocol for Maguma Workbench, and during my time there, eZ Systems approached me to come and work for them. I had previously spoken at an eZ summer camp, which provided a nice introduction to start a 5+ year employment relationship.

I started speaking a while earlier though, mostly on Xdebug, but also on SRM — some sort of application server for PHP. That never really got off the ground though, as it doesn't really fit the shared-nothing architecture of PHP. Since these first few speaking engagements, I have now given 326 talks at 201 conferences and user group meet-ups in 92 different locations.

After eZ Systems, and due to my work on Xdebug I decided to work for myself for a while. I started writing PHP extensions as a job, and due to that, I ended up contributing a little to the MongoDB driver. And that turned later into becoming an employee of MongoDB, where I now work on both the PHP and HHVM drivers.

Without PHP, none of this would have happened. I would not have moved to London without the persuasiveness of Johanna, whom I met at php[tek] in Chicago. She managed to convince me to give up the boringness of Norway and move into the "PHP House" in London. With her, Helgi, and Arpad (and on the weekends Scott), we wrote a lot of code, and threw a few parties as well. I wouldn't have met my wonderful wife Morag, met countless friends at various conferences and online, and I would certainly not have had a few dozen colourful pachyderms in my house! And it all started with Michael Schumacher, and some Danish guy that wrote some counters for his website.

Here's to another twenty years! Skål!

Shortlink

This article has a short URL available: http://drck.me/20yrs-brd

Comments

No comments yet

Xdebug 2.3: Profiler File Compression

This is the seventh (and last) article in a series about new features in Xdebug 2.3, which was first released on February 22nd.

When you make profiling dumps with Xdebug, the file size can grow quite large. Even with a simple Drupal page a profile file is easily close to 1Mb.

For each function call, the file contains the location and name of the calling function, and then a list of functions that have been called. For example, the following snippet shows that the _cache_get_object function in /home/httpd/drupal-test/drupal-7.15/includes/cache.inc calls variable_get in /home/httpd/drupal-test/drupal-7.15/includes/bootstrap.inc twice, and DrupalDatabaseCache->__construct once:

fl=/home/httpd/drupal-test/drupal-7.15/includes/cache.inc
fn=_cache_get_object
22 68
cfl=/home/httpd/drupal-test/drupal-7.15/includes/bootstrap.inc
cfn=variable_get
calls=1 0 0
27 4
cfl=/home/httpd/drupal-test/drupal-7.15/includes/bootstrap.inc
cfn=variable_get
calls=1 0 0
29 2
cfl=/home/httpd/drupal-test/drupal-7.15/includes/cache.inc
cfn=DrupalDatabaseCache->__construct
calls=1 0 0
31 5

In those 15 lines, you can already spot a lot of duplicated information. The Callgrind Profile Format which Xdebug uses for writing profiling information allows for Name Compression. This allows a write, such as Xdebug, to simple refer to longer strings by a number. This number is assigned the first time a certain file name or function name is encountered. The above snippet, from the same request of a Drupal site, with Name Compression enabled now looks like:

fl=(4)
fn=(53) _cache_get_object
22 80
cfl=(2)
cfn=(45)
calls=1 0 0
27 4
cfl=(2)
cfn=(45)
calls=1 0 0
29 3
cfl=(4)
cfn=(52)
calls=1 0 0
31 5

Every file name and function name, except for _cache_get_object has been previously defined, and hence, is not repeated in its entirety, but just by its identifier. This sort of compression makes the profiling files a lot smaller. In this case, my simple-Drupal-profiling file went down from 926Kb to 385Kb.


Other parts in this series:

Shortlink

This article has a short URL available: http://drck.me/profilecomp23-bp6

Comments

No comments yet

Xdebug 2.3: Shared Secret to Enable Tracing or Profiling

This is the sixth article in a series about new features in Xdebug 2.3, which was first released on February 22nd.

Xdebug's profiling and trace file capabilities can both be triggered by a cookie, GET or POST variable, as long as you have enabled xdebug.profiler_enable_trigger and/or xdebug.trace_enable_trigger. With these triggers enabled, anybody could initiate a profile run, or trace file, by simply sending the XDEBUG_PROFILE or XDEBUG_TRACE cookies with an HTTP request.

Although you should not really run Xdebug in production, you can see that this is not an optimal solution.

Xdebug 2.3 adds supports for shared secrets for the trace file and profiler triggers through the xdebug.trace_enable_trigger_value and xdebug.profiler_enable_trigger_value. If these settings are changed from their default (empty string), then the value of XDEBUG_PROFILE needs to match the value of xdebug.profiler_enable_trigger_value, and the value of XDEBUG_TRACE needs to match the value of xdebug.trace_enable_trigger_value in order for the profiling to start, or the trace file to be generated.

Often users would use one of the browser extensions for triggering profile runs or enabling tracing, these extensions need to be updated. The author of The easiest Xdebug, Nikita Nikitin, managed to get an updated version out before I could complete this article. It now has support for supplying your own values for XDEBUG_TRACE and XDEBUG_PROFILE:

The other two browser helpers have not been updated yet. I have emailed the author of Chrome's Xdebug helper, and I have filled an issue for Safari's xdebug-helper-for-safari on Github. Let's hope they get updated soon too.


Other parts in this series:

Shortlink

This article has a short URL available: http://drck.me/sharedsecret23-bow

Comments

No comments yet

Xdebug 2.3: Improvements to Tracing

This is the fifth article in a series about new features in Xdebug 2.3, which was first released on February 22nd.

In this instalment we are going to have a look at the additions to the trace file functionality. Trace files are a way to document every function call, and if you enable it, variable assignment and function's return values — including when these functions were called, and how much memory PHP was using at the moment of function entry (and exit).

Xdebug 2.3 adds a new type of parameter rendering for stack traces and function traces through the xdebug.collect_params setting. Existing options are 1/2 for just the variable type, 3 for a string description, and 4 for a string description and a variable name. Xdebug 2.3 now also features a base64 encoded serialized representation of the variable's contents (option 5). Which means that the following lines:

$a = array(1, "foo", M_PI);
var_dump($a);

show up in the following 5 ways with different variations of the xdebug.collect_params setting:

  1. var_dump(array(3))

  2. var_dump(array(3))

  3. var_dump(array (0 => 1, 1 => 'foo', 2 => 3.1415926535898))

  4. var_dump(array (0 => 1, 1 => 'foo', 2 => 3.1415926535898))

  5. var_dump(YTozOntpOjA7aToxO2k6MTtzOjM6ImZvbyI7aToyO2Q6My4xNDE1OTI2NTM1ODk3OTMxO30=)

This is probably more useful with the computerized trace files that are easier to parse.


In Xdebug 2.3, normal (human readable) trace files now also show the time index and memory usage for function exit lines. Function exit lines are generated when xdebug.collect_return is set to 1. Which means that with with Xdebug 2.2 you would see:

TRACE START [2015-03-28 18:48:39]
  0.0008     275928   -> test1() …/closure-trace.phpt:27
  0.0008     276848     -> {closure:…/closure-trace.phpt:20-22}() …/closure-trace.phpt:24
  0.0009     277168       -> strlen() …/closure-trace.phpt:21
                           >=> 3
                         >=> 3
                       >=> NULL
  0.0010     276056   -> xdebug_stop_trace() …/closure-trace.phpt:28
  0.0010     276184
TRACE END   [2015-03-28 18:48:39]

But in Xdebug 2.3 you instead see:

TRACE START [2015-03-28 18:48:45]
  0.0008     269144   -> test1() …/closure-trace.phpt:27
  0.0009     270096     -> {closure:…/closure-trace.phpt:20-22}() …/closure-trace.phpt:24
  0.0009     270336       -> strlen() …/closure-trace.phpt:21
  0.0010     270504        >=> 3
  0.0010     270216      >=> 3
  0.0010     269264    >=> NULL
  0.0011     269264   -> xdebug_stop_trace() …/closure-trace.phpt:28
  0.0011     269384
TRACE END   [2015-03-28 18:48:45]

This makes it easier to see how much time and memory a specific function call took, similarly to what already was shown for "computerized" trace files.


And lastly, the computerized format (xdebug.trace_format=1), has now support for showing return values as well. The return value is part of a new "frame" for a function.

Take the following invocation of PHP:

php -dxdebug.collect_params=0 -dxdebug.collect_return=1 \
        -dxdebug.trace_format=1 \
        -r '$a = array(1, "foo", M_PI); xdebug_start_trace("/tmp/trace"); echo
        serialize($a);' \
>/dev/null && cat /tmp/trace.xt

In Xdebug 2.2 this shows:

TRACE START [2015-03-29 00:08:34]
2    1    1    0.000417    267192
2    2    0    0.000547    266848    serialize   0   Command line code   1   1   array (0 => 1, 1 => 'foo', 2 => 3.1415926535898)
2    2    1    0.000626    267024
1    0    1    0.000695    266968
               0.000925    8536
TRACE END   [2015-03-29 00:08:34]

But Xdebug 2.3 has an extra "return frame" with the return value:

TRACE START [2015-03-29 00:10:11]
2    1    1    0.000462    263152
2    2    0    0.000520    262928    serialize   0   Command line code   1   1   array (0 => 1, 1 => 'foo', 2 => 3.1415926535898)
2    2    1    0.000594    263096
2    2    R                          'a:3:{i:0;i:1;i:1;s:3:"foo";i:2;d:3.1415926535897931;}'
1    0    1    0.000676    263040
               0.000881    8512
TRACE END   [2015-03-29 00:10:11]

Function 2 (the call to serialize) now has an entry frame (0) and exit frame (1) showing the time index and memory usage, as well as the new R frame showing the return value. Unlike human readable traces it is not on the same line as the exit frame because of backwards compatibility reasons.


Other parts in this series:

Shortlink

This article has a short URL available: http://drck.me/tracing23-bom

Comments

No comments yet

Life Line