Map Kibera recently had some exciting (and very hard!) work to do in the Mt Elgon district, collecting the locations of over a hundred schools. Those schools double as polling places, and were needed for NDI’s election monitoring work; Primoz will post details soon on the why and what of the week he and Mildred spent in the Mt Elgon hills.
What I want to write about now is a geeky post describing the process of producing OpenStreetMap extracts. NDI needs the data in a different form from OSM’s XML format, namely Shapefiles or CSV matching the schemas of their postgres database, for easy import.
This is something which has become almost routine for Map Kibera, as we’ve been producing extracts for download of our data, filtered by theme (health, security, education, watsan), and in a number of formats. It’s become so routine that I can see a clear way to automate and build non-technical interfaces around the process. An easy interface to get data out of OpenStreetMap, in the format and with the data that you want, shouldn’t require grappling with our tools and would benefit GIS people and others greatly. That’s part of the motivation behind the Humanitarian Data Model.
For now, a dive into the steps and underlying tools, by skipping through the process-mtelgon.sh script. This, and many other bits and pieces of Map Kibera code are up on mapkibera github.
# GET fresh Mt Elgon extract from OSM
wget http://www.openstreetmap.org/api/0.6/map?bbox=34.40356,0.74961,34.83117,0.95577 -O /home/mikel/mtelgon/mtelgon.osm
First, simply download a chunk of OSM xml from the API, around the Mt Elgon district.
/home/mikel/src/osmosis-0.34/bin/osmosis --read-xml file="/home/mikel/mtelgon/mtelgon.osm" --tf accept-nodes "education:type=*" --tf reject-ways --tf reject-relations --write-xml file="/home/mikel/mtelgon/mtelgon.polling.osm"
osmosis is a powerful command line tool for processing OpenStreetMap data in many ways. Here, osmosis is used to take the Mount Elgon data, and extract only nodes that have a tag with key “education:type=*”. That corresponds to all the polling places/schools Primoz and Mildred collected.
cd /home/mikel/mtelgon/shapefile; rm polling.*; osmexport ./shp-polling.oxr /home/mikel/mtelgon/mtelgon.polling.osm .; zip polling-shapefile.zip polling.*; rm polling.*
osmexport is a command line utility packaged with osmlib, a ruby library for handling OSM data. osmexport reads rule files which is a format specifying how to match osm tags into various output formats (Shapefile, CSV, KML). Rule files provide a quite simple way to describe this mapping, but can also incorporate any arbitrary ruby code, so more complicated processing is possible.
The example above is uses a rule file to output polling places shapefiles.
setup :Shp do
point :polling do
string :id, 20
string :name, 100
string :pollstat, 16
string :type, 32
end
end
nodes do
if tags['education:type']
:polling << {:id => id, :name => name, :pollstat => tags['polling_station'], :type => tags['education:type']}
end
end
The first part “setup” describes the schema of the shapefile. The second part, “nodes”, iterates through every node in the given OSM file, and builds up an array in polling, which it output into the defined shapefile.
There are plenty of other examples in github.
Fredrick Bary
