Voice of Kibera, Ushahidi 2.0, and Our Wishlist

by: August 22nd, 2010 comments: 3

With the much appreciated help of crowd-sorcerer Henry Addo, we have upgraded Voice of Kibera to Ushahidi 2.0, and the plugin system.

The first plugin we installed immediately was the mobile plugin, which sure, gave us iphone and ipad support but what really excited us was support for any phone, really any phone, with any kind of internet access, via a web browser. In Kenya, mobile internet is booming and it is not uncommon for Kibera people to have internet enabled phones, mostly accessing Facebook. The cheapest internet enabled phone on the market is only 2000 KSH ($25). Phones and data are only going to get cheaper. For many people, the phone will be the first real chance to access the internet, and I reckon WAP is going to really take off. 10 years ago in the EU, WAP was over-hyped and crashed because people were used to a full internet experience, and didn’t really get interested in mobile internet until the iPhone. Here in Kenya, WAP is the right technology, right now. I’m incredibly excited to see what happens now that Voice of Kibera is available on the phone.


An Alert on Alerts

With 2.0 finally out of the way, I had a chance to examine our bugs and features against what’s in 2.0. One long standing issue for VoK, and I’m told other instances, were that Alerts didn’t work correctly. Sometimes they didn’t get sent out at all, or got sent out in huge numbers, almost spamming subscribers (this happened with Uchaguzi, I’m told). I had never investigated or confirmed this, but after a quick test yesterday with VoK, yes, alerts weren’t working!

I examined the code and the database, and discovered the problem. Reports are marked for alerting when approved via “admin/reports/index”, but not via “admin/reports/edit”. This means that if someone marks a report as approved while applying or reviewing location and category, it’s never sent out for alerting. At least with Voice of Kibera, this is the common usage pattern, and I suspect the majority of instances … the same person who creates and geocodes the report, approves it. The approver is often going to check out the map, rather than just read a summary. It’s only in specific circumstances that index would be used on Ushahidi instances, and I can’t say how often that the report listing would be used. This inconsistency made it a troublesome bug to figure out.

Anyhow, I submitted a bug on this, and David Kobia quickly submitted this fix. I was a little concerned that such a core feature had a major bug, but very glad that Ushahidi quickly responded to my report.

If you’ve noticed any problem with Alerts in your Ushahidi instance, I suggest at least applying David’s fix, if not upgrading to the latest codebase completely.

SMS Wishlist

Along with WAP, we see SMS Alerts as a major way Voice of Kibera will be accessible in Kibera. We’ve examined how things work, and have come up with a number of improvements.

  • Should be able to sign up for Alerts to specific category, rather than everything. I believe the Haiti instance had this, but that hasn’t been integrating to 2.0
  • Should be able to sign up for alerts via SMS. For example, someone interested in sporting events could text in “Kibera subscribe sports” and be signed up. That will text them back information on how to unsubscribe via SMS, etc.
  • Admins should be able to toggle whether a specific message is sent out for alerts. Looking at the code around the bug above, I see this would be straightforward.
  • Admins should be able to mark a report for sending only via SMS, and not on the site. These could be special communications, or take the form of a daily/weekly digest of information.
  • Finally, it would be helpful to assign a name to SMS reporters and subscribers. Reports should be linked to messages that come in via SMS, so that you can see the original message and reporter when approving.

Geo and Other Stuff

Naturally being a mapping guy, I have lots of ideas! One thing that happened in Uchaguzi, and in Haiti and Chile, was a choice of base map layers, so that both OpenStreetMap and Goog were available. These were done by hacking in a little OpenLayers javascript. It would actually be pretty simple to offer a choice of several base map layers in core Ushahidi. Also helpful would be a little design work to make base map choice more obvious.

That could lead to more custom base map layers. During Uchaguzi, there was an unfulfilled need to overlay polling place districts on the map. Since that’s a fairly large KML, a more efficient method on the browser side would be to build up semi-transparent tiles.

Another place to look is geocoding. Currently only Google geocoding is offered, while there are other good, and free, services like Nominatim (based on OSM data) and Geonames. Which geocoder is in use should be somewhat invisible to the reporting interface, and done in an efficient cascade. Also, need to present choices of results to user, rather than just the first.

There may be circumstances where you want to build your own custom geocoder. Again, Uchaguzi could have benefited from geocoding on polling place locations; that database was available, but not with a license shareable with OpenStreetMap (it’s a looong story). What could be done is build up a geocoder using the open source geocommons geocoder, and integrate it with Ushahidi via its RESTful interface.

Anyhow, just a few ideas, which we’ll be processing into specific bug reports and feature requests, and yes, finding time to work on … it’s open source after all!


by: July 19th, 2010 comments: 0

Map Kibera recently had some exciting (and very hard!) work to do in the Mt Elgon district, collecting the locations of over a hundred schools. Those schools double as polling places, and were needed for NDI’s election monitoring work; Primoz will post details soon on the why and what of the week he and Mildred spent in the Mt Elgon hills.

What I want to write about now is a geeky post describing the process of producing OpenStreetMap extracts. NDI needs the data in a different form from OSM’s XML format, namely Shapefiles or CSV matching the schemas of their postgres database, for easy import.

This is something which has become almost routine for Map Kibera, as we’ve been producing extracts for download of our data, filtered by theme (health, security, education, watsan), and in a number of formats. It’s become so routine that I can see a clear way to automate and build non-technical interfaces around the process. An easy interface to get data out of OpenStreetMap, in the format and with the data that you want, shouldn’t require grappling with our tools and would benefit GIS people and others greatly. That’s part of the motivation behind the Humanitarian Data Model.

For now, a dive into the steps and underlying tools, by skipping through the process-mtelgon.sh script. This, and many other bits and pieces of Map Kibera code are up on mapkibera github.

# GET fresh Mt Elgon extract from OSM
wget http://www.openstreetmap.org/api/0.6/map?bbox=34.40356,0.74961,34.83117,0.95577 -O /home/mikel/mtelgon/mtelgon.osm

First, simply download a chunk of OSM xml from the API, around the Mt Elgon district.

/home/mikel/src/osmosis-0.34/bin/osmosis --read-xml file="/home/mikel/mtelgon/mtelgon.osm" --tf accept-nodes "education:type=*" --tf reject-ways --tf reject-relations --write-xml file="/home/mikel/mtelgon/mtelgon.polling.osm"

osmosis is a powerful command line tool for processing OpenStreetMap data in many ways. Here, osmosis is used to take the Mount Elgon data, and extract only nodes that have a tag with key “education:type=*”. That corresponds to all the polling places/schools Primoz and Mildred collected.

cd /home/mikel/mtelgon/shapefile; rm polling.*; osmexport ./shp-polling.oxr /home/mikel/mtelgon/mtelgon.polling.osm .; zip polling-shapefile.zip polling.*; rm polling.*

osmexport is a command line utility packaged with osmlib, a ruby library for handling OSM data. osmexport reads rule files which is a format specifying how to match osm tags into various output formats (Shapefile, CSV, KML). Rule files provide a quite simple way to describe this mapping, but can also incorporate any arbitrary ruby code, so more complicated processing is possible.

The example above is uses a rule file to output polling places shapefiles.

setup :Shp do
    point :polling do
        string :id, 20
        string :name, 100
        string :pollstat, 16
        string :type, 32

nodes do
    if tags['education:type']
        :polling << {:id => id, :name => name, :pollstat => tags['polling_station'], :type => tags['education:type']}

The first part “setup” describes the schema of the shapefile. The second part, “nodes”, iterates through every node in the given OSM file, and builds up an array in polling, which it output into the defined shapefile.

There are plenty of other examples in github.

Paper Mapping in Community Meetings

by: June 15th, 2010 comments: 1

Map Kibera’s focus is online, but net access and technical knowledge is Kibera is still a big challenge. This is why our plan has always been to print up large number of maps (or an atlas) and distribute to all the schools, organizations, clinics, churches, whatever in Kibera, so the work of Map Kibera has maximum exposure and impact.

We’ve also been holding community meetings that feature paper maps. This is very much like what’s called Public Participatory GIS. That decade-old-plus methodology intends to bring mapping to a wide audience, through discussion and map drawing, or sometimes using some lightweight digital technology. The idea is not to capture precise information, but more subjective experience and ideas around place. But now with web mapping, and particularly OpenStreetMap, the public is simply participating, and the line between precise and subjective information is blurring.

Still, paper and hand drawing is a powerful tool. Walking Papers is the prime example of that. As we have moved into the community meetings, we wanted to capture this hybrid … subjective, familiar experience, but in a toolset that leads to easy digitization, reuse and sharing of results. I would’ve previously critiqued PPQIS for the lack of re-use … the results often disappeared to a desktop or report. What we came up with here was much inspired by Local Ground (pdf), which used modifications of Walking Papers to bring paper drawing in Bay Area high schools online.

All the dirty details on our organizational process for this is in the wiki. Here I wanted to document the technical process, and highlight a couple areas for improvement.

Paper Maptake a look.

We start off by printing maps by generating a A1 sized pdf map, using an extract of OSM data on whatever theme. Those get printed down in Nairobi’s industrial area, small run of 5 prints, costs about $10 each (we could probably do better if we had time to bargain). We buy good quality tracing paper from the stationary store in >Westgate. We have sets of colored markers; each marker is associated with a kind of question during the session.

At the meeting we tape down tracing paper on top of the map print, and draw a line over the border of the map and the north arrow. Discussions are led by mappers and us coordinators, a writer is chosen, and the mapping commences. The quality and revelations of the meetings have been amazing, Besides the written map, everything is documented by video, photos, audio, and write ups. We’re still compiling everything, but this page on security shows a preview.

We roll up all the maps for transport home. Unroll, and each tracing paper is affixed to the wall, backed by the blank opposite side of one of the paper maps. The map is photographed with a Digital SLR on a tripod, with flash, at a distance that reduces parallax as much as possible. Once a suitable image is captured, that’s copied to the computer. The image is clipped to the drawn boundary (this is sometimes a little bit of an art), and uploaded to the server. Also placed on the server is a “bounds” file, which simply contains the west,south,east,north bounding box of the map.

On the server, this ruby script is run. It goes over all images in the directory, and creates a world file, then uses gdal to convert the image to a GeoTIFF. The script outputs configuration for TileCache and some javascript to configure the OpenLayers maps.

The result is pretty decent, and very close to the mark. It certainly could be improved, if we had a large scanner, but that’s not available. Alignment, lighting and parallax introduces distortion. Could automatically extract the writing, like Local Ground does with PIL, but for now simply allow for altering transparency of the entire image in the display. The background tiles should be customized to the theme. Some of the text lies outside the clipping area, which could be solved by simply recording registration marks of the four corners. Perhaps with Walking Papers properly integrated, this would be easy. Finally, we’ll be integrated the drawn maps into pages included narratives and media, with links that highlight OSM features in browser.

Where Am I?

You are currently browsing the tech category at Map Kibera.