Posts Tagged ‘osm’

Code Sprint Photo

OSM hard at work

I am sitting on a plane flying eastbound at 37,000 feet over middle of America. A few hours ago we left San Francisco the host city for State Of The Map US 2013. State of the Map US is the American regional conference for OpenStreetMap participants and enthusiasts. A common theme for the weekends events was building community.


This weekend we had the second annual Toronto OpenStreetMap developer weekend. The nice folks at the Ryerson Department of Geography hosted us. My focus this weekend was to work the Serge and Martijn on maproulette

Maproulette is software that presents an easy to do mapping task to users which they can complete and then mark the task as completed. Examples of past maproulette mapping challenges include fixing connectivity errors or fixing objects touched by the license change.

I found the documentation on urllib2 a bit unclear about how to get cookie handling working properly.

I was working on a python script that needed to contact the OpenStreetMap web server, login with my OSM credentials and interact with the website.

The first step is to setup a urllib2 opener instance that is configured to store cookies.

import cookielib,Cookie,urllib2,urllib
import xml.etree.cElementTree as ElementTree

cookies = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookies))

This will create an opener that can be used to retrieve URL’s. Any cookies set in the HTTP response will be stored in the cookie’s cookie jar. If I needed to add additional openers (ie for special redirect handling) I would just add them as additional parameters to the build_opener call. ie urllib2.build_opener(handler1, handler2,handler3)…

Next we need to contact OpenStreetMap to get a blank login form. The blank login screen has a hidden variable ‘authenticity_token’ that needs to be passed back as part of the POST with my login credentials

inptag = '{}input'
formtag = '{}form'
# fetch the blank login form
response_tokenfetch =
html =
# parse the HTML elements in the form
# extract any input fields for later resubmission
# this will pick up the authenticity_token and anything else
xml_tree = ElementTree.parse(htmlfile)
for form in xml_tree.getiterator(formtag):                
    for field in form.getiterator(inptag):
        if 'name' in field.attrib and 'value' in field.attrib:
                login_payload[field.attrib['name']] = field.attrib['value']
login_payload['username'] = username
login_payload['password'] = password
login_payload['remember_me'] = 'yes'
login_payload['cookie_test'] = 'true'

Next we submit the LOGIN request as a POST. Any session cookies returned as part of the blank form will be added to the second request.

response =,urllib.urlencode(login_payload))

If our login was successful then cookies contains an _osm_session and _osm_username that will be used in subsequent API calls.

response2 =

You could then parse the HTML to extract a list of messages.
If your using the formal OpenStreetMap API (ie calls under /api/0.6/…) then you should instead use oauth for authentication instead of logging in through the website. Some OSM features such as messaging can only be accessed by pretending to be a web session and parsing/faking HTML.

Slides from my 2012 PGCon presentation Making your Own Maps are now available

The presentation covered

  • Common reasons people render their own maps
  • Where to get OpenStreetMap data and how to load it into your PostGIS database
  • How to use Tilemill to design your own map style
  • How to render map tiles, both statically and dynamically
  • How to use OpenLayers to display your map

The presentation was recorded. I will update this post when the recording comes online.

I’ve been a regular attendee of PGCON since the first year it was held in Ottawa. Like past years I enjoyed the conference and I would like to thank Dan Langille for putting together another first-rate conference. My favourite part of PostgreSQL conferences is meeting and reconnecting with users people in the community.

Updated: The video is available here

Peninsula Lake Map

Peninsula Lake, Huntsville Ontario

I have published a new version of the Peninsula Lake map. This edition includes new cartography and map updates based on my survey work last summer.

I had written the previous map style in Cascadnik, a CSS like language that translates to mapnik XML. Cascadnik has since been replaced by Carto so I have decided to rewrite the map style using TileMill (Map design software that uses Carto and Mapnik).

One of my goals while redesigning the map was to put to use what I have learned about cartography over the past year. The original PenLake map style was based on a sample Cascadnik style adapted to my colour scheme, nautical features and a print scale. The new style was designed by me from scratch to focus on

  • The lake and nearby rivers, streams and marshes
  • Recreational attractions around the lake including hotels and golf courses
  • Roads near the lake
  • The water depth of the lake including suspected rocks

This style is much smaller than the previous one ( the style is less than 300 lines of CSS) and it behaves well at different zoom levels between 12 and 18. I suspect the number of layers can still be further reduced for faster rendering times.

I have made a web version of the map available on Peninsula Lake Map my map hosting site. The map tiles are currently being served by MapBox.

Some of the water depth data in this version of the map is based on observations I manually made while kayaking on the lake in the summer of 2011. I have tried to correct this data to match the sounding datum used by the CHS in their observations. I don’t expect the observations taken by me with a kayak, rope and a tape measure to match the accuracy of the CHS with their fancy sonar but we have to work with the equipment that we have available to us. The depths on the map should be based on a datum of 913 feet.

The Source code is still available from github

The Toronto OSM hack weekend (2012) started on friday with an introduction to OpenStreetMap presentation at the Ryerson Geography department. Experienced OpenStreetMap presenter, Richard Weait gave a talk to a room of Geography students, staff and faculty. After Richard explained the basics of OpenStreetMap A.J. and Tristen gave a short presentation on some of the work that they have been doing for MapBox.

After the presentations we did a QA session with the audience. The OSM ‘experts‘ in the room tried to answer the questions from the audience. Most people in the audience were geographers and GIS people who had already heard of OSM. A number of the attendees already had worked with various components in the OSM stack. We had questions on projections, layers, TileMill, Mapnick, tagging and an assortment of other topics.

After the Q/A I went with A.J and Tristen to Chiplote for some Burritos. At Chipotle we learned three things. 1) Chipotle in Washington does more of the meat preperation in the back room than they do in Toronto; Toronto should take a hint. 2) Keeping the Burrito in the tinfoil wrap it in keeps it from falling apart as you eat it. 3) Next time I need to remember to bring Buritos back to my fellow hackers. Friday night we had a social at the hack weekend social HQ, there was plenty of beer and the room was packed.

Saturday morning I took the train back downtown for the first day of hacking. We had local OSM developers and out of town hackers from three different countries participate along with a group of Ryerson Geography students. I spent some time Saturday morning helping someone get the rails port working on their OSX laptop. The part they seemed to have the most difficulty with was getting PostgresSQL running. This tells me two things. First that the work people in the OSM community have done to make the rails port easier to install has paid off and some PostgreSQL binary distributions could make their user experience a bit better.

I had suggested the EDB one-click installer for OSX but they opt’d for the KyngChao binaries. They were okay installing the package but they weren’t clear how to startup PostgreSQL and connect to it. I feel that the EDB installer makes it easier for an inexperienced PostgreSQL user to get up and running.

In addition to helping people with installing PostgreSQL I also helped some people with SQL queries and helped with changes to the website look cleaner for users logged in but haven’t uploaded any GPX traces.

Saturday night we had another party at social HQ. Party highlights included beer with ingredients from seven continents, comparision of political systems, talk of election rigging. Thai food, wine and beer were also spotted. Most of the geography students from the hack sessions were able to make it out and saw how fun OSM parties can be. I hope to see them at the regular Toronto OSM pub nights.

The GO train back to Oakville was pretty full. This was St. Patricks day and the streets of Toronto were filled with drunk people in green. The 11:43pm train to the suburbs is a bit too early for serious leprecons. Many of the riders had been at a Van Halen concert, it didn’t seem like many of them were interested in making maps.

There is a saying in the OpenStreetMap world that if you want to build an local OpenStreetMap community you should import a bunch of Germans. If you can’t import some Germans the next best thing is to import Richard Weait. He almost single-handedly organized the Toronto hack weekend, secured a venue and provided food, drinks and social entertainment. I also want to thank the Ryerson Geography department and Claus Rinner for letting us use their facilities and providing eager GIS students.

The neighborhood I live in is between 20 and 30 years old, other parts of Oakville are under constant growth. When a new sub-division is built the developer will put in sewers, roads and start building houses. Sometimes the street signs will look home-made other times proper street signs will already be up.

When I see a new subdivision going up near my I try drive (or walk) the roads with my GPS capturing the data for OpenStreetMap. Often my GPS traces in OpenStreetMap will be the first time these roads show up on a map outside of the towns GIS. It might take years before some of the commercial maps send trucks to map these roads.

On Sunday I was driving past a sub-division that was being built and decided to map some of the roads. I got about GPS traces and names for about half of the roads. I need to make a point of going back and finishing the soon. Sub-division construction sites usually have construction workers and heavy equipment moving about. It is important to pay close attention to your surroundings and the condition of the road your mapping often it is a work in progress.

I doubt anyone is going to be moving into these houses until the spring. In the spring when people move into their new houses they will be able to use OpenStreetMap to invite there friends to the house-warming party.