State Of The Map US 2013

OSM hard at work

I am sitting on a plane flying eastbound at 37,000 feet over middle of America. A few hours ago we left San Francisco the host city for State Of The Map US 2013. State of the Map US is the American regional conference for OpenStreetMap participants and enthusiasts. A common theme for the weekends events was building community.

Advertisement

Maproulette Hack Weekend

This weekend we had the second annual Toronto OpenStreetMap developer weekend. The nice folks at the Ryerson Department of Geography hosted us. My focus this weekend was to work the Serge and Martijn on maproulette

Maproulette is software that presents an easy to do mapping task to users which they can complete and then mark the task as completed. Examples of past maproulette mapping challenges include fixing connectivity errors or fixing objects touched by the license change.

Python urllib2, cookies and OpenStreetMap

I found the documentation on urllib2 a bit unclear about how to get cookie handling working properly.

I was working on a python script that needed to contact the OpenStreetMap web server, login with my OSM credentials and interact with the website.

The first step is to setup a urllib2 opener instance that is configured to store cookies.

import cookielib,Cookie,urllib2,urllib
import xml.etree.cElementTree as ElementTree

cookies = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookies))

This will create an opener that can be used to retrieve URL’s. Any cookies set in the HTTP response will be stored in the cookie’s cookie jar. If I needed to add additional openers (ie for special redirect handling) I would just add them as additional parameters to the build_opener call. ie urllib2.build_opener(handler1, handler2,handler3)…

Next we need to contact OpenStreetMap to get a blank login form. The blank login screen has a hidden variable ‘authenticity_token’ that needs to be passed back as part of the POST with my login credentials

inptag = '{http://www.w3.org/1999/xhtml}input'
formtag = '{http://www.w3.org/1999/xhtml}form'
# fetch the blank login form
response_tokenfetch = opener.open(request)
html = response_tokenfetch.read()
htmlfile=StringIO.StringIO(html)
# parse the HTML elements in the form
# extract any input fields for later resubmission
# this will pick up the authenticity_token and anything else
xml_tree = ElementTree.parse(htmlfile)
for form in xml_tree.getiterator(formtag):                
    for field in form.getiterator(inptag):
        if 'name' in field.attrib and 'value' in field.attrib:
                login_payload[field.attrib['name']] = field.attrib['value']
login_payload['username'] = username
login_payload['password'] = password
login_payload['remember_me'] = 'yes'
login_payload['cookie_test'] = 'true'

Next we submit the LOGIN request as a POST. Any session cookies returned as part of the blank form will be added to the second request.

cookies.add_cookie_header(request)
response = opener.open(request,urllib.urlencode(login_payload))

If our login was successful then cookies contains an _osm_session and _osm_username that will be used in subsequent API calls.

request2=urllib2.Request('http://api06.dev.openstreetmap.org/user/stevens/inbox)
cookies.add_cookie_header(request2)
response2 = opener.open(request2)
html=response2.read()

You could then parse the HTML to extract a list of messages.
If your using the formal OpenStreetMap API (ie calls under /api/0.6/…) then you should instead use oauth for authentication instead of logging in through the website. Some OSM features such as messaging can only be accessed by pretending to be a web session and parsing/faking HTML.

PGCon2012 – Making Maps

Slides from my 2012 PGCon presentation Making your Own Maps are now available

The presentation covered

• Common reasons people render their own maps
• Where to get OpenStreetMap data and how to load it into your PostGIS database
• How to use Tilemill to design your own map style
• How to render map tiles, both statically and dynamically
• How to use OpenLayers to display your map

The presentation was recorded. I will update this post when the recording comes online.

I’ve been a regular attendee of PGCON since the first year it was held in Ottawa. Like past years I enjoyed the conference and I would like to thank Dan Langille for putting together another first-rate conference. My favourite part of PostgreSQL conferences is meeting and reconnecting with users people in the community.

Updated: The video is available here

Penlake Map May 2012

Peninsula Lake, Huntsville Ontario

I have published a new version of the Peninsula Lake map. This edition includes new cartography and map updates based on my survey work last summer.

I had written the previous map style in Cascadnik, a CSS like language that translates to mapnik XML. Cascadnik has since been replaced by Carto so I have decided to rewrite the map style using TileMill (Map design software that uses Carto and Mapnik).

One of my goals while redesigning the map was to put to use what I have learned about cartography over the past year. The original PenLake map style was based on a sample Cascadnik style adapted to my colour scheme, nautical features and a print scale. The new style was designed by me from scratch to focus on

• The lake and nearby rivers, streams and marshes
• Recreational attractions around the lake including hotels and golf courses
• Roads near the lake
• The water depth of the lake including suspected rocks

This style is much smaller than the previous one ( the style is less than 300 lines of CSS) and it behaves well at different zoom levels between 12 and 18. I suspect the number of layers can still be further reduced for faster rendering times.

I have made a web version of the map available on Peninsula Lake Map my map hosting site. The map tiles are currently being served by MapBox.

Some of the water depth data in this version of the map is based on observations I manually made while kayaking on the lake in the summer of 2011. I have tried to correct this data to match the sounding datum used by the CHS in their observations. I don’t expect the observations taken by me with a kayak, rope and a tape measure to match the accuracy of the CHS with their fancy sonar but we have to work with the equipment that we have available to us. The depths on the map should be based on a datum of 913 feet.

The Source code is still available from github

OSM Hack Weekend (Toronto 2012)

The Toronto OSM hack weekend (2012) started on friday with an introduction to OpenStreetMap presentation at the Ryerson Geography department. Experienced OpenStreetMap presenter, Richard Weait gave a talk to a room of Geography students, staff and faculty. After Richard explained the basics of OpenStreetMap A.J. and Tristen gave a short presentation on some of the work that they have been doing for MapBox.

After the presentations we did a QA session with the audience. The OSM ‘experts‘ in the room tried to answer the questions from the audience. Most people in the audience were geographers and GIS people who had already heard of OSM. A number of the attendees already had worked with various components in the OSM stack. We had questions on projections, layers, TileMill, Mapnick, tagging and an assortment of other topics.

After the Q/A I went with A.J and Tristen to Chiplote for some Burritos. At Chipotle we learned three things. 1) Chipotle in Washington does more of the meat preperation in the back room than they do in Toronto; Toronto should take a hint. 2) Keeping the Burrito in the tinfoil wrap it in keeps it from falling apart as you eat it. 3) Next time I need to remember to bring Buritos back to my fellow hackers. Friday night we had a social at the hack weekend social HQ, there was plenty of beer and the room was packed.

Saturday morning I took the train back downtown for the first day of hacking. We had local OSM developers and out of town hackers from three different countries participate along with a group of Ryerson Geography students. I spent some time Saturday morning helping someone get the rails port working on their OSX laptop. The part they seemed to have the most difficulty with was getting PostgresSQL running. This tells me two things. First that the work people in the OSM community have done to make the rails port easier to install has paid off and some PostgreSQL binary distributions could make their user experience a bit better.

I had suggested the EDB one-click installer for OSX but they opt’d for the KyngChao binaries. They were okay installing the package but they weren’t clear how to startup PostgreSQL and connect to it. I feel that the EDB installer makes it easier for an inexperienced PostgreSQL user to get up and running.

In addition to helping people with installing PostgreSQL I also helped some people with SQL queries and helped with changes to the website look cleaner for users logged in but haven’t uploaded any GPX traces.

Saturday night we had another party at social HQ. Party highlights included beer with ingredients from seven continents, comparision of political systems, talk of election rigging. Thai food, wine and beer were also spotted. Most of the geography students from the hack sessions were able to make it out and saw how fun OSM parties can be. I hope to see them at the regular Toronto OSM pub nights.

The GO train back to Oakville was pretty full. This was St. Patricks day and the streets of Toronto were filled with drunk people in green. The 11:43pm train to the suburbs is a bit too early for serious leprecons. Many of the riders had been at a Van Halen concert, it didn’t seem like many of them were interested in making maps.

There is a saying in the OpenStreetMap world that if you want to build an local OpenStreetMap community you should import a bunch of Germans. If you can’t import some Germans the next best thing is to import Richard Weait. He almost single-handedly organized the Toronto hack weekend, secured a venue and provided food, drinks and social entertainment. I also want to thank the Ryerson Geography department and Claus Rinner for letting us use their facilities and providing eager GIS students.

Virgin Roads

The neighborhood I live in is between 20 and 30 years old, other parts of Oakville are under constant growth. When a new sub-division is built the developer will put in sewers, roads and start building houses. Sometimes the street signs will look home-made other times proper street signs will already be up.

When I see a new subdivision going up near my I try drive (or walk) the roads with my GPS capturing the data for OpenStreetMap. Often my GPS traces in OpenStreetMap will be the first time these roads show up on a map outside of the towns GIS. It might take years before some of the commercial maps send trucks to map these roads.

On Sunday I was driving past a sub-division that was being built and decided to map some of the roads. I got about GPS traces and names for about half of the roads. I need to make a point of going back and finishing the soon. Sub-division construction sites usually have construction workers and heavy equipment moving about. It is important to pay close attention to your surroundings and the condition of the road your mapping often it is a work in progress.

I doubt anyone is going to be moving into these houses until the spring. In the spring when people move into their new houses they will be able to use OpenStreetMap to invite there friends to the house-warming party.

Denver Day 1, What’s new in PostgreSQL 9.1

Saturday was my first full day in Denver for the OpenStreetMap State Of The Map conference. A few years ago I presented a talk at PGCon on OpenStreetMap. I have now returned the favour and presented a PostgreSQL talk (What’s new in PostgreSQL 9.1) at an OpenStreetMap conference.

Denver, Day 0

Friday morning I woke up and got an early start to my workday so I could get a full days work in before leaving for Denver. I did not get a Slony release packaged on Friday but Chris might get to this early next week.

The cab ride, checkin, and US customs (you clear US customs in Toronto before departure) were mostly un-noteworthy. There were no big delays or holdsups at Airport security, I was directed into a line for a scanner that was shared with an Air Crew line and a large (5 people?) United crew skipped in ahead of me. One of them apologized (but didn’t offer to wait their turn in line) she said she was based out of Denver and that they were flying to Chicago. The weather was rainy when she left Denver. I also noticed that the million+ dollar full body airport scanners were sitting in the middle of the room unused (Sept 9, 2011).

The flight to Denver was pretty un-noteworthy, clear skies with some light chop at times. United Express is a bit behind the times and hasn’t upgraded their planes to include laptop power. On arrival at Denver we sat at the gate for about 5 minutes as they tried to get the gate to move (so it would line up with the airplane door). They almost resorted to having to push the airplane into position. It sounded like rebooting the gate helped, isn’t technology great.

In Denver I am staying at the Sheraton which is hosting FOSS4G. The hotel seems nice but the rooms don’t have a fridge. They have a box that looks like a fridge, enough so that I went out and bought a jug of orange juice for the week only to discover that the fridge looking box is am empty. Bye-bye warm Orange juice.

After checking in and getting a feel for the area it was a little after 10pm local (midnight eastern) and I noticed many restaurants were starting to close. I went on a hunt for a small dinner. I found Wild Bangkok a Thai place that was still open. When I walked in I was greeted by a guy who wasn’t really dressed like a waiter but he took me to a seat at the bar, handed me a menu, told me how this was really authentic thai and stood there talking to me as I looked through the menu. On each page of the menu he said that the restaraunts specialty was something else, first page seafood, second page curries, third page bowls. I finally ordered a Bangkok bowl with tofu, mostly because I didn’t want to examine the menu more closely with this guy staring at me. As far as Thai food goes it was okay but nothing special. Later I noticed the ‘waiter’ sitting at a table being served drinks. A bartender in a uniform served me my food and gave me my bill.

I am now (Saturday morning) going to try to find my way to the SOTM conference site.

Happy OSM Birthday (to me)

Yesterday (It is still the 10’th in my timezone) was my 3rd OSM birthday. Pascal Neis has built http://osmbirthday.neis-one.org a site that shows you your OSM signup is anniversary and how you rank (in terms of signup time) with respect to other OSM contributors that have made an edits.

I am contributor 17,540 (out of about 143,000 and growing).

In the summer of 2008 I bought an Openmoko Freerunner, an open cellphone running Linux (with xterms and vi). The phone includes a GPS, I needed a source of map data for it to be useful. The first GPS application that I installed (tangogps) used OpenStreetMap tiles. The OpenStreetMap was missing many streets in Oakville and it wasn’t long before I was taking summer walks with my Openmoko and notepad collecting data for OpenStreetMap.

A little over a month later on a friday night I drove to a cafe in Hamilton where I nervously climbed up a set of creeky old stairs. I walked into a dark room illuminated by an overhead project and laptop LCD backlight. The room was filled with a handful of people sitting around a table. Glancing at the people in the room I was pretty sure that I hadn’t just walked into a meeting of the local chamber of commerce, which was fine since I was attending my first OpenStreetMap mapping party and not my first chamber of commerce meeting.

During the evening and the mapping party the next day I met some interesting people, learned some new mapping techniques and made it out to do some mapping. The trend of meeting cool people and learning new things (and not just about OSM), has continued for the past three years.

During the past three years in addition making friends and attending mapping parties I have

• Conducted an import of GeoBase roads in Alberta and Ontario. I am proud of and happy with both the process we used for the import and the results of it. The imported roads aren’t perfect ( no map is ever perfect) but resulting map is a lot more than the map we had before
• Given presentations on OpenStreetMap at a number of conferences
• Seen looks of delight on peoples faces when I’ve handed them a free ledger sized colour map of there local area made with OpenStreetMap data
• Perhaps made a few enemies along the way

I’m looking forward to meeting more mappers next month in Denver at StateOfTheMap and many more years of mapping.