Posts Tagged ‘cookies’

I found the documentation on urllib2 a bit unclear about how to get cookie handling working properly.

I was working on a python script that needed to contact the OpenStreetMap web server, login with my OSM credentials and interact with the website.

The first step is to setup a urllib2 opener instance that is configured to store cookies.

import cookielib,Cookie,urllib2,urllib
import xml.etree.cElementTree as ElementTree

cookies = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookies))

This will create an opener that can be used to retrieve URL’s. Any cookies set in the HTTP response will be stored in the cookie’s cookie jar. If I needed to add additional openers (ie for special redirect handling) I would just add them as additional parameters to the build_opener call. ie urllib2.build_opener(handler1, handler2,handler3)…

Next we need to contact OpenStreetMap to get a blank login form. The blank login screen has a hidden variable ‘authenticity_token’ that needs to be passed back as part of the POST with my login credentials

inptag = '{}input'
formtag = '{}form'
# fetch the blank login form
response_tokenfetch =
html =
# parse the HTML elements in the form
# extract any input fields for later resubmission
# this will pick up the authenticity_token and anything else
xml_tree = ElementTree.parse(htmlfile)
for form in xml_tree.getiterator(formtag):                
    for field in form.getiterator(inptag):
        if 'name' in field.attrib and 'value' in field.attrib:
                login_payload[field.attrib['name']] = field.attrib['value']
login_payload['username'] = username
login_payload['password'] = password
login_payload['remember_me'] = 'yes'
login_payload['cookie_test'] = 'true'

Next we submit the LOGIN request as a POST. Any session cookies returned as part of the blank form will be added to the second request.

response =,urllib.urlencode(login_payload))

If our login was successful then cookies contains an _osm_session and _osm_username that will be used in subsequent API calls.

response2 =

You could then parse the HTML to extract a list of messages.
If your using the formal OpenStreetMap API (ie calls under /api/0.6/…) then you should instead use oauth for authentication instead of logging in through the website. Some OSM features such as messaging can only be accessed by pretending to be a web session and parsing/faking HTML.