Last week I was hanging out on the top of San Francisco for puppetconf 2013. PuppetConf is the annual conference dedicated to the puppet configuration management system. Some of the lessons I learned are worth sharing.

1. Walking to the top of Nob Hill is less fun that it looks

After landing at SFO I went from the airport to the BART station. Inside the BART station, at the vending machine, I encountered a pair having difficulty figuring out how to buy a ticket. Having been in San Francisco a few months ago forSOTM-US I considered myself competent at operating the BART vending machine so I helped them put money on a ticket. I then rode the train to Powell station. My hotel was at the top of Nob Hill and the cable-car line seemed long so I decided to walk. Walking up a big steep hill with a laptop and luggage on a warmish summer day is a lot of work.

2. Puppet(conf) is more popular than PosgreSQL(conf).

I have been to a lot of PostgreSQL (and other open-source) conferences over the past 7 years and puppetconf with close to 1200 attendees was larger than FOSS4G 2010 and probably 4 times larger than most of the other conferences I’ve recently attended. Puppet and DevOps are hot stuff these days and all the cool kids appear to be doing it. I spoke to numerous people from old school industries like retail and utilities along with people from technology industry giants like salesforce.com, and three letter I.T vendors. They are all starting to use puppet to help with managing their I.T configuration. The main competitor to puppet seems is Chef, another open-source configuration management solution but I didn’t talk to a single person at the conference who could point to a closed source alternative with comparable features that they were using instead of puppet. Puppet is replacing hundreds of poorly written, and misunderstood shell scripts.

3. Culture Is A Big Deal
An entire conference track along with a keynote was dedicated to the DevOps Culture. No one is really sure what DevOps means but it appears to be almost as vague as cloud and getting almost as much hype. The reason why DevOps and culture is so closely linked to puppet and configuration management is because puppet is a tool to help bridge the cultural divides in your organization. Communicating between people and teams with spoken/written languages such as English is in-precise but communicating how things should be configured though a declarative and executable syntax like puppet manifests is very precise. A puppet manifest puts the developers and operations staff on the same page and can help ensure what QA is testing is actually what will be deployed. Automated configuration management is also really helpful if you want to do frequent deployments into test and production environments.

4. Orchestration is next
Puppet deals with configuring a single server and makes it easy for many servers to be configured in the same way. Often you need to configure multiple servers at the same time in a related way. For example if you are deploying a new version of your software you might need to make some database changes, remove a web-server from the load-balancer, apply the software update to the web-server then add it back into the load balancer pool and repeat for all of your web-servers. Once all of your web-servers have been updated you might need to make another set of database schema changes. Conducting the changes on each of the servers in the right order is called orchestration. Puppetlabs has software called mcollective (and PuppetDB) to help with this but it mostly deals with the RCP aspects of the problem and doesn’t (at least not yet) provide facilities for dealing with errors on one of the servers and rolling back or tools to help with managing dependencies between servers. VMWare claims to have products (vDirector and vSphere Orchastrator) that might do better at this but I didn’t make it to a presentation on them. I expect to see more tooling in this area in the next year.

I will be giving a talk at the PGOpen Conference in Chicago later this month on how Puppet is used to manage PostgreSQL databases that make up key parts of the domain name registry system. Come to PGOpen so you can learn more.

Code Sprint Photo

OSM hard at work

I am sitting on a plane flying eastbound at 37,000 feet over middle of America. A few hours ago we left San Francisco the host city for State Of The Map US 2013. State of the Map US is the American regional conference for OpenStreetMap participants and enthusiasts. A common theme for the weekends events was building community.

Read the rest of this entry »

symbol

E.O Wilson recently published an article in the Wall Street Journal where he argues Scientists don’t need math skills. He feels that many potential scientists are being turned away from science because they find math hard. Wilson goes onto argue that scientists in many fields don’t really need much math. I’m not going to address Wilson’s argument directly but instead talk about what I learned about math in school.

Read the rest of this entry »

I’ve written in the past about compiling recent versions of Firefox for the powerpc under Linux. The Firefox rapid release schedule means that a new firefox release happens almost every 6 weeks. I last wrote about firefox 16, which is now almost 4 releases old. In the past 6 months I’ve been regularly testing firefox builds on the powerpc as they make their way through the development process. I am going to summarize the state of recent firefox versions for ppc users.

Read the rest of this entry »

This weekend we had the second annual Toronto OpenStreetMap developer weekend. The nice folks at the Ryerson Department of Geography hosted us. My focus this weekend was to work the Serge and Martijn on maproulette

Maproulette is software that presents an easy to do mapping task to users which they can complete and then mark the task as completed. Examples of past maproulette mapping challenges include fixing connectivity errors or fixing objects touched by the license change.
Read the rest of this entry »

I found the documentation on urllib2 a bit unclear about how to get cookie handling working properly.

I was working on a python script that needed to contact the OpenStreetMap web server, login with my OSM credentials and interact with the website.

The first step is to setup a urllib2 opener instance that is configured to store cookies.

import cookielib,Cookie,urllib2,urllib
import xml.etree.cElementTree as ElementTree

cookies = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookies))

This will create an opener that can be used to retrieve URL’s. Any cookies set in the HTTP response will be stored in the cookie’s cookie jar. If I needed to add additional openers (ie for special redirect handling) I would just add them as additional parameters to the build_opener call. ie urllib2.build_opener(handler1, handler2,handler3)…

Next we need to contact OpenStreetMap to get a blank login form. The blank login screen has a hidden variable ‘authenticity_token’ that needs to be passed back as part of the POST with my login credentials

inptag = '{http://www.w3.org/1999/xhtml}input'
formtag = '{http://www.w3.org/1999/xhtml}form'
# fetch the blank login form
response_tokenfetch = opener.open(request)
html = response_tokenfetch.read()
htmlfile=StringIO.StringIO(html)
# parse the HTML elements in the form
# extract any input fields for later resubmission
# this will pick up the authenticity_token and anything else
xml_tree = ElementTree.parse(htmlfile)
for form in xml_tree.getiterator(formtag):                
    for field in form.getiterator(inptag):
        if 'name' in field.attrib and 'value' in field.attrib:
                login_payload[field.attrib['name']] = field.attrib['value']
login_payload['username'] = username
login_payload['password'] = password
login_payload['remember_me'] = 'yes'
login_payload['cookie_test'] = 'true'

Next we submit the LOGIN request as a POST. Any session cookies returned as part of the blank form will be added to the second request.

cookies.add_cookie_header(request)
response = opener.open(request,urllib.urlencode(login_payload))

If our login was successful then cookies contains an _osm_session and _osm_username that will be used in subsequent API calls.

request2=urllib2.Request('http://api06.dev.openstreetmap.org/user/stevens/inbox)
cookies.add_cookie_header(request2)
response2 = opener.open(request2)
html=response2.read()

You could then parse the HTML to extract a list of messages.
If your using the formal OpenStreetMap API (ie calls under /api/0.6/…) then you should instead use oauth for authentication instead of logging in through the website. Some OSM features such as messaging can only be accessed by pretending to be a web session and parsing/faking HTML.

I spent the weekend attending Pycon Canada where I gave a talk on Pl/Python. I want to thank the conference organizers for putting on an excellent conference. I am told that this was the first time Pycon had a regional conference in Canada and that it was put together by a group of volunteers in less than 6 months.

One of my favourite parts of local/regional conferences held on weekends is that they tend to attract attendees who are passionate about computers and technology. The people who I spoke with at the conference were there because they wanted to be there,not because there boss wanted them to be there, and either loved Python or wanted to learn more about it. I’ve attended many great PostgreSQL conferences over the past few years but it was nice to spend sometime talking with people from broader development backgrounds.

In my discussions with people at the conference I noticed a trend. People I spoke with who are working at companies that did Python development tended to be using PostgreSQL. The ones that weren’t currently using PostgreSQL were using MySQL and talking about moving to PostgreSQL or were apologetic for still being on MySQL. The MySQL users were often apologizing before I told them that I was a PostgreSQL contributor. Some of the MySQL users also mentioned that they were using non-Oracle forks like Percona.

This was in contrast to the people at the Python conference that described their workplaces as doing primarily Java development. The Java development shops tended to be using Oracle or SQL Server. I admit that the sample size of of the Java developers wasn’t that big (this was a Python conference after all) but my observations are worth keeping in mind since they might be indicating a pattern. Other people have commented about the popularity of PostgreSQL in the Ruby community.

I wonder how much of this observations is because older written in Java are already using SQL Server/Oracle and there hasn’t been a strong enough driver to change to PostgreSQL. While newer software projects are tending to choose Python or Ruby over Java and at the same time picking a FLOSS database such as PostgreSQL where they don’t have to worry about migrating a legacy application.

My talk on writing stored functions in Pl/Python was well received. A lot of people saw appeal in being able to write their stored functions in Python instead of pl/SQL but that shouldn’t be a surprise considering this was a Python conference.

My slides are available here, the video of the talk is posted at pyvideo