Posts Tagged ‘software’

PostgreSQL is becoming a more popular choice for an embedded database because of its BSD license, relatively low memory footprint and great list of features. A few people have asked me if Slony would be a good choice for replication in an embedded environment. Embedded deployments haven’t been a primary use-case for Slony and some of the challenges you would face are worth writing about.

(more…)

Last week I was in Chicago giving a talk at PostgresOpen on managing PostgreSQL with puppet. The talk was well attended and appears to have been well received.

Puppet is configuration management software that allows you to describe how your servers should look using a declarative syntax. You describe what packages you want to install (obviously postgres) and how your configuration files should look. Puppet also allows you to run commands to create databases or database objects such as users.

In my talk I discuss why it is important to use a repeatable procedure for building production database servers and how this is a tool in bridging the divide between developers and operations staff.

I talk about how deploying servers with automation allows your servers to be similar. Similar might not mean identical but the differences between your database servers is controlled and managed. This also applies to your development and QA servers. If you deploy your staging, QA, and development servers using the same puppet manifest as your production servers but with possibly different configuration options then you will be more confident in your testing.

You can view my slides. They recorded the talk and I will update this post with a link to the talk when it is posted.

Updated: You can view a recording of the video below

This weekend we had the second annual Toronto OpenStreetMap developer weekend. The nice folks at the Ryerson Department of Geography hosted us. My focus this weekend was to work the Serge and Martijn on maproulette

Maproulette is software that presents an easy to do mapping task to users which they can complete and then mark the task as completed. Examples of past maproulette mapping challenges include fixing connectivity errors or fixing objects touched by the license change.
(more…)

I found the documentation on urllib2 a bit unclear about how to get cookie handling working properly.

I was working on a python script that needed to contact the OpenStreetMap web server, login with my OSM credentials and interact with the website.

The first step is to setup a urllib2 opener instance that is configured to store cookies.

import cookielib,Cookie,urllib2,urllib
import xml.etree.cElementTree as ElementTree

cookies = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookies))

This will create an opener that can be used to retrieve URL’s. Any cookies set in the HTTP response will be stored in the cookie’s cookie jar. If I needed to add additional openers (ie for special redirect handling) I would just add them as additional parameters to the build_opener call. ie urllib2.build_opener(handler1, handler2,handler3)…

Next we need to contact OpenStreetMap to get a blank login form. The blank login screen has a hidden variable ‘authenticity_token’ that needs to be passed back as part of the POST with my login credentials

inptag = '{http://www.w3.org/1999/xhtml}input'
formtag = '{http://www.w3.org/1999/xhtml}form'
# fetch the blank login form
response_tokenfetch = opener.open(request)
html = response_tokenfetch.read()
htmlfile=StringIO.StringIO(html)
# parse the HTML elements in the form
# extract any input fields for later resubmission
# this will pick up the authenticity_token and anything else
xml_tree = ElementTree.parse(htmlfile)
for form in xml_tree.getiterator(formtag):                
    for field in form.getiterator(inptag):
        if 'name' in field.attrib and 'value' in field.attrib:
                login_payload[field.attrib['name']] = field.attrib['value']
login_payload['username'] = username
login_payload['password'] = password
login_payload['remember_me'] = 'yes'
login_payload['cookie_test'] = 'true'

Next we submit the LOGIN request as a POST. Any session cookies returned as part of the blank form will be added to the second request.

cookies.add_cookie_header(request)
response = opener.open(request,urllib.urlencode(login_payload))

If our login was successful then cookies contains an _osm_session and _osm_username that will be used in subsequent API calls.

request2=urllib2.Request('http://api06.dev.openstreetmap.org/user/stevens/inbox)
cookies.add_cookie_header(request2)
response2 = opener.open(request2)
html=response2.read()

You could then parse the HTML to extract a list of messages.
If your using the formal OpenStreetMap API (ie calls under /api/0.6/…) then you should instead use oauth for authentication instead of logging in through the website. Some OSM features such as messaging can only be accessed by pretending to be a web session and parsing/faking HTML.

I spent the weekend attending Pycon Canada where I gave a talk on Pl/Python. I want to thank the conference organizers for putting on an excellent conference. I am told that this was the first time Pycon had a regional conference in Canada and that it was put together by a group of volunteers in less than 6 months.

One of my favourite parts of local/regional conferences held on weekends is that they tend to attract attendees who are passionate about computers and technology. The people who I spoke with at the conference were there because they wanted to be there,not because there boss wanted them to be there, and either loved Python or wanted to learn more about it. I’ve attended many great PostgreSQL conferences over the past few years but it was nice to spend sometime talking with people from broader development backgrounds.

In my discussions with people at the conference I noticed a trend. People I spoke with who are working at companies that did Python development tended to be using PostgreSQL. The ones that weren’t currently using PostgreSQL were using MySQL and talking about moving to PostgreSQL or were apologetic for still being on MySQL. The MySQL users were often apologizing before I told them that I was a PostgreSQL contributor. Some of the MySQL users also mentioned that they were using non-Oracle forks like Percona.

This was in contrast to the people at the Python conference that described their workplaces as doing primarily Java development. The Java development shops tended to be using Oracle or SQL Server. I admit that the sample size of of the Java developers wasn’t that big (this was a Python conference after all) but my observations are worth keeping in mind since they might be indicating a pattern. Other people have commented about the popularity of PostgreSQL in the Ruby community.

I wonder how much of this observations is because older written in Java are already using SQL Server/Oracle and there hasn’t been a strong enough driver to change to PostgreSQL. While newer software projects are tending to choose Python or Ruby over Java and at the same time picking a FLOSS database such as PostgreSQL where they don’t have to worry about migrating a legacy application.

My talk on writing stored functions in Pl/Python was well received. A lot of people saw appeal in being able to write their stored functions in Python instead of pl/SQL but that shouldn’t be a surprise considering this was a Python conference.

My slides are available here, the video of the talk is posted at pyvideo

A few months ago I wrote a blog post on compiling FireFox for the powerpc on Debian Linux. I had built Firefox 9 when I was writing that post and in the intervening months, thanks to the Mozilla rapid release cycle, Firefox 16 has been released. Does Firefox 16 still build on the powerpc with Debian Squeeze?

  1. I downloaded the latest firefox source code from mozilla
  2. This time I didn’t need to apply any patches for the powerpc.
  3. When I ran ./configure followed by ‘make’ as described in my post on Firefox9 I encountered a compile error.
    syntax error in VERSION script when trying to build libraries in the subdirectories under security/nss/lib such as util (libnssutil).

The issue is that the mapfiles such as security/nss/lib/util/nssutil.def that are included in the source tree are not suitable as input files to the linker but need to be pre-processed first. The rules to process the ‘MAPFILE’ are defined in security/coreconf/Linux.mk and are invoked by a rule in security/coreconf/Rules.mk that build a proper mapfile by running the mapfile distributed in the source tree (such as nssutil.def) through sed and grep to produce a file that can be passed to the linker.

The make rule takes a mapfile $(MAPFILE_SOURCE)=nssutil.def and generates $(MAPFILE)=nssutil.def from it. If you try building firefox as I describe above it will put the output files in the same directory as the input files. This means that the sed/grep rule won’t be executed because the target (nssutil.def) is already present and isn’t older than the dependency (nssutil.def), itself.

If you instead

  1. Set the shell environment variable MOZ_OBJDIR to be a directory other than the source directory
  2. Put your configure options in the file .mozconfig, as described in the Mozilla build instructions
  3. Run make -f client.mk

firefox 16 will build successfully. The issues I encountered with the version script shouldn’t be specific to the powerpc or debian.

CN Freight Train
Software infrastructure is a lot like municipal infrastructure such as highways, subways, and stadiums. Everyone likes to enjoy the benefits but we seem to have difficulties on when to build it, what to build, or how to pay for it.

(more…)

Firefox 9 on PowerPC

Posted: January 25, 2012 in Uncategorized
Tags: , , ,


My main desktop computer at home is a PowerPC based Mac Mini. I have been running it as a debian Linux system since I bought the machine in 2006. One of my reasons for going with PowerPC was to be different. Choice in CPU architecture is important just like choice in software.

Debian Squeeze includes Firefox (iceweasel) 3.6. The Mozilla rapid release schedule means that in the past year firefox 4, 5, 6, 7, 8 have all come and gone. Newer versions of firefox are rumored to be faster and more memory efficient. Speed and memory efficiency are important if your running a 6+ old mac mini.

I don’t want to upgrade my debian installation to something unstable. Nor could I find packaged powerpc binaries at the Debian Mozilla Team Site. This left me with compiling my own.

I am pleased to report that all I had to do was

  1. Download a source tar for Mozilla 9.0.1
  2. Manually apply the fix for Bug 703534 which involved editing one line in js/src/jscompartment.cpp to fix a compile error
  3. run ./configure
  4. run make
  5. run make install

I have heard rumors that the Mozilla team isn’t officially supporting PowerPC anymore. I am glad that firefox still builds and hope people continue to submit patches and take the time to keep Mozilla running on as many platforms as possible including Linux PowerPC.

Looking back on my activities in the past few months I realize that there have been a few occasions where my role has been to help with the troubleshooting process. In the past year I have given talks at PGEast and PGEU on troubleshooting Slony but the process I use applies to most technology problems.

(more…)

One of the least favourite parts of my job as a software developer is when the project manager comes to me with printout of a project plan done in Microsoft project and asks asks if the plan is okay.

The project plan printout shows a list of tasks, described by a 3-6 word title, along with start and end dates for each task. I am usually asked to look through this and figure out if a) The tasks are taking place in the right order, b) If any tasks are taking place at the same time that shouldn’t be (maybe because they require the same computers, components or people) and if the task estimates sound reasonable.

Making sense of a 100-500 task project plan from this type of list is pretty difficult. It would be much easier to view this as an interactive calendar that lets you view the tasks taking place in a given month, week or day. The other day I found a blog post by Ashish Paliwal on some software to convert a Microsoft Project .MPP file to Google Calendar.

I downloaded the software and spent the morning adding features to it, including

  • Task constraints, dependencies and resources now show up in the event description
  • Events on the calendar run from the start to end date
  • An option to skip tasks that have already completed

The 500+ item project plan I’ve been staring at is now much clearer.
My modified version of the tool is available at github. You can download an executable JAR with my changes from my sourceforge site

To convert a MS-Project .mpp to a Google Calendar just run

java -jar mpp.jar MyProjectPlan.mpp myemail@google.com

Then select one of the calendars from the list and the tool will update it to include the tasks from your project plan.