In the summer of 2001 I was working for Navtech System Support, an aviation software company. We were using PostgreSQL 6.x to store data for one of our applications. We needed to have an up to date copy of the database available on servers at a remote site. We also needed a standby database server in case our primary server failed.

PostgreSQL didn’t have any built in replication but it did have triggers. My colleagues at Navtech and myself figured that if we wrote a trigger that captured a record of all changes made to a database table we would then know what changes needed to be made on a replica to keep it up to date with the master.

I wrote the initial version of DBMirror over a few weeks during the summer of 2001 targetting PostgreSQL 7.1. I no longer have a copy of the original version but I think it only stored the primary key of each row and referenced the original table while replicating. In the September/October time-frame I spent a few more weeks reworked DBMirror into the form we released it in. This version stored the entire contents of a modified row. This is was the first version that saw production use.

In February of 2002 the management team at Navetch was kind enough to allow us too open-source DBMirror. DBMirror was included with the PostgreSQL source code as contrib/dbmirror where it remained as a contrib module until 2007. No one tracks how many users any particular PostgreSQL replication solution has (or had) but based on mailing list traffic DBMirror was far more popular than contrib/rserv, or any of the other options available until Slony become more mature almost 4 years later.

DBMirror was not particularly efficient, it stored each column of a replicated table in its own row of the pendingData table, the replication daemon was written in perl and the query to approximate the commit order taxed the database server. The cleanup script also created a lot of work for vacuum.

The biggest strength DBMirror had going for it was that it was simple to setup and it was flexible. People could install DBMirror and replicate their database after reading a three page README file. The replication daemon was written in perl and was easy to understand. Many people customized dbmirror.pl to perform site specific tasks like renaming tables or excluding rows that matched a particular regex.

I haven’t modified the dbmirror code in years and I would be surprised if there were many dbmirror installations still running. I am truly grateful that I was given the opportunity to develop and release DBMirror. I continue to be involved with PostgreSQL replication as a Slony developer. I also assist with built in replication where by skills and time allow.

In the past 10 years PostgreSQL replication has come a long way from the few thousand lines of C and perl code that made up dbmirror. I look forward to the advances in replication we will see in the next 10 years.

It has been years since I have heard from the other people at Navtech who were involved in the development, testing and deployment of DBMirror. If any of them are reading this they should send me a note.

Debian Squeeze includes Firefox (iceweasel) 3.6. The Mozilla rapid release schedule means that in the past year firefox 4, 5, 6, 7, 8 have all come and gone. Newer versions of firefox are rumored to be faster and more memory efficient. Speed and memory efficiency are important if your running a 6+ old mac mini.

I don’t want to upgrade my debian installation to something unstable. Nor could I find packaged powerpc binaries at the Debian Mozilla Team Site. This left me with compiling my own.

I am pleased to report that all I had to do was

1. Download a source tar for Mozilla 9.0.1
2. Manually apply the fix for Bug 703534 which involved editing one line in js/src/jscompartment.cpp to fix a compile error
3. run ./configure
4. run make
5. run make install

I have heard rumors that the Mozilla team isn’t officially supporting PowerPC anymore. I am glad that firefox still builds and hope people continue to submit patches and take the time to keep Mozilla running on as many platforms as possible including Linux PowerPC.

I feel that it is worthwhile to review my rules of troubleshooting

Rule One: Take Ownership
My first rule of troubleshooting is that someone needs to take ownership of the troubleshooting process. If no one else volunteers then this should be you. Taking ownership of the troubleshooting doesn’t mean you are accepting responsibility for fixing the problem and it doesn’t mean you are accepting responsibility for causing the problem. Taking ownership of the process means that your taking responsibility for identifying the problem so that solutions can be proposed.

Most people who have worked in I.T for more than a few weeks have witnessed the finger pointing game. In the finger pointing game one department or vendor points their finger at another department or vendor and blames them for the problem. That department or vendor in turn blames someone else who will then in turn point their finger at the first department or vendor. The problem bounces around between departments, often for weeks, without any real progress being made.

Don’t play the finger pointing game instead say “This problem probably isn’t the fault of my component but the problem causes me pain so I will stick with this problem until solutions are identified, no matter where the root cause may lie or whos fault the problem is“

Rule Two: Ask What is going on
In order to understand why something isn’t working you have to understand what is going on. When troubleshooting a problem I am constantly asking myself “What the **** is going on?“. The conversation typically looks like this

me: What is going on?
me: I don’t know
me: Well what do you know?
me: That the database is giving me an error instead of returning my result
me: What do you know about the error you are getting?
me: Well it means that the database is trying to execute a bad query
me: and what query is it trying to execute?
.
.

Asking yourself the same question again and again might get you strange looks when you are walking down the street but it is a great way to keep you focused and to remind yourself what you have so far discovered.

Rule Three: Understand Why it is happening
Not only do you need to know what is happening but you have to figure out why it is happening. This is a lot harder than finding out what is going on. One way of understanding why something is happening is to have some theoretical knowledge of the system your working with. First I like to get a high level understanding of how the software components fit together. Next I will pick a starting point in the information flow and start tracing the flow of information through the system.

To troubleshoot a broken web application I might start at the incoming HTTP request and follow the the request through the different layers of the system. What is the presentation layer doing in response to the request and why is it doing so? What does the business-rules layer receive as an input and what does it do in response? What queries are being issued to the database and why?

The key thing to remember is that you are tracing cause and effect. Why did X happen? X happened because it was caused by Y

Rule Four: Make a hypothesis and test it
This is the scientific method you should have learned in middle school. Come up with a theory, then figure out what type of experiment will test your theory. Test the theory and observe the results. This will help you validate, or more importantly disprove your theory.

When troubleshooting software problems you need to test your theories and you need to be prepared for your tests to show that your theories are wrong. There is nothing wrong when you come up with a theory that turns out to be wrong. You should tell people (managers, coworkers etc..) about the theories you have tested and turned out to be wrong because it shows you are working on the problem and making progress (ruling out a theory or cause is progress).

Rule Five: Record your findings
After the problem has been identified and fixed (or at least mitigated) it is important to circle back and make a written record of your findings. I like sending an email with my report because they are easily searchable and I tend not to delete the emails I send. Wiki-pages or your companies ticketing system might also be a good place to record the results of your investigation. The important thing is that the report is written, stored somewhere and is searchable. This report should contain

1. A description of what started the troubleshooting investigation. You want to record what was observed before any in-depth troubleshooting began. Ie ‘the site was down, and the server console had a kernel panic message’
2. A list of of the items investigated during the troubleshooting process, what log files were checked, what sub-systems were found to be operating normally and which ones showed issues. It is important to include components that were checked and verified as working normally because later on it might be useful to know which components were unaffected by the issue and you can only know that if you keep a record of what was checked.
3. Any theories that were considered and ruled out during the investigation.
4. The cause of the problem and what testing was done to verify the hypothosis

This record will help the next time a similar problems shows up. You can’t trust your memory to remember these details 6 months or 3 years from now.

The project plan printout shows a list of tasks, described by a 3-6 word title, along with start and end dates for each task. I am usually asked to look through this and figure out if a) The tasks are taking place in the right order, b) If any tasks are taking place at the same time that shouldn’t be (maybe because they require the same computers, components or people) and if the task estimates sound reasonable.

Making sense of a 100-500 task project plan from this type of list is pretty difficult. It would be much easier to view this as an interactive calendar that lets you view the tasks taking place in a given month, week or day. The other day I found a blog post by Ashish Paliwal on some software to convert a Microsoft Project .MPP file to Google Calendar.

I downloaded the software and spent the morning adding features to it, including

• Task constraints, dependencies and resources now show up in the event description
• Events on the calendar run from the start to end date
• An option to skip tasks that have already completed

The 500+ item project plan I’ve been staring at is now much clearer.
My modified version of the tool is available at github. You can download an executable JAR with my changes from my github site

To convert a MS-Project .mpp to a Google Calendar just run

java -jar mpp.jar MyProjectPlan.mpp myemail@google.com

Then select one of the calendars from the list and the tool will update it to include the tasks from your project plan.

The issues I encountered can be summarized as

1. Make sure that ‘PLAIN’ does not show up in the disable-these-authenticators setting of your .pinerc file. Hotmail supports PLAIN and LOGIN SMTP authentication. Alpine will disable LOGIN authentication if the server supports PLAIN. This means that if your config file has disabled PLAIN then you don’t have any valid authenticators
2. Do not specify the /secure option on the options list for your SMTP server. PLAIN authentication is not considered ‘secure’ by alpine (even if your connection uses ssl or tls).

The configuration I used to create an alternate role in Pine to send email through my own domain is:

patterns-roles=
LIT:pattern=”/NICK=steve/FROM=steve@ssinger.info/FLDTYPE=EMAIL” action=”/ROLE=1/FROM=Steve Singer /SMTP=smtp.live.com:587\/tls\/User=steve@ssinger.info/RTYPE=YES/FTYPE=YES/CTYPE=YES”

This is after I’ve setup a free hotmail account to manage my domain (ssinger.info) at http://domains.live.com

The neighborhood I live in is between 20 and 30 years old, other parts of Oakville are under constant growth. When a new sub-division is built the developer will put in sewers, roads and start building houses. Sometimes the street signs will look home-made other times proper street signs will already be up.

When I see a new subdivision going up near my I try drive (or walk) the roads with my GPS capturing the data for OpenStreetMap. Often my GPS traces in OpenStreetMap will be the first time these roads show up on a map outside of the towns GIS. It might take years before some of the commercial maps send trucks to map these roads.

On Sunday I was driving past a sub-division that was being built and decided to map some of the roads. I got about GPS traces and names for about half of the roads. I need to make a point of going back and finishing the soon. Sub-division construction sites usually have construction workers and heavy equipment moving about. It is important to pay close attention to your surroundings and the condition of the road your mapping often it is a work in progress.

I doubt anyone is going to be moving into these houses until the spring. In the spring when people move into their new houses they will be able to use OpenStreetMap to invite there friends to the house-warming party.

My talk covered Slony and streaming replication. It gave an overview of different replication patterns that can crop on in the GIS space. I then gave an overview of the key features and limitation of Slony and streaming replication.

A video of the talk is available at FOSSLC

My slides are available here

FOSS4G is a big tent conference that attracted about 900 attendees from all over the world with over 400 of them from the United States, over 50 from Canada.
A big tent conference is a conference is similar to a big-tent political party. The idea is appeal to a broad base of people by catering to as many groups as possible while keeping true to some key common themes. Out of the 900 people at the conference I’d say 75% raised their hands when asked if it was their first FOSS4G. Last years FOSS4G was held in Europe and also had over 800 attendees.

At FOSS4G the common theme is open source geo-spatial software. This includes open-source desktop GIS systems like QGIS and GRASS. It includes open source map rendering platforms like Mapserver and Mapnik. It also includes open source spatial databases such as PostGIS and MongoDB. The tent is also big enough to include commercial vendors such as ESRI, Safe(authors of FME) who produce commercial products that interface with some open-source geospartial software. Many of the software packages (open-source and commercial) represented at the conference compete with each other but they are able to share the same tent. At the opening reception one attendee (who I won’t name) said “I’m need to find Chris Schmidt and tell him why OpenLayers sucks” conferences that get everyone under one roof for a week allow us to learn from each other so we can improve our software, our documentation and our communities. It is much harder to have a flame-war on a mailing list with someone you have sat down and had a beer with than it is to have a flame-war with someone you have never met.

At FOSS4G almost everyone I spoke to used PostGIS in some fashion, some of them used couchDB or MongoDB for tasks, others had legacy Oracle Spatial or ESRI systems but everyone was aware of PostGIS and most had used it in some fashion. If you are running PostGIS then you are running PostgreSQL. FOSS4G 2011 was the largest collection of PostgreSQL users under one roof in North America to date. As a member of the PostgreSQL community I feel we need to do a better job at making sure that geospatial types feel welcome in our community. Excluding the core PostGIS developers I recognized less than 5 people from the PostgreSQL community at the conference and most of us were there because of our connections to other communities. Having said that, most people I spoke to said that they loved PostGIS and PostgreSQL.

The flight home from FOSS4G is also a good time to reflect on what it means to be a community conference.

• Each year OSGEO asks for submissions/proposals from the community on groups that want to host the next FOSS4G. An OSGEO committee examines these submissions and selects a host city for the next conference.
• OSGEO moves the conference around the world each year. They won’t hold the conference in the same continent two years in a row. This makes it easier for people in different parts of the world too attend. It also means that when the conference is in your backyard you make a special effort to attend because it probably isn’t coming back for at least three years.
• FOSS4G has a call for papers, the papers are then voted on by the wider community as an input into the paper selection processes. Anyone in the community can have input into which talks are selected.
• After FOSS4G a financial report on the conference along with many other documents used in planning the conference are published in a subversion repository. This acts as a resource for future groups considering hosting FOSS4G. It also means that the revenues and expenses are public so people can can see where the money goes.
• FOSS4G does not just appeal to developers. In addition to developers and sysadmins there where many cartographers and geographers at the conference

Next year FOSS4G will be held in Bejing. Various political, cultural, logistical and financial reasons will make it hard for many of the American attendees to go to FOSS4G. Some people in Denver were talking about how there should be a North American FOSS4G next year as well. My thoughts on this are

• It is important to not dilute the FOSS4G brand, FOSS4G is the big geo-spatial conference that travels around the world. If you start splitting the FOSS4G brand you will soon end up with a FOSS4G-EU, FOSS4G-US , FOSS4G-Asia instead of a FOSS4G. Other communities do this (*BSD as an example). Figure out the relationship between FOSS4G, OSGEO and regional conferences before someone announces FOSS4G-US, maybe these regional conferences should be called something other than FOSS4G
• If you are thinking about creating another North American geo-spatial conference think about what other communities your conference can reach out to (OSM, PostgreSQL, Python, etc…). State of the Map was fun for a lot of the same reasons PGCon is fun. You get to spend a few days meeting and hanging out with people that you regularly chat with online and everyone shares a deep passion of a very specific project. A lot of the fun I had with FOSS4G was from the energy of being at such a large conference.
• If you do hold some sort of regional conference, for god sakes don’t hold the conference two weeks before the Bejing conference. Space the conferences out so that it is easier for speakers, attendees and sponsors to attend both. I think the next year will turn out to be a great year for the struggling OpenStreetMap community in the US because of the interest and enthusiasm generated at SOTM and FOSS4G. I hope FOSS4G in Bejing brings a lot more great Asian contributors into the open source geospatial tent next year.

It feels like the plane is descending. I will try to send this off once I’m home.

The thing about plans is that they rarely work as planned. My lift back to Denver had to cancel so I rented my own car intending to meet up with the other OSMers at the park. We never did get never sync’d up so instead I hiked on my own up to Emerald lake, about a 6km round trip hike. The landscape combined with smell of fresh pine was very nice. The terrain reminded me of Banff National park in Canada. The trail I choose is pretty popular (I don’t think I was ever out of eyesight of another hiker for more than a few minutes). The trail was also already well mapped.

After my hike I ventured into the town of Estes Park for lunch. Estes park is a tourist town with a main street that consists of t-shirt shops, candy stores and restaurants. I expected that other OSMers had been through the town mapping earlier in the week but some POI’s might have been missed. I collected a random sampling of tourist centric businesses in the town. Unfortunately my GPS wasn’t recording a GPX trace and I only have the waypoints I captured and my photos. To my surprise (well if I think about it then it makes sense) JOSM can’t sync photos up with GPX waypoints only a trace. This means I will need to come up with a way of converting my collection of waypoints into a track before I can add most of what I captured. It also appears that OSMers were less busy in Estes Park this past week than I was expecting.

Monday night grabbed a seat at the bar in Kattie Mullen’s, a pub attached to the conference hotel, and it didn’t take long for few of the FOSS4G folks at the bar to introduce ourselves and start up a conversation.

I am writing this on Tuesday afternoon sitting outside the conference rooms at the Sheraton surrounded by geo people working on laptops and talking about feature ideas to there respective open-source projects.

The talk itself went well, the talk immediately before mine was on Walking Papers and had standing room only. Many people stayed on for my talk leaving me a room with a good crowd. The audience for this talk was the largest of any talks I’ve presented so far (I’m hoping my PostGIS replication talk I present at FOSS4G later in the week will top that). During the lunch break I spoke with a number of people who seemed excited about unlogged tables and KNN.

A video of the talk is available from FOSSLC here.

You can also download a PDF of my slides

I also want to mention the large number of people who have come up to me since the talk and told me how much they like PostgreSQL and PostGIS and how them and their customers are using PostgreSQL or in the process of moving to it from something like Oracle Spatial. You tend to not meet these people at PostgreSQL specific conferences because for them PostgreSQL is just a tool that solves there problems, and just works.

I will try to post regularly while I’m in Denver this week, many of the posts won’t get picked up by the planet.postgresql.org aggregator so check the blog directly if you want to follow all of my Denver adventures.