One of my favourite parts of local/regional conferences held on weekends is that they tend to attract attendees who are passionate about computers and technology. The people who I spoke with at the conference were there because they wanted to be there,not because there boss wanted them to be there, and either loved Python or wanted to learn more about it. I’ve attended many great PostgreSQL conferences over the past few years but it was nice to spend sometime talking with people from broader development backgrounds.

In my discussions with people at the conference I noticed a trend. People I spoke with who are working at companies that did Python development tended to be using PostgreSQL. The ones that weren’t currently using PostgreSQL were using MySQL and talking about moving to PostgreSQL or were apologetic for still being on MySQL. The MySQL users were often apologizing before I told them that I was a PostgreSQL contributor. Some of the MySQL users also mentioned that they were using non-Oracle forks like Percona.

This was in contrast to the people at the Python conference that described their workplaces as doing primarily Java development. The Java development shops tended to be using Oracle or SQL Server. I admit that the sample size of of the Java developers wasn’t that big (this was a Python conference after all) but my observations are worth keeping in mind since they might be indicating a pattern. Other people have commented about the popularity of PostgreSQL in the Ruby community.

I wonder how much of this observations is because older written in Java are already using SQL Server/Oracle and there hasn’t been a strong enough driver to change to PostgreSQL. While newer software projects are tending to choose Python or Ruby over Java and at the same time picking a FLOSS database such as PostgreSQL where they don’t have to worry about migrating a legacy application.

My talk on writing stored functions in Pl/Python was well received. A lot of people saw appeal in being able to write their stored functions in Python instead of pl/SQL but that shouldn’t be a surprise considering this was a Python conference.

My slides are available here, the video of the talk is posted at pyvideo

The presentation covered

• Common reasons people render their own maps
• Where to get OpenStreetMap data and how to load it into your PostGIS database
• How to use Tilemill to design your own map style
• How to render map tiles, both statically and dynamically
• How to use OpenLayers to display your map

The presentation was recorded. I will update this post when the recording comes online.

I’ve been a regular attendee of PGCON since the first year it was held in Ottawa. Like past years I enjoyed the conference and I would like to thank Dan Langille for putting together another first-rate conference. My favourite part of PostgreSQL conferences is meeting and reconnecting with users people in the community.

Updated: The video is available here

In the summer of 2001 I was working for Navtech System Support, an aviation software company. We were using PostgreSQL 6.x to store data for one of our applications. We needed to have an up to date copy of the database available on servers at a remote site. We also needed a standby database server in case our primary server failed.

PostgreSQL didn’t have any built in replication but it did have triggers. My colleagues at Navtech and myself figured that if we wrote a trigger that captured a record of all changes made to a database table we would then know what changes needed to be made on a replica to keep it up to date with the master.

I wrote the initial version of DBMirror over a few weeks during the summer of 2001 targetting PostgreSQL 7.1. I no longer have a copy of the original version but I think it only stored the primary key of each row and referenced the original table while replicating. In the September/October time-frame I spent a few more weeks reworked DBMirror into the form we released it in. This version stored the entire contents of a modified row. This is was the first version that saw production use.

In February of 2002 the management team at Navetch was kind enough to allow us too open-source DBMirror. DBMirror was included with the PostgreSQL source code as contrib/dbmirror where it remained as a contrib module until 2007. No one tracks how many users any particular PostgreSQL replication solution has (or had) but based on mailing list traffic DBMirror was far more popular than contrib/rserv, or any of the other options available until Slony become more mature almost 4 years later.

DBMirror was not particularly efficient, it stored each column of a replicated table in its own row of the pendingData table, the replication daemon was written in perl and the query to approximate the commit order taxed the database server. The cleanup script also created a lot of work for vacuum.

The biggest strength DBMirror had going for it was that it was simple to setup and it was flexible. People could install DBMirror and replicate their database after reading a three page README file. The replication daemon was written in perl and was easy to understand. Many people customized dbmirror.pl to perform site specific tasks like renaming tables or excluding rows that matched a particular regex.

I haven’t modified the dbmirror code in years and I would be surprised if there were many dbmirror installations still running. I am truly grateful that I was given the opportunity to develop and release DBMirror. I continue to be involved with PostgreSQL replication as a Slony developer. I also assist with built in replication where by skills and time allow.

In the past 10 years PostgreSQL replication has come a long way from the few thousand lines of C and perl code that made up dbmirror. I look forward to the advances in replication we will see in the next 10 years.

It has been years since I have heard from the other people at Navtech who were involved in the development, testing and deployment of DBMirror. If any of them are reading this they should send me a note.

My talk covered Slony and streaming replication. It gave an overview of different replication patterns that can crop on in the GIS space. I then gave an overview of the key features and limitation of Slony and streaming replication.

A video of the talk is available at FOSSLC

My slides are available here

FOSS4G is a big tent conference that attracted about 900 attendees from all over the world with over 400 of them from the United States, over 50 from Canada.
A big tent conference is a conference is similar to a big-tent political party. The idea is appeal to a broad base of people by catering to as many groups as possible while keeping true to some key common themes. Out of the 900 people at the conference I’d say 75% raised their hands when asked if it was their first FOSS4G. Last years FOSS4G was held in Europe and also had over 800 attendees.

At FOSS4G the common theme is open source geo-spatial software. This includes open-source desktop GIS systems like QGIS and GRASS. It includes open source map rendering platforms like Mapserver and Mapnik. It also includes open source spatial databases such as PostGIS and MongoDB. The tent is also big enough to include commercial vendors such as ESRI, Safe(authors of FME) who produce commercial products that interface with some open-source geospartial software. Many of the software packages (open-source and commercial) represented at the conference compete with each other but they are able to share the same tent. At the opening reception one attendee (who I won’t name) said “I’m need to find Chris Schmidt and tell him why OpenLayers sucks” conferences that get everyone under one roof for a week allow us to learn from each other so we can improve our software, our documentation and our communities. It is much harder to have a flame-war on a mailing list with someone you have sat down and had a beer with than it is to have a flame-war with someone you have never met.

At FOSS4G almost everyone I spoke to used PostGIS in some fashion, some of them used couchDB or MongoDB for tasks, others had legacy Oracle Spatial or ESRI systems but everyone was aware of PostGIS and most had used it in some fashion. If you are running PostGIS then you are running PostgreSQL. FOSS4G 2011 was the largest collection of PostgreSQL users under one roof in North America to date. As a member of the PostgreSQL community I feel we need to do a better job at making sure that geospatial types feel welcome in our community. Excluding the core PostGIS developers I recognized less than 5 people from the PostgreSQL community at the conference and most of us were there because of our connections to other communities. Having said that, most people I spoke to said that they loved PostGIS and PostgreSQL.

The flight home from FOSS4G is also a good time to reflect on what it means to be a community conference.

• Each year OSGEO asks for submissions/proposals from the community on groups that want to host the next FOSS4G. An OSGEO committee examines these submissions and selects a host city for the next conference.
• OSGEO moves the conference around the world each year. They won’t hold the conference in the same continent two years in a row. This makes it easier for people in different parts of the world too attend. It also means that when the conference is in your backyard you make a special effort to attend because it probably isn’t coming back for at least three years.
• FOSS4G has a call for papers, the papers are then voted on by the wider community as an input into the paper selection processes. Anyone in the community can have input into which talks are selected.
• After FOSS4G a financial report on the conference along with many other documents used in planning the conference are published in a subversion repository. This acts as a resource for future groups considering hosting FOSS4G. It also means that the revenues and expenses are public so people can can see where the money goes.
• FOSS4G does not just appeal to developers. In addition to developers and sysadmins there where many cartographers and geographers at the conference

Next year FOSS4G will be held in Bejing. Various political, cultural, logistical and financial reasons will make it hard for many of the American attendees to go to FOSS4G. Some people in Denver were talking about how there should be a North American FOSS4G next year as well. My thoughts on this are

• It is important to not dilute the FOSS4G brand, FOSS4G is the big geo-spatial conference that travels around the world. If you start splitting the FOSS4G brand you will soon end up with a FOSS4G-EU, FOSS4G-US , FOSS4G-Asia instead of a FOSS4G. Other communities do this (*BSD as an example). Figure out the relationship between FOSS4G, OSGEO and regional conferences before someone announces FOSS4G-US, maybe these regional conferences should be called something other than FOSS4G
• If you are thinking about creating another North American geo-spatial conference think about what other communities your conference can reach out to (OSM, PostgreSQL, Python, etc…). State of the Map was fun for a lot of the same reasons PGCon is fun. You get to spend a few days meeting and hanging out with people that you regularly chat with online and everyone shares a deep passion of a very specific project. A lot of the fun I had with FOSS4G was from the energy of being at such a large conference.
• If you do hold some sort of regional conference, for god sakes don’t hold the conference two weeks before the Bejing conference. Space the conferences out so that it is easier for speakers, attendees and sponsors to attend both. I think the next year will turn out to be a great year for the struggling OpenStreetMap community in the US because of the interest and enthusiasm generated at SOTM and FOSS4G. I hope FOSS4G in Bejing brings a lot more great Asian contributors into the open source geospatial tent next year.

It feels like the plane is descending. I will try to send this off once I’m home.

The talk itself went well, the talk immediately before mine was on Walking Papers and had standing room only. Many people stayed on for my talk leaving me a room with a good crowd. The audience for this talk was the largest of any talks I’ve presented so far (I’m hoping my PostGIS replication talk I present at FOSS4G later in the week will top that). During the lunch break I spoke with a number of people who seemed excited about unlogged tables and KNN.

A video of the talk is available from FOSSLC here.

You can also download a PDF of my slides

I also want to mention the large number of people who have come up to me since the talk and told me how much they like PostgreSQL and PostGIS and how them and their customers are using PostgreSQL or in the process of moving to it from something like Oracle Spatial. You tend to not meet these people at PostgreSQL specific conferences because for them PostgreSQL is just a tool that solves there problems, and just works.

I will try to post regularly while I’m in Denver this week, many of the posts won’t get picked up by the planet.postgresql.org aggregator so check the blog directly if you want to follow all of my Denver adventures.

The talk covers both Slony and Streaming replication. The key points covered in the talk are

• Why use replication
• Some common load balancing architectures
• 6 Simple steps to setting up Slony
• 5 Simple steps to setting up streaming replication

I will update this post to link to a video of the talk when FOSSLC makes it available.

Updated: A video of the presentation is available at here

Bulk Adding of Tables
In previous Slony versions you had to issue a ‘set add table’ or ‘set add sequence’ command for every table you wanted to replicate. We have added the ability to use regular expressions to specify a set of tables (or sequences). For example

set add table( set id=1, tables='public.*');
set add sequences(set id=1, tables='public.foo_.*seq');
set add tables(set id=1, tables='public.foo_[1234]',add sequences=true);


Improved Monitoring
A new table sl_component has been added to Slony that can be used to get insight into the current activities of your slon process.
On each node that a slon is running on sl_component will contain a row for each active thread. As the components of slon do interesting things they will update sl_component.

Implicit Wait For
Properly written slonik scripts are filled with this ‘wait for event(…)‘ commands. The problem is that few people understand when you need them. As a result a lot of people leave out ‘wait for event‘ commands when they need them. Slony 2.1.0 will implicitly perform a ‘wait for event’ in between other commands within a slonik script where they are required. When using slonik you should keep a few things in mind

• Having multiple slonik scripts running at the same time is a bad idea and will confuse the implicit wait for behaviours
• Slonik will perform an implicit wait for in between slonik commands whenever the event node changes
• Before slonik submits a SUBSCRIBE SET command it will wait until the provider node is caught up with all other nodes in the cluster
• Before slonik submits a DROP NODE event it will wait until all other nodes are caught up.
• Before slonik submits a CLONE NODE command slonik will wait until the node being cloned is caught up
• Before a CREATE SET command slonik will wait until any outstanding DROP SET commands have been confirmed by the entire cluster
• wait for event commands have never worked in a TRY block. If you have commands inside of a try block that require an wait for event then slonik will now abort your script from inside the TRY block. This means that some scripts that worked in previous versions of Slony will now abort.

What this means is that in the above situations slonik will wait until your cluster is caught up before submitting commands. For example if you have a node that is a few hours behind and want to drop a different node, that drop node command won’t happen until your behind node has caught up. The implicit ‘wait for’ behaviour can be disabled with a slonik command line switch but remember we made these changes to avoid configuration race conditions.

TRUNCATE TABLE support
If your running against PostgreSQL 8.4 or higher Slony will now replicate TRUNCATE commands

Better performance when Slony is behind
Many people have experienced situations where a node in their cluster gets behind. When this happens the sl_log tables grow and pulling data from a provider to a subscriber to takes so long that the cluster can’t catch up. The query to pull data from sl_log_1 or sl_log_2 was causing a sequential scan of a large table. The queries used for pulling data from the sl_log tables have been modified, they should now use the index when backlogged resulting in better performance.

These are not the only changes in Slony 2.1.0 just some of my favourite picks. Now is a great time to try the beta out and let us know what you think. The more people that try 2.1.0 while it is a beta or release candidate means the less change there will be of bugs making into the 2.1.0 release.

I look forward to reading reports of both successful tests and bugs found during testing.

Note: The regular expressions in my ‘set add table’ examples don’t do exactly what most people intend. There is a ‘bug’ in my example, fixing it might expose a bug in 2.1.0 beta3. Can anyone spot it by testing?

Following State Of the Map I will be hanging around in Denver for FOSS4G 2011 (September 12-16). FOSS4G is the annual conference for open source geo-spatial software. I will be giving a talk on ‘PostGIS replication‘ where I will give an overview of built in replication and Slony. My blog post comparing Slony and 9.0 replication is by far the most popular post on this blog, and the talk will expand on that material.

They are expecting about 1000 people to attend FOSS4G this year. I am expecting there to be a lot of maps and talk about maps. In addition to my talk there are many other PostGIS/PostgreSQL talks on the schedule. If your going to be attending a conference related to databases this September what better place to be than Denver? The early registration discounts end on June 30′th so remember to register before the price goes up

A repository with the changes to clustertest including a sample test script can be found at
https://github.com/ssinger/clustertest-framework/tree/sync_rep_tests

The clustertest framework is a Java program that runs test scripts written in Javascript using the Rhino engine.

Test Framework
The framework is built with ‘ant jar-coordinator’.
The framework consists of

• a test client engine (src/info/slony/clustertest/client) that can be deployed as a client worker on remote machines that checks in with the test coordinator for work. In the below example the client engine is running within the same JVM as the test coordinator.
• A test coordinator (src/info/slony/clustertest/coordinator) that provides classes for interfacing with PostgreSQL commands (createdb, initdb, psql, slonik, slon etc…) and a framework for launching, running and tracking the results of javascript tests.
• Some utility classes common to both components

The Example Test
The example test script src/info/slony/clustertest/example/sync_rep_test.js has a Javascript class Example1 that extends the Javascript class StreamingRepBase.

Some of the more interesting methods of StreamingRepBase are:

• initCluster() creates a new database cluster by calling initdb and then enables streaming replication by modifying the postgresql.conf and pb_hba.conf files
• setupSlave() copies the data directory and creates a recovery.conf file. This method assumes that postgres is not running when it is called (it does not call pg_start_backup).
• seedData() Populates the disorder tables. This should be called after disorder-1.sql has been run but before disorder-2.sql
• generateLoad() Will create a set of client workers that will perform database transactions. These transactions run in the background until stop() is called on the ClientScript object.
• sync() will grab the current WAL write_location from the master and then wait until the replay_location() is caught up to that point

Configuring the example test
The file src/info/slony/clustertest/example/sync_rep_example.properties contains property settings used by the test. Settings such as, where to find the postgresql binary files, and what ports, users and passwords to use, are set in this file.

To run the sample test (after editing the properties file) you would run
java -jar clustertest-coordinator.jar src/info/slony/clustertest/examples/sync_rep_example.js src/info/slony/clustertest/examples/sync_rep_example.properties)

A sample test script
src/info/slony/clustertest/example/sync_rep_example.js
contains an example test script. This script will

1. Perform an initdb to create a new database cluster
2. Modify the postgresql.conf to use the port specified in the properties file and configure synchronous streaming replication
3. Copy the data directory to so it can act as a base backup for a slave
4. Perform the required config changes on the slave data directory
5. Start up both instances of PostgreSQL

Once both instances of PostgreSQL are running the test script will setup the disorder schema. disorder is a simulated retail workload that was written to test Slony. The schema consists of tables storing orders, items, inventory and customers along with stored procedures that populate the dataset.

The sample script will then

1. Generate some load by launching a set of concurrent clients that execute scripts (see disorder.js) that perform business transactions on the store (buy items, add inventory etc…) .
2. The load will run for a minute then it is stopped. The test script will then wait until the slave database is caught up in applying the WAL for these transactions.
3. The testing framework compares the contents of the two databases to make sure all data is visible on the slave.
4. The testing framework then starts up a long running transaction on the slave. This transaction will read all rows in do_inventory and then sit idle with the transaction open.
5. While the transaction on the slave is open, the test script will generates more load on the master and verifies that the order count is increasing. The test script also periodically vacuums the do_inventory table
6. The transaction load will run for 5 minutes. Then the test script will verify that the original slave transaction has in fact been aborted. We expect the vacuum to invoke the cancellation behaviour in PostgreSQL 9.1 for slaves. If this does not happen the test records a failure.

When the test run finishes the results can be viewed in the results/$testrun/testResult.test.txt file

This sample is intended to demonstrate how clustertest can be used to test PostgreSQL built-in replication. It is not a comprehensive regression test for streaming replication. The changes to the clustertest framework have not yet been merged into the main clusterteset git repository but are available on my github fork.


]]> http://scanningpages.wordpress.com/2011/05/29/testing-streaming-replication-with-clustertest/feed/ 3 scanningpages