PostgreSQL is becoming a more popular choice for an embedded database because of its BSD license, relatively low memory footprint and great list of features. A few people have asked me if Slony would be a good choice for replication in an embedded environment. Embedded deployments haven’t been a primary use-case for Slony and some of the challenges you would face are worth writing about.
Last week I was in Chicago giving a talk at PostgresOpen on managing PostgreSQL with puppet. The talk was well attended and appears to have been well received.
Puppet is configuration management software that allows you to describe how your servers should look using a declarative syntax. You describe what packages you want to install (obviously postgres) and how your configuration files should look. Puppet also allows you to run commands to create databases or database objects such as users.
In my talk I discuss why it is important to use a repeatable procedure for building production database servers and how this is a tool in bridging the divide between developers and operations staff.
I talk about how deploying servers with automation allows your servers to be similar. Similar might not mean identical but the differences between your database servers is controlled and managed. This also applies to your development and QA servers. If you deploy your staging, QA, and development servers using the same puppet manifest as your production servers but with possibly different configuration options then you will be more confident in your testing.
You can view my slides. They recorded the talk and I will update this post with a link to the talk when it is posted.
Updated: You can view a recording of the video below
Tags: configuration management, devops, open source, pgopen2013, puppet, puppetconf2013, puppetlabs, SFO, technology
Last week I was hanging out on the top of San Francisco for puppetconf 2013. PuppetConf is the annual conference dedicated to the puppet configuration management system. Some of the lessons I learned are worth sharing.
1. Walking to the top of Nob Hill is less fun that it looks
After landing at SFO I went from the airport to the BART station. Inside the BART station, at the vending machine, I encountered a pair having difficulty figuring out how to buy a ticket. Having been in San Francisco a few months ago forSOTM-US I considered myself competent at operating the BART vending machine so I helped them put money on a ticket. I then rode the train to Powell station. My hotel was at the top of Nob Hill and the cable-car line seemed long so I decided to walk. Walking up a big steep hill with a laptop and luggage on a warmish summer day is a lot of work.
2. Puppet(conf) is more popular than PosgreSQL(conf).
I have been to a lot of PostgreSQL (and other open-source) conferences over the past 7 years and puppetconf with close to 1200 attendees was larger than FOSS4G 2010 and probably 4 times larger than most of the other conferences I’ve recently attended. Puppet and DevOps are hot stuff these days and all the cool kids appear to be doing it. I spoke to numerous people from old school industries like retail and utilities along with people from technology industry giants like salesforce.com, and three letter I.T vendors. They are all starting to use puppet to help with managing their I.T configuration. The main competitor to puppet seems is Chef, another open-source configuration management solution but I didn’t talk to a single person at the conference who could point to a closed source alternative with comparable features that they were using instead of puppet. Puppet is replacing hundreds of poorly written, and misunderstood shell scripts.
3. Culture Is A Big Deal
An entire conference track along with a keynote was dedicated to the DevOps Culture. No one is really sure what DevOps means but it appears to be almost as vague as cloud and getting almost as much hype. The reason why DevOps and culture is so closely linked to puppet and configuration management is because puppet is a tool to help bridge the cultural divides in your organization. Communicating between people and teams with spoken/written languages such as English is in-precise but communicating how things should be configured though a declarative and executable syntax like puppet manifests is very precise. A puppet manifest puts the developers and operations staff on the same page and can help ensure what QA is testing is actually what will be deployed. Automated configuration management is also really helpful if you want to do frequent deployments into test and production environments.
4. Orchestration is next
Puppet deals with configuring a single server and makes it easy for many servers to be configured in the same way. Often you need to configure multiple servers at the same time in a related way. For example if you are deploying a new version of your software you might need to make some database changes, remove a web-server from the load-balancer, apply the software update to the web-server then add it back into the load balancer pool and repeat for all of your web-servers. Once all of your web-servers have been updated you might need to make another set of database schema changes. Conducting the changes on each of the servers in the right order is called orchestration. Puppetlabs has software called mcollective (and PuppetDB) to help with this but it mostly deals with the RCP aspects of the problem and doesn’t (at least not yet) provide facilities for dealing with errors on one of the servers and rolling back or tools to help with managing dependencies between servers. VMWare claims to have products (vDirector and vSphere Orchastrator) that might do better at this but I didn’t make it to a presentation on them. I expect to see more tooling in this area in the next year.
I will be giving a talk at the PGOpen Conference in Chicago later this month on how Puppet is used to manage PostgreSQL databases that make up key parts of the domain name registry system. Come to PGOpen so you can learn more.
Tags: community, maps, open source, opendata, openstreetmap, osm, SFO, sotm, sotmus
I am sitting on a plane flying eastbound at 37,000 feet over middle of America. A few hours ago we left San Francisco the host city for State Of The Map US 2013. State of the Map US is the American regional conference for OpenStreetMap participants and enthusiasts. A common theme for the weekends events was building community.
Tags: computer-science, education, math, mathematics, science, science eduation, uwaterloo
E.O Wilson recently published an article in the Wall Street Journal where he argues Scientists don’t need math skills. He feels that many potential scientists are being turned away from science because they find math hard. Wilson goes onto argue that scientists in many fields don’t really need much math. I’m not going to address Wilson’s argument directly but instead talk about what I learned about math in school.
Tags: firefox, linux, mozilla, open source, powerpc, ppc, ppc64
I’ve written in the past about compiling recent versions of Firefox for the powerpc under Linux. The Firefox rapid release schedule means that a new firefox release happens almost every 6 weeks. I last wrote about firefox 16, which is now almost 4 releases old. In the past 6 months I’ve been regularly testing firefox builds on the powerpc as they make their way through the development process. I am going to summarize the state of recent firefox versions for ppc users.
Tags: community, hack weekend, open source, openstreetmap, osm, ryerson, software, software design, technology, toronto
This weekend we had the second annual Toronto OpenStreetMap developer weekend. The nice folks at the Ryerson Department of Geography hosted us. My focus this weekend was to work the Serge and Martijn on maproulette
Maproulette is software that presents an easy to do mapping task to users which they can complete and then mark the task as completed. Examples of past maproulette mapping challenges include fixing connectivity errors or fixing objects touched by the license change.
Read the rest of this entry »