Archive for the ‘Uncategorized’ Category

Last week I attended the second Canadian DevOps Days event which was the first to be held in Toronto. DevOps days is a conference dedicated to DevOps. I had a great time and would like to thank the conference organizers and sponsors for making this happen.

DevOps Days was different than most I.T. conferences that I attend because none of the talks were technical in nature. There are lots software products and technologies used in the practice of DevOps such as Puppet, Chef, various CI servers , test frameworks, and logging frameworks but the conference wasn’t really about those. Talks referenced these technologies but that wasn’t the focus of the talks. The key challenges in DevOps today are softer issues. What is How do you go about introducing and promoting DevOps was the key themes of the conference.

One of the most memorable stories from the conference was told during an open-spaces session. The woman telling the story was a manager of a team responsible for some I.T service at a government ministry. She told us how formal initiatives required approvals and budgets that were difficult and time consuming to put in place. Instead of trying to get a “Devops Program” approved or budget to hire “Devops engineers” she took a different approach. Each time one of her reports came to her asking if they could do something such as make a change, fix an issue or propose a solution to a problem she would ask one question.

Will this make us suck less?

If the answer was yes, even if it made them suck just a tiny bit less she often went along with the request. If the answer was no she asked them to rethink the request. This approach allowed her team to make small incremental improvements to the systems they run. Over time their service became much more reliable meaning her team was spending less time fighting fires and had more time to implement bigger improvements within their existing budget. Running a more reliable service also relived some of the pressure on her managers and business partners earning them goodwill. She knew that she couldn’t fix all of the problems with her system in one go and she didn’t sugar coat the reality of their system. The motto wasn’t about being being great, or striving for excellence which would have seemed like a far off goal. The motto was about sucking less something they could archive in the short term while still leaving more room for improvement . The lesson to take away from this story is

Make small incremental improvements that are focused on reducing the pain your co-workers are feeling

Another story from the conference that stood out for me was from Telus. Telus started introducing DevOps practices into various teams a while ago. They started slowly but soon saw dramatic decreases in the number of production issues on products that they had tried DevOps practices on. These success where noticed and helped them roll DevOps practices out to more projects leading to more success. The CEO of Telus (a large Canadian telco) is now giving executive level support and encouragement to the DevOps initiative. The lesson to take away from this story is

Start small but in a position prepared to measure your success. Use those measured success to make a business case for expanding the effort

Firefox 29 was released in early May and contained a re-design of the user-interface called “Australis”. Unfortunately Firefox 29 crashes on startup for 32 bit powerpc users on Linux, and BSD along with most SPARC platforms.

This is because of Bug 961488 which I encountered in early January. I was able to isolate the commit that introduced this issue but figuring out why was more difficult.

It had to do with differences in the calling convention with how C structures are passed to functions. Firefox allows javascript code to call functions that are actually implemented in C++. Different plaforms use different conventions for passing data between functions if the caller and callee don’t agree on how function arguments will be passed then you have problems. The 32 bit ppc platform on Linux and the BSD platforms use a different calling convention than ppc64, AIX and ppc OSX.

The good news is that this bug is now fixed and firefox 30 should work on the powerpc. (note ppc64 builds were uneffected by this issue).

I want to thank Ginn Chen of Oracle, Gustavo Luiz Duarte of IBM for their work in coming up with the fix and Landry Breuil for shepherding the patch though the commit process.

Last week I was hanging out on the top of San Francisco for puppetconf 2013. PuppetConf is the annual conference dedicated to the puppet configuration management system. Some of the lessons I learned are worth sharing.

1. Walking to the top of Nob Hill is less fun that it looks

After landing at SFO I went from the airport to the BART station. Inside the BART station, at the vending machine, I encountered a pair having difficulty figuring out how to buy a ticket. Having been in San Francisco a few months ago forSOTM-US I considered myself competent at operating the BART vending machine so I helped them put money on a ticket. I then rode the train to Powell station. My hotel was at the top of Nob Hill and the cable-car line seemed long so I decided to walk. Walking up a big steep hill with a laptop and luggage on a warmish summer day is a lot of work.

2. Puppet(conf) is more popular than PosgreSQL(conf).

I have been to a lot of PostgreSQL (and other open-source) conferences over the past 7 years and puppetconf with close to 1200 attendees was larger than FOSS4G 2010 and probably 4 times larger than most of the other conferences I’ve recently attended. Puppet and DevOps are hot stuff these days and all the cool kids appear to be doing it. I spoke to numerous people from old school industries like retail and utilities along with people from technology industry giants like salesforce.com, and three letter I.T vendors. They are all starting to use puppet to help with managing their I.T configuration. The main competitor to puppet seems is Chef, another open-source configuration management solution but I didn’t talk to a single person at the conference who could point to a closed source alternative with comparable features that they were using instead of puppet. Puppet is replacing hundreds of poorly written, and misunderstood shell scripts.

3. Culture Is A Big Deal
An entire conference track along with a keynote was dedicated to the DevOps Culture. No one is really sure what DevOps means but it appears to be almost as vague as cloud and getting almost as much hype. The reason why DevOps and culture is so closely linked to puppet and configuration management is because puppet is a tool to help bridge the cultural divides in your organization. Communicating between people and teams with spoken/written languages such as English is in-precise but communicating how things should be configured though a declarative and executable syntax like puppet manifests is very precise. A puppet manifest puts the developers and operations staff on the same page and can help ensure what QA is testing is actually what will be deployed. Automated configuration management is also really helpful if you want to do frequent deployments into test and production environments.

4. Orchestration is next
Puppet deals with configuring a single server and makes it easy for many servers to be configured in the same way. Often you need to configure multiple servers at the same time in a related way. For example if you are deploying a new version of your software you might need to make some database changes, remove a web-server from the load-balancer, apply the software update to the web-server then add it back into the load balancer pool and repeat for all of your web-servers. Once all of your web-servers have been updated you might need to make another set of database schema changes. Conducting the changes on each of the servers in the right order is called orchestration. Puppetlabs has software called mcollective (and PuppetDB) to help with this but it mostly deals with the RCP aspects of the problem and doesn’t (at least not yet) provide facilities for dealing with errors on one of the servers and rolling back or tools to help with managing dependencies between servers. VMWare claims to have products (vDirector and vSphere Orchastrator) that might do better at this but I didn’t make it to a presentation on them. I expect to see more tooling in this area in the next year.

I will be giving a talk at the PGOpen Conference in Chicago later this month on how Puppet is used to manage PostgreSQL databases that make up key parts of the domain name registry system. Come to PGOpen so you can learn more.

symbol

E.O Wilson recently published an article in the Wall Street Journal where he argues Scientists don’t need math skills. He feels that many potential scientists are being turned away from science because they find math hard. Wilson goes onto argue that scientists in many fields don’t really need much math. I’m not going to address Wilson’s argument directly but instead talk about what I learned about math in school.

(more…)

I’ve written in the past about compiling recent versions of Firefox for the powerpc under Linux. The Firefox rapid release schedule means that a new firefox release happens almost every 6 weeks. I last wrote about firefox 16, which is now almost 4 releases old. In the past 6 months I’ve been regularly testing firefox builds on the powerpc as they make their way through the development process. I am going to summarize the state of recent firefox versions for ppc users.

(more…)

A few months ago I wrote a blog post on compiling FireFox for the powerpc on Debian Linux. I had built Firefox 9 when I was writing that post and in the intervening months, thanks to the Mozilla rapid release cycle, Firefox 16 has been released. Does Firefox 16 still build on the powerpc with Debian Squeeze?

  1. I downloaded the latest firefox source code from mozilla
  2. This time I didn’t need to apply any patches for the powerpc.
  3. When I ran ./configure followed by ‘make’ as described in my post on Firefox9 I encountered a compile error.
    syntax error in VERSION script when trying to build libraries in the subdirectories under security/nss/lib such as util (libnssutil).

The issue is that the mapfiles such as security/nss/lib/util/nssutil.def that are included in the source tree are not suitable as input files to the linker but need to be pre-processed first. The rules to process the ‘MAPFILE’ are defined in security/coreconf/Linux.mk and are invoked by a rule in security/coreconf/Rules.mk that build a proper mapfile by running the mapfile distributed in the source tree (such as nssutil.def) through sed and grep to produce a file that can be passed to the linker.

The make rule takes a mapfile $(MAPFILE_SOURCE)=nssutil.def and generates $(MAPFILE)=nssutil.def from it. If you try building firefox as I describe above it will put the output files in the same directory as the input files. This means that the sed/grep rule won’t be executed because the target (nssutil.def) is already present and isn’t older than the dependency (nssutil.def), itself.

If you instead

  1. Set the shell environment variable MOZ_OBJDIR to be a directory other than the source directory
  2. Put your configure options in the file .mozconfig, as described in the Mozilla build instructions
  3. Run make -f client.mk

firefox 16 will build successfully. The issues I encountered with the version script shouldn’t be specific to the powerpc or debian.

CN Freight Train
Software infrastructure is a lot like municipal infrastructure such as highways, subways, and stadiums. Everyone likes to enjoy the benefits but we seem to have difficulties on when to build it, what to build, or how to pay for it.

(more…)