E.O Wilson recently published an article in the Wall Street Journal where he argues Scientists don’t need math skills. He feels that many potential scientists are being turned away from science because they find math hard. Wilson goes onto argue that scientists in many fields don’t really need much math. I’m not going to address Wilson’s argument directly but instead talk about what I learned about math in school.

There is a degree hanging on my mother’s wall with my name on it that says “Bachelor of Mathematics in Computer Science”. In high school I didn’t particularly enjoy math and I can’t say I was all that good at it. I wanted to go into computers and Waterloo had a reputation for having one of the best computer science programs in the country. At the time getting a computer science degree from Waterloo meant enrolling in the faculty of mathematics and getting a bachelor of math. I understand you can now do computer science at Waterloo without getting a B. Math but that wasn’t the case in my day.

Getting into any respected computer science program required good marks in high school math. High math marks weren’t coming naturally to me so I spent long hard hours working on getting acceptable math marks. Lesson one: If you get put in long hours learning something your not good at, and don’t enjoy, you can often get good enough at it. Other people learn this playing a musical instrument or through sports but the important point is learning how effort and determination can often make up for lack of talent. Even though you might never get as good as someone who is naturally talented you can often get good enough. Hard problems often require hard work and it is important to learn this.

The University of Waterloo Computer Science B.Math program gave students a broad exposure to most branches of mathematics in addition to a much deeper treatment of computer science. The math courses I took in my first two years were the same math courses any other (pure math, applied math etc..) math major took. As a first year math major I discovered that I didn’t know very much about math. First year mathies at Waterloo took a classical algebra course where we were taught how to prove things with mathematical rigour. By the end of the first week I was totally confused. High school math for me was about the mechanical manipulation of numbers we weren’t taught how to prove things. I would spend much of the next four years writing mathematical proofs. One of the early proofs in the classical algebra course was to show when a linear Diophantine equation had a solution. Until I dug-out my old textbook ago I had forgotten what a Diophantine equation was but looking at the textbook I can now clearly see what we were actually being taught in first year. The point of wasn’t to learn how to solve a Diophantine equation it was to learn how to go about proving things. Lesson Two: To show something mathematically you break the problem down into small pieces and solve each component. You build up your solution by first proving the parts (theorems and lemmas) that you need for your overall solution.

One of the reasons why teaching logical reasoning through mathematics works so well is because math is a very precise language. Lesson Three: When solving a problem, using precise language is important. Precise language allows me to be very specific about what I am talking about which allows me to say things that are verifiable. I can get away with saying almost anything using general non-specific language because I can phrase things with such generalizations such that I am not really saying anything at all. The language of generalizations works great for politicians and blog authors but doesn’t actually contribute much to coming up with real solutions to scientific problems.

A common type of question on my math assignments and exams was to show that a particular mathematical statement was true. In this type of question you need to provide an answer that walks the reader from a starting point to a known conclusion. I wasn’t being asked to find the answer to a question but to explain why the answer they have given me is true. Sometimes I couldn’t figure out all of the steps to go from statement A to statement B but I knew it was possible and required since I could work backwards from the result I was trying to show. One approach I tried (more than once) was to wave my hands and say A=B (as if by magic) and move on with the rest of the solution. The markers almost always called me out on this and put “Why” in big red letters next to a -1. I knew A was equal to B and the markers knew A was equal to B but knowing something is true doesn’t count for much in math if you can’t explain why it is true. Lesson four: If you don’t know the reasons behind something and you try to fake or hand-wave your way through you are going to be called out on it even if your conclusion is correct. This lesson is very important for scientists because science is about explaining the reasons behind the things we observe in the real world. Science isn’t about calculating how long a ball dropped from a 100m high building will take to hit the ground, science is about explaining why it takes 4.5 seconds.

These four lessons aren’t the only things I learned about math while getting my math degree but they are some of the most useful lessons I learned and are well worth the time and effort for anyone considering a career in science. Some scientific fields don’t require advanced calculus but many do and most require at least basic statistics. All scientific fields require the disciplined rigour that I learned in advanced math courses. Science isn’t about coming up with a creative story explaining the world that sounds good at cocktail parties it is about doing careful work to collect and analyze data that validates, or invalidates a theory. E.O Wilson’s students might be finding math hard but doing good science is also hard and our scientists of tomorrow shouldn’t be afraid of some hard work.

Firefox 19
Firefox 19 is not a bad choice for 32 bit powerpc users, a number of Linux distributions have ppc packages for this release. Powerpc users of FF19 will find that many images show up with a blue tint. This is due to bug 817356 You can avoid this bug by setting image.high_quality_downscaling.enabled to false in about:config. If your building firefox yourself you can also apply my patch from bug 817356 before compiling to fix this issue.

Firefox 20
Firefox 20 has the blue tint issue fixed and should work fine on 32 bit powerpc. Firefox 20 will be released in the next week I hope the community will find it works well but there is always a chance issues will be found when distributions package it.

Firefox 21
Firefox 21 goes into beta next week and I have been using development builds of ff21 on my 32 bit PPC machine during the past month without issues. This version includes a bit of additional altivec acceleration (817153) support.

When I build firefox 21 on my ppc32 machine I get ./../dist/bin/libxul.so: undefined reference to `JSContext::updateMallocCounter(unsigned int) errors. I have reported these under bug 846986 and have attached a patch that allows me to get a working firefox. I don’t understand why I am hitting this issue with undefined inlines and on others are not.

64 bit PPC
Running firefox on a 64 bit PPC machine will be more problematic firefox 19 crashes on the first web page you try to view. This is documented in bug 844883. The patch I attached to that bug fixes the issue and will be included in Firefox 22. The patch should apply cleanly against versions 19 through 21. Debug builds will also need the patch from 851859 to avoid a startup time assertion. With these two patches I find that firefox is still crashing on the second or third page. I suspect this is related to the problem being addressed in 840242 but as of now this is still an unresolved issue and I don’t have a a good solution for ppc64 users.

Compiling Firefox 21 or higher requires Python 2.7. Much of my testing was done on a Debian squeeze installation and Debian squeeze doesn’t include Python 2.7. I built my version of python 2.7 (installed in (/usr/local/python2.7) to allow me to compile firefox. The firefox build system will install all of the required python packages in a virtualenv.

Updated: I’ve added the note on 846986 since originally publishing this article.

Maproulette is software that presents an easy to do mapping task to users which they can complete and then mark the task as completed. Examples of past maproulette mapping challenges include fixing connectivity errors or fixing objects touched by the license change.

Serge and Martijn wanted to redesign parts of maproulette to make it easier develop more challenge modules. I am going to discuss some of the design decisions we debated during the weekend.

Challenges
A Challenge is a type of mapping activity that map roulette can assign to users. Examples of challenges might be ‘Add bridges to roads with where we suspect a bridge exists’ or ‘Add the street address tags to restaurants that don’t have addresses’.

Tasks
A task is a specific work item within a challenge. In the ‘Add bridges to roads’ challenge a task might be to add a bridge to a specific intersection (or determine that bridge should be there)

Maproulette needs to provide a web based interface that displays challenge tasks to users(task assignments). This requires us to somehow store information about the tasks that make up a challenge. Challenge specific modules will compute tasks (ie find missing bridges). Figuring out how the web interface and the programs that find challenge tasks should be connected was one of the questions we discussed this weekend.

Proposal 1: Discrete Modules

Each challenge has a challenge module that supports operations such as

• Give me some available tasks near this lat/lon
• Mark this task as assigned to a user
• Mark this task as complete/skipped

Each challenge module would be responsible for storing task data in it’s own
format (or using database tables specific to the module). The user’s web-browser would talk to a common front-end component that would invoke the individual challenge modules. The challenge modules would return HTML/Javascript/JSON suitable that the browser would use in displaying information about the task to the user.

Proposal 2: Shared Task Representation

Challenge modules compute a list of tasks by examining a mixture of OSM and external data sources. The challenge modules then convert these tasks into a common representation made up of a task and associated sets of task_objects. A task to find add a missing bridge might be made up of three objects. An object for the place we think the bridge belongs, an object for the road the bridge is part of and a third object for the road/stream that the bridge crosses over. These tasks and associated objects get stored in a database. Tasks and associated objects for different challenges get stored in the same pair of database tables. The web application that users interact with can then display and manipulate information about tasks from all challenges.

Contrasting the options

When designing software systems you often have two or more different approaches to a given problem. Deciding which option to proceed with is the challenge software designers have to face. Some of the questions I asked myself this weekend included

• How can code between different challenges be shared or re-used in each of the options? . The second approach sounds like it has more places where code can be re-used because all challenges use the same data model for assigning tasks
• Is one approach more flexible than the other? Are there things different challenges might deal with differently that will be hard to accommodate with one of the designs? Our concern with the second proposal was that we were generalizing all challenges to object + geometry and not much else. What if some challenges needed to display extra things such as textual hints?
• How do the proposals make it easier or harder for someone to come along and develop a new challenge? In the first approach a challenge developer can has free reign to implement things in any fashion they like as long as they implement the task assignment API I discuss above. In the second proposal challenge authors can still pick their programming language and methods for detecting challenges but they must use the database schema we provide for storing information about tasks. With the second approach challenge developers only need to worry about writing programs to find challenge tasks and store them in the database. Task assignment/management is handled by the common code

A third proposal

We were still debating the merits of these different approaches on late-morning on Sunday when we decided to take a break and go for lunch. One would think that a group of mappers would know lots of places nearby for lunch in downtown Toronto but we seemed to be wondering aimlessly back and forth down the street. While we were doing this we came up with the idea of a shared database schema approach like proposal 2 but letting each challenge store a blob of JSON with the task. Each challenge would also implement challenge specific rendering/display could that would know how to take the JSON describing the task and display information about the task to the user in a meaning full way.

I want to thank Richard Weait for organizing this weekend and Serge and Martijn for letting me participate in the design process. Next we need to finish implementing what we discussed. This should help us figure out if our ideas were good ideas.

I was working on a python script that needed to contact the OpenStreetMap web server, login with my OSM credentials and interact with the website.

The first step is to setup a urllib2 opener instance that is configured to store cookies.

import cookielib,Cookie,urllib2,urllib
import xml.etree.cElementTree as ElementTree

cookies = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookies))

This will create an opener that can be used to retrieve URL’s. Any cookies set in the HTTP response will be stored in the cookie’s cookie jar. If I needed to add additional openers (ie for special redirect handling) I would just add them as additional parameters to the build_opener call. ie urllib2.build_opener(handler1, handler2,handler3)…

Next we need to contact OpenStreetMap to get a blank login form. The blank login screen has a hidden variable ‘authenticity_token’ that needs to be passed back as part of the POST with my login credentials

inptag = '{http://www.w3.org/1999/xhtml}input'
formtag = '{http://www.w3.org/1999/xhtml}form'
# fetch the blank login form
response_tokenfetch = opener.open(request)
html = response_tokenfetch.read()
htmlfile=StringIO.StringIO(html)
# parse the HTML elements in the form
# extract any input fields for later resubmission
# this will pick up the authenticity_token and anything else
xml_tree = ElementTree.parse(htmlfile)
for form in xml_tree.getiterator(formtag):                
    for field in form.getiterator(inptag):
        if 'name' in field.attrib and 'value' in field.attrib:
                login_payload[field.attrib['name']] = field.attrib['value']
login_payload['username'] = username
login_payload['password'] = password
login_payload['remember_me'] = 'yes'
login_payload['cookie_test'] = 'true'

Next we submit the LOGIN request as a POST. Any session cookies returned as part of the blank form will be added to the second request.

cookies.add_cookie_header(request)
response = opener.open(request,urllib.urlencode(login_payload))

If our login was successful then cookies contains an _osm_session and _osm_username that will be used in subsequent API calls.

request2=urllib2.Request('http://api06.dev.openstreetmap.org/user/stevens/inbox)
cookies.add_cookie_header(request2)
response2 = opener.open(request2)
html=response2.read()

You could then parse the HTML to extract a list of messages.
If your using the formal OpenStreetMap API (ie calls under /api/0.6/…) then you should instead use oauth for authentication instead of logging in through the website. Some OSM features such as messaging can only be accessed by pretending to be a web session and parsing/faking HTML.

One of my favourite parts of local/regional conferences held on weekends is that they tend to attract attendees who are passionate about computers and technology. The people who I spoke with at the conference were there because they wanted to be there,not because there boss wanted them to be there, and either loved Python or wanted to learn more about it. I’ve attended many great PostgreSQL conferences over the past few years but it was nice to spend sometime talking with people from broader development backgrounds.

In my discussions with people at the conference I noticed a trend. People I spoke with who are working at companies that did Python development tended to be using PostgreSQL. The ones that weren’t currently using PostgreSQL were using MySQL and talking about moving to PostgreSQL or were apologetic for still being on MySQL. The MySQL users were often apologizing before I told them that I was a PostgreSQL contributor. Some of the MySQL users also mentioned that they were using non-Oracle forks like Percona.

This was in contrast to the people at the Python conference that described their workplaces as doing primarily Java development. The Java development shops tended to be using Oracle or SQL Server. I admit that the sample size of of the Java developers wasn’t that big (this was a Python conference after all) but my observations are worth keeping in mind since they might be indicating a pattern. Other people have commented about the popularity of PostgreSQL in the Ruby community.

I wonder how much of this observations is because older written in Java are already using SQL Server/Oracle and there hasn’t been a strong enough driver to change to PostgreSQL. While newer software projects are tending to choose Python or Ruby over Java and at the same time picking a FLOSS database such as PostgreSQL where they don’t have to worry about migrating a legacy application.

My talk on writing stored functions in Pl/Python was well received. A lot of people saw appeal in being able to write their stored functions in Python instead of pl/SQL but that shouldn’t be a surprise considering this was a Python conference.

My slides are available here, the video of the talk is posted at pyvideo

1. I downloaded the latest firefox source code from mozilla
2. This time I didn’t need to apply any patches for the powerpc.
3. When I ran ./configure followed by ‘make’ as described in my post on Firefox9 I encountered a compile error.
   syntax error in VERSION script when trying to build libraries in the subdirectories under security/nss/lib such as util (libnssutil).

The issue is that the mapfiles such as security/nss/lib/util/nssutil.def that are included in the source tree are not suitable as input files to the linker but need to be pre-processed first. The rules to process the ‘MAPFILE’ are defined in security/coreconf/Linux.mk and are invoked by a rule in security/coreconf/Rules.mk that build a proper mapfile by running the mapfile distributed in the source tree (such as nssutil.def) through sed and grep to produce a file that can be passed to the linker.

The make rule takes a mapfile $(MAPFILE_SOURCE)=nssutil.def and generates $(MAPFILE)=nssutil.def from it. If you try building firefox as I describe above it will put the output files in the same directory as the input files. This means that the sed/grep rule won’t be executed because the target (nssutil.def) is already present and isn’t older than the dependency (nssutil.def), itself.

If you instead

1. Set the shell environment variable MOZ_OBJDIR to be a directory other than the source directory
2. Put your configure options in the file .mozconfig, as described in the Mozilla build instructions
3. Run make -f client.mk

firefox 16 will build successfully. The issues I encountered with the version script shouldn’t be specific to the powerpc or debian.

In the corporate world when an event, company or product falls on its face the detailed reasons are hushed up and rarely get a public airing. When the government screws up sometimes we get an expensive public inquiry where lawyers argue about how someone else did or didn’t screw up. OSGEO has a tradition of building consensus and discussing non-technical issues on their mailing lists. Good open-source projects not only be give us amazing software but they allow us to learn valuable lessons on public relations, finances and management by letting us watch their inner-workings.

When I was learning too fly airplanes I made a point of regularly reading the investigation reports of plane crashes. A common theme was that a single mistake didn’t cause the accident but a chain of mistakes, malfunctions and bad luck all added up to the crash. Common thinking in the aviation community is that if you learn about the mistakes others have made then you will be more likely to avoid making the same mistakes. The story Cameron tells is about limited options, ignoring red flags, communication break downs and bad luck. These are all issues we see everyday across all aspects of life.

The mailing list discussion about the lessons that can be learned for future FOSS4G events is still on-going but as outsiders we can follow their discussion and try to learn from their mistakes. When our events, initiatives or products don’t succeed we should also conduct a carefully look at how we got to that state and what things, with 20/20 hindsight, we should have paid more attention too.

I like to think of software infrastructure as the software components you use that you aren’t in the core business of providing or selling. If your running a cloud hosting company then the virtualization and provisioning software are the product your selling. The software you use for processing credit card payments and emailing customers are infrastructure components you use. If your business is providing credit card payment services to websites then the software that does this is part of your core product but the virtualization software you use is part of your infrastructure.

Facilities for logging, message queues, automated deployment and configuration, monitoring builds and unit tests, tracking performance are all common examples of software infrastructure. This isn’t a post about build versus buy, this is is a post about figuring out when you need components, how good they need to be and how you should go about building and integrating them.

Infrastructure components like municipal infrastructure are usually used by different teams in your organization for different purposes. The individual communities along an expressway often can’t justify building a 6 line highway through their community but can justify it once they partner with neighboring towns to build a connected road. Software infrastructure works the same way. When multiple departments or business units start to connect with shared infrastructure benefits show up. Shared infrastructure components allow different departments to share data, expertise and costs.

However not all infrastructure will help accomplish this. One of my favourite episodes of The Simpsons is the monorail episode. The town of Springfield comes into some money and have a townhall to decide how to spend it. The townsfolk buy into a song and dance routine and build a mono-rail. The monorail is good for some laughs but not much else and it results in a trainwreck. This frequently happens with software where a slick sales demo promises a solution to cure all your problems convincing most people to go for snappy tunes instead of thinking things through.

So when do you build a piece of infrastructure? Many agile techniques including scrum say that you should only allocate resources for infrastructure items when you need them to meet your immediate business needs. The agile approach is to only build enough infrastructure to meet your needs in this sprint. You don’t pave the highway if the items in this sprint can get by with a dirt road.

One of the problems with this approach is that it gives other departments very little incentive to collaborate or share your infrastructure. It is easier for the team next door to build their own dirt road then to negotiate access rights and a route diversion on your dirt road. If you had built 4 lane paved road then it might be worth their while to work with you and help fund the ongoing maintenance but you didn’t build that paved road.

A good option is to work with the next town to upgrade the dirt road to a paved road when they show interest. With this approach you might build a lightweight logging infrastructure that meets your immediate needs. You let other departments know that you’ve done this and when they have some additional logging needs you invite them in to improve the logging infrastructure you wrote. Your neighbour turns your chicken-wire fence into a wood one to keep kids from wondering into his yard. This approach allows you to avoid expensive upfront infrastructure costs and temptations to build mono-rails while still being able to collaborate on shared infrastructure.

In the open-source world this happens all the time. One open-source project will submit patches ‘upstream’ to another open-source project to make the software more useful for themselves or to avoid having to duplicate something that another project has written. Open-source project developers tend to be good collabarators (otherwise they wouldn’t have an open-source project they will just have released their code under an open source license). The tools we use in open-source (git, public mailing lists with archives, open bug trackers, build instructions) all make this type of activity easier.

In the corporate world things are rarely this simple. Politics, juvenile ideas of code ownership, security policies, and cost accounting often come into play to discourage this type of collaboration. Centrally planned infrastructure components can avoid this problem, just like a good way to build a railway is to have the federal government drive the process. This might work if your upper management cares enough about a particular piece of infrastructure but, more often than not, this doesn’t happen because infrastructure components are not what your in the business of selling. Building centrally planned infrastructure is also not keeping with agile principals and requires a big up-front investment. A lot of the time agile development teams won’t know exactly what their requirements are for components far enough in advance to allow them to give meaningful requirements to a long term centrally planned project.

Another approach that can work is to have a team dedicated to building (or purchasing) software infrastructure components and working with other teams in integrating these components. This allows an infrastructure team to see the requirements of multiple teams and find common solutions. The infrastructure team would have a budget and mandate to build infrastructure. The challenge is to manage just-in-time demands from different teams in providing the components in a useable manner that meets the agile time-lines of the teams that need the infrastructure. A dedicated infrastructure team can also work on the infrastructure backlog by finding deficient infrastructure components that multiple products lack and then upgrading and integrating the improved components into multiple products.

Don’t settle for a spaghetti network of dirt roads or mono-rail train-wrecks. Learn about the infrastructure that other projects and departments in your company have built (or have bought/downloaded and integrated) and find a way that you can expand on it instead of building or buying your own. As your team builds components identify which ones fall into the infrastructure category and decouple them from your projects core code so that other teams can easily reuse these components when the need comes up.

The presentation covered

• Common reasons people render their own maps
• Where to get OpenStreetMap data and how to load it into your PostGIS database
• How to use Tilemill to design your own map style
• How to render map tiles, both statically and dynamically
• How to use OpenLayers to display your map

The presentation was recorded. I will update this post when the recording comes online.

I’ve been a regular attendee of PGCON since the first year it was held in Ottawa. Like past years I enjoyed the conference and I would like to thank Dan Langille for putting together another first-rate conference. My favourite part of PostgreSQL conferences is meeting and reconnecting with users people in the community.

Updated: The video is available here

Peninsula Lake, Huntsville Ontario

I have published a new version of the Peninsula Lake map. This edition includes new cartography and map updates based on my survey work last summer.

I had written the previous map style in Cascadnik, a CSS like language that translates to mapnik XML. Cascadnik has since been replaced by Carto so I have decided to rewrite the map style using TileMill (Map design software that uses Carto and Mapnik).

One of my goals while redesigning the map was to put to use what I have learned about cartography over the past year. The original PenLake map style was based on a sample Cascadnik style adapted to my colour scheme, nautical features and a print scale. The new style was designed by me from scratch to focus on

• The lake and nearby rivers, streams and marshes
• Recreational attractions around the lake including hotels and golf courses
• Roads near the lake
• The water depth of the lake including suspected rocks

This style is much smaller than the previous one ( the style is less than 300 lines of CSS) and it behaves well at different zoom levels between 12 and 18. I suspect the number of layers can still be further reduced for faster rendering times.

I have made a web version of the map available on Peninsula Lake Map my map hosting site. The map tiles are currently being served by MapBox.

Some of the water depth data in this version of the map is based on observations I manually made while kayaking on the lake in the summer of 2011. I have tried to correct this data to match the sounding datum used by the CHS in their observations. I don’t expect the observations taken by me with a kayak, rope and a tape measure to match the accuracy of the CHS with their fancy sonar but we have to work with the equipment that we have available to us. The depths on the map should be based on a datum of 913 feet.

The Source code is still available from github