Posts Tagged ‘software’

The @Bean annotation is a great way to declare Spring beans when writing code.  This allows the programmer to control which beans get instantiated and with what arguments.   Some situations call for deciding if beans should be instantiated at runtime based on configuration this is often done with the @Conditional annotation on @Configuration classes.   The @Conditional annotation works well if you are enabling or disabling a particular bean through configuration but sometimes the number of beans needs to be completely dynamic.

This often comes up if your application needs to connect to a bunch of external services such as a database.  You want the number of  services or databases that the application connects to be configurable.

(more…)

Advertisements

Spring integration makes it easy to monitor an sftp server for new files and inject those files into your application for processing.  I like to configure my spring integration applications with annotations and @Bean configurations.  I found the documentation, and online examples for doing this annotations a bit lacking.

 

This post demonstrates the basics on how to connect get a spring-boot (1.3.x or 1.4.x) application to monitor a sftp server.

(more…)

Sometimes the behaviour of an application is controlled through properties
and the application needs to detect changes to the property file so it can switch to the new configuration. You also want to ensure that a particular request either uses the old configuration or the new configuration but not a mixture of old and new. Think of this as ACID like isolation for properties to ensure that your requests don’t get processed using an inconsistent configuration.

We accomplish this by using a property lookup bean (PropertyBean) that returns property lookups. The PropertyLookupBean is a request scoped bean meaning that a new instance is created for each request in a spring MVC application. The PropertyLookup bean has an init method that runs post construction and calls the PropertyCache bean to get the current set of properties. Each instance of the PropertyLookup bean will return the property values from the Properties object it gets at initialization.

properties

The property lookup bean has a reference to a singleton scoped PropertyCache. The PropertyCache maintains a reference to the current Properties object. When the PropertyCache bean is asked to return the current properties it checks to see if they have changed on disk and if so reloads the properties.

class PropertyBean {
    private Properties currentProperties;

    @Autowired
    PropertyCache propertyCache;

    @PostConstruct
    void doInit() {
        currentProperties = propertyCache.getProperties();
    }
    public String getProperty(String propertyName) {
        return currentProperties.getProperty(propertyName);
    }
}

 

The PropertyBean would look similar to the above sample class. This class simply caches an instance of a Properties object post-construction and delegates property requests to this instance.

The PropertyBean needs to be created as a request scoped bean. This can be done with a Spring annotation configuration class as follows.

@Configuration
class MyConfig {
    @Bean
    @Scope(value="request", proxyMode=ScopedProxyMode.TARGET_CLASS)
    public PropertyBean propertyBean() {
        return new PropertyBean();
    }
}

 

Managing the loading of properties falls to the PropertyCache class which would look similar to the below class

@Component
public class PropertyCache {
    private Properties currentProperties;
    private Long lastReloadTime=0;
    @Value("${dynamic.property.file.path")
    private String propertyFileName;


    public Properties getProperties()  {
         reloadIfRequiredProperties();   
         synchronized(this) {
             return currentProperties;
         }
    }

    private reloadPropertiesIfRequired() {
         FileInputStream stream = null;
         try {
             File f = new File(propertyFileName);
             if(f.lastModified() > lastReloadTime) {
                 Properties newProperties = new Properties();
                 stream = new FileInputStream(f);
                 newProperties.load(stream);
                 synchronized(this) {
                     lastReloadTime = System.getCurrentMillis();
                     currentProperties = newProperties;
                 }
             } 
          } 
    }
    catch(IOException exp) {
        //do something intelligent on the error
    }
    finally {
        if (stream != null)
            stream.close();
    }
}

 

All of the business logic code that needs properties to process a request will use the request scoped PropertyBean instance bean. This ensures that the entire request uses the same set of properties even if the underlying property file changes.

class SomeBusinessService {
    @Autowired
    private PropertyBean propertyBean;

   public void someOperation() {
      String v = propertyBean.getProperty("some.property");
       .
       .
    }

}

When a new request is started a new instance of the PropertyBean is created. The doInit() method invokes the PropertyCache getProperties() method. The Property Cache checks the timestamp of the file and looks for modifications, if it detects one it reloads the property file. Any PropertyBean instances that have already been initialized will continue to use the old properties instance.

PostgreSQL is becoming a more popular choice for an embedded database because of its BSD license, relatively low memory footprint and great list of features. A few people have asked me if Slony would be a good choice for replication in an embedded environment. Embedded deployments haven’t been a primary use-case for Slony and some of the challenges you would face are worth writing about.

(more…)

Last week I was in Chicago giving a talk at PostgresOpen on managing PostgreSQL with puppet. The talk was well attended and appears to have been well received.

Puppet is configuration management software that allows you to describe how your servers should look using a declarative syntax. You describe what packages you want to install (obviously postgres) and how your configuration files should look. Puppet also allows you to run commands to create databases or database objects such as users.

In my talk I discuss why it is important to use a repeatable procedure for building production database servers and how this is a tool in bridging the divide between developers and operations staff.

I talk about how deploying servers with automation allows your servers to be similar. Similar might not mean identical but the differences between your database servers is controlled and managed. This also applies to your development and QA servers. If you deploy your staging, QA, and development servers using the same puppet manifest as your production servers but with possibly different configuration options then you will be more confident in your testing.

You can view my slides. They recorded the talk and I will update this post with a link to the talk when it is posted.

Updated: You can view a recording of the video below

This weekend we had the second annual Toronto OpenStreetMap developer weekend. The nice folks at the Ryerson Department of Geography hosted us. My focus this weekend was to work the Serge and Martijn on maproulette

Maproulette is software that presents an easy to do mapping task to users which they can complete and then mark the task as completed. Examples of past maproulette mapping challenges include fixing connectivity errors or fixing objects touched by the license change.
(more…)

I found the documentation on urllib2 a bit unclear about how to get cookie handling working properly.

I was working on a python script that needed to contact the OpenStreetMap web server, login with my OSM credentials and interact with the website.

The first step is to setup a urllib2 opener instance that is configured to store cookies.

import cookielib,Cookie,urllib2,urllib
import xml.etree.cElementTree as ElementTree

cookies = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookies))

This will create an opener that can be used to retrieve URL’s. Any cookies set in the HTTP response will be stored in the cookie’s cookie jar. If I needed to add additional openers (ie for special redirect handling) I would just add them as additional parameters to the build_opener call. ie urllib2.build_opener(handler1, handler2,handler3)…

Next we need to contact OpenStreetMap to get a blank login form. The blank login screen has a hidden variable ‘authenticity_token’ that needs to be passed back as part of the POST with my login credentials

inptag = '{http://www.w3.org/1999/xhtml}input'
formtag = '{http://www.w3.org/1999/xhtml}form'
# fetch the blank login form
response_tokenfetch = opener.open(request)
html = response_tokenfetch.read()
htmlfile=StringIO.StringIO(html)
# parse the HTML elements in the form
# extract any input fields for later resubmission
# this will pick up the authenticity_token and anything else
xml_tree = ElementTree.parse(htmlfile)
for form in xml_tree.getiterator(formtag):                
    for field in form.getiterator(inptag):
        if 'name' in field.attrib and 'value' in field.attrib:
                login_payload[field.attrib['name']] = field.attrib['value']
login_payload['username'] = username
login_payload['password'] = password
login_payload['remember_me'] = 'yes'
login_payload['cookie_test'] = 'true'

Next we submit the LOGIN request as a POST. Any session cookies returned as part of the blank form will be added to the second request.

cookies.add_cookie_header(request)
response = opener.open(request,urllib.urlencode(login_payload))

If our login was successful then cookies contains an _osm_session and _osm_username that will be used in subsequent API calls.

request2=urllib2.Request('http://api06.dev.openstreetmap.org/user/stevens/inbox)
cookies.add_cookie_header(request2)
response2 = opener.open(request2)
html=response2.read()

You could then parse the HTML to extract a list of messages.
If your using the formal OpenStreetMap API (ie calls under /api/0.6/…) then you should instead use oauth for authentication instead of logging in through the website. Some OSM features such as messaging can only be accessed by pretending to be a web session and parsing/faking HTML.