This entry is a part of the ‘What can you build in 40hrs‘ series– which, in a nutshell, involved an experiment to see what we could build in about typical weeks worth of work. In the last post we effectively completed the basic functional requirements of the application, and now have a working system that can be used to manage two sets of to-do lists as part of a task management application.

Cloud Computing

As part of this effort, we wanted to try something ‘new’ as part of the application initiative. We decided to make one of the system requirements of the application involve its deployment into a Cloud Computing infrastructure– the leading candidates at time of construction being Amazon’s Elastic Cloud Computing (EC2) and Google’s App Engine.

While both Amazon and Google offer similar infrastructure platforms, as it shall be seen, the level of effort necessary to deploy a stock Grails application into these infrastructures varies greatly.

This entry concerns itself with the deployment of the TwoDo application into Amazon’s EC2 infrastructure. A subsequent entry will detail the efforts necessary to get the same application running on Google’s AppEngine platform. (At the time I started this effort Google’s AppEngine had only recently released its Java API, and was dribbling out access to its infrastructure in a controlled fashion (FIFO?) As such, I did not initially have the rights to deploy an application to AppEngine– hence the decision to go with EC2 first.)

Cloud Tools Plugin

We are going to leverage the CloudTools Plugin for all the heavy lifting aspects of getting our application deployed into Amazon. Cloud Tools was developed by the founder of CloudFoundry– a service designed to deploy Java/Grails apps to Amazon’s EC2. I suppose I could have gone with the CloudFoundry as part of my deployment efforts (and indeed, will probably try it out) but for whatever reason chose to start at a lower level by using the plugin.

Getting Started

A prerequisite for using the CloudTools Plugin is to have an existing Amazon AWS account. The setup of which is easy, and well documented outside of this blog… so we aren’t going to concern ourselves with much of it here. If nothing else simply go to the main site and click the ‘Sign Up for Amazon EC2′ button

The first basic step was to install the plugin. I downloaded the plugin from the main CloudTools site and installed it from my local drive:

grails install-plugin /path/to/downloads/grails-cloud-tools-0.6.zip

The installation process creates a conf/CloudTools.groovy file which, in theory, could be simply added to the list of configuration files that Grails should slurp during boot time via an explicit specification in the conf/Config.groovy file like so:

...
grails.config.locations = ["classpath:CloudTools.groovy"]
..

However, I could never get this file to be located during the deployment process, so I eventually punted and copied the contents of the conf/CloudTools.groovy file directly into conf/Config.groovy

Once the contents of the conf/CloudTools.groovy file are pasted into the standard conf/Config.groovy file it needs to be edited with the specifics of our application.

Database Setup

The CloudTools plugin will create a MySql schema for us, as well as a user account used to access the DB as part of the deployment process. All we need to to is edit the parameters passed to the schema() method of the ClusterSpec() object that we pasted into our Config.groovy file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

cloudTools {
...
  clusterSpec = new ClusterSpec()
      .tomcats(1)
      .topology("SingleInstanceTopology")
       .instanceType(EC2InstanceType.SMALL)
       .slaves(0)
       .bucketPrefix(System.getProperty("grails.env"))
       .catalinaOptsBuilder("""{builder, databasePrivateDnsName, slaves ->
            builder.arg("-Xmx500m")
	    builder.prop("com.sun.management.jmxremote")
	    builder.prop("com.sun.management.jmxremote.port", 8091)
	    builder.prop("com.sun.management.jmxremote.authenticate", false)
	    builder.prop("com.sun.management.jmxremote.ssl", false)
	    builder.prop("ptrack.application.environment", "ec2")
	    builder.prop("log4j.configuration", "log4j-minimal.properties")
	    builder.prop("jdbc.db.server", databasePrivateDnsName)}""")
        .schema("blogtwodo", ["twodouser": "password"], [])

On line 18 above we configure the ClusterSpec to create the ‘blogtwo‘ database and the ‘twodouser‘, the specifications of which should match the contents of the conf/DataSource.groovy file.

CloudTools AWS Properties File

We need to indicate the location of our Amazon Web Service property file, which is set by editing the value for the awsPropertiesFile property within the conf/Config.groovy file.

...
cloudTools {
  awsPropertiesFile = "etc/ec2.properties"  ...
}

As you can see we have created a new grails-app/etc directory to hold this new file. The properties file is a CloudTools specific artifact that contains the relevant configuration information required by Amazon to host our application. In order to deploy into Amazon’s platform we are going to need provide several bits of information by way of this properties file. Let’s take a look at the contents of ours and dissect it line-by-line:

1
2
3
4
5
6
7
8
9

imageId.m1.small=ami-6f2cc906
imageId.m1.large=ami-0129cc68
imageId.m1.xlarge=ami-0129cc68
accountId=XXXX-XXXX-XXXX
accessKey=XXXXXXXXXXXXXXXXXXX
secretKey=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
keyName=blogtwodo
keyPairFile=/Users/don/DevelopmentWork/blogtwodo/etc/blogtwodo.pem
sshDir=/usr/bin

AMI Identification
An application deployed into the EC2 will reside in Amazon Machine Image (AMI). Think of this as a identifier for the base OS image/VPS. In our case (lines 1-3) we will simply use the ones specified by the plugin documentation:

1
2
3

imageId.m1.small=ami-6f2cc906
imageId.m1.large=ami-0129cc68
imageId.m1.xlarge=ami-0129cc68

Security Credentials
The next few lines contain part of the security credentials we need to provide to Amazon. From within the Amazon Web Services Console, we navigate to ‘Your Account -> Security Credentials‘. The accountId (line 4) is your 12-digit account displayed on this page under the ‘Your AWS Account ID, E-mail Address and Password‘ section.

While within the ‘Access Keys‘ tab, I clicked ‘Create a New access Key‘ and used the resulting generated information to fill in the accessKey (line 5) and secretKey (line 6) properties.

4
5
6

accountId=XXXX-XXXX-XXXX
accessKey=XXXXXXXXXXXXXXXXXXX
secretKey=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Amazon’s site does a good job describing the purpose of the above access credentials:

Access Key ID
Your Access Key ID identifies you as the party responsible for service requests. Use your Access Key ID as the value of the AWSAccessKeyId parameter in requests you send to Amazon Web Services (when required).

Secret Access Key
Each Access Key ID has a Secret Access Key associated with it. Use your Secret Access Key to calculate a signature to include in requests to web services that require authenticated requests. Your Secret Access Key is a secret, and should be known only by you and AWS. You should never include your Secret Access Key in your requests to AWS. You should never e-mail your Secret Access Key to anyone. It is important to keep your Secret Access Key confidential to protect your account.

Key Pair
In addition to the above credentials we need to provide a public/private key pair in the form of an PEM file. Keys are created from within the EC2 dashboard and I created a new keypair with the application name of ‘blogtwodo’ by logging into the ‘Amazon EC2 dashboard -> Key Pairs‘ and clicking the ‘Create Key Pair‘ link. The resulting blogtwodo.pem file was saved locally and moved into the grails-app/etc directory, where its configured by lines 7 and 8 below:

7
8

keyName=blogtwodo
keyPairFile=/Users/don/DevelopmentWork/blogtwodo/etc/blogtwodo.pem

Finally we need to indicate the path to an ssh client (line 9) to be used to securely upload our application to Amazon. I am using OSX on a Mac which includes this natively in the /usr/bin directory:

9

sshDir=/usr/bin

Deployment

OK. With our configuration out of the way lets try and deploy this thing. From the command line we invoke the gant deployment target added by the CloudTools plugin:

grails cloud-tools-deploy

… which promptly reports the following error:

Error executing script CloudToolsDeploy: No such property: DEFAULT_J5_DEPS for class: ExplodedWar_groovy

Whats the deal? Well, the CloudTools plugin was developed pre Grails v1.1.1, as it turns out there are a few incompatibilities that need to be addressed in order to get our application deployed.

This error is addressed by adding the following highlighted code (line 10) to the $GRAILS_HOME/scripts/_GrailsWar.groovy

1
2
3
4
5
6
7
8

...
DEFAULT_DEPS = [
    "ant-*.jar",
     ...
     ...
]
 
DEFAULT_J5_DEPS = DEFAULT_DEPS

OK. Lets try to deploy again….. this time, although we get farther, we eventually fail with the error:

ssh: connect to host ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com port 22: Operation timed out

Drat. Now what? After a little digging it’s determined that the problem is that we have not allowed the SSH protocol through the built in firewall associated with our application. This is fixed by navigating from within the EC Dashboard to ‘Security Groups -> default group’ and adding the SSH protocol to the table of connection methods. Let’s try our deployment again.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/Users/don/DevelopmentWork/blogtwodo/etc/blogtwodo.pem' are too open.
It is recommended that your private key files are NOT accessible by others.
This private key will be ignored.

Sigh. OK. Trying again after chmod‘ing the PEM file to 400 (chmod 400 blogtwodo.pem).

After several minutes (and I do mean several.. deployment took close to 10 minutes) this time we get:

...
...
Location: http://localhost/blogtwodo/ [following]
--10:34:55--  http://localhost/blogtwodo/
Connecting to localhost|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 404 /blogtwodo/
10:34:55 ERROR 404: /blogtwodo/.
 
Finished deploying cluster

Hmm. The ERROR 404 does not look good, but it does look like the application technically deployed. A check on the EC2 console confirms that we have a single running instance, and clicking on it drills down to a panel that shows us the public DNS address we can use to test it out with a browser (running grails cloud-tools-describe gets us the same information from the command line.) When I visit the base address I get a generic Apache test page, which is encouraging:

But, sure enough, I am greeted with a 404 error when I try to hit our applications URL with a browser.

So now what? Well, when in doubt, go for the log files. We can ssh directly into our running instance using our PEM file for authentication:

ssh -i etc/blogtwodo.pem root@ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com

A bit of poking around/exploring our VPS leads me to a promising log file: /var/log/tomcat5/catalina.out with a suspicious stacktrace:

...
java.lang.NoClassDefFoundError: net.chrisrichardson.ec2deployer.cluster.ClusterSpec
        at Config$_run_closure3.class$(Config.groovy)
        at Config$_run_closure3.$get$$class$net$chrisrichardson$ec2deployer$cluster$ClusterSpec(Config.groovy)
        at Config$_run_closure3.doCall(Config.groovy:87)
        at Config$_run_closure3.doCall(Config.groovy)
        at Config.run(Config.groovy:76)
...

It would seem that the CloudTools jars don’t appear to be packaged into our deployment war file. After a bit of investigation, this problem turns out to be another issue caused by behavior changes to Grails v1.1.1– notably in the management of plugins. The CloudTools plugin expects to find stuff located local to the grails project directory and the new Grails v1.1.1 paradigm moves all that stuff to centralized ~/.grails (on a Mac anyway) location. My hack to get things working involved creating a symbolic link so that CloudTools would deploy all the necessary bits (for all our other plugins as well):

ln -s ~/.grails/1.1.1/projects/blogtwodo/plugins ./plugins

OK. Surely that must be the last step. After deploying once again, we again get the 404 error. Now what? Well checking the logfile again reveals this:

org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Access denied for user 'twodouser'@'localhost' (using password: YES))

I don’t doubt that there are better solutuions, but simply I punted and simply forcibly granted our ‘twoduser’ access to the database via the mysql command line:

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 6 to server version: 5.0.22-log
 
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
 
mysql> use blogtwodo
Database changed
mysql> GRANT USAGE ON *.* TO 'twodouser'@'localhost' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.00 sec)
 
mysql> GRANT ALL PRIVILEGES ON `blogtwodo`.* TO 'blogtwodo'@'localhost' WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)

This time we want to make sure that we redeploy the application to the same AMI where we just made our database changes, so we deploy the app using the cloud-tools-redeploy gant target:

grails cloud-tools-redeploy

At last we get something that appears to be a clean deployment:

Connecting to localhost|127.0.0.1|:8080... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: http://localhost:8080/blogtwodo/ [following]
--10:48:48--  http://localhost:8080/blogtwodo/
Connecting to localhost|127.0.0.1|:8080... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: http://localhost:8080/blogtwodo/public [following]
--10:48:49--  http://localhost:8080/blogtwodo/public
Connecting to localhost|127.0.0.1|:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [text/html]
200 OK
 
Finished redeploying cluster

Pointing the browser at the site confirms it as working:

click to enlarge

Awesome.

Conclusion and Wrapup

All told the above took about 2 days (16 hours) to get working. Our total number of hours spent prior to this was about 30 hours, which brings our total to ~46 hours… just over our 40 hour goal. Still not too bad IMO. Let’s survey the pros and cons about the approach we used today:

Pros:

• Integration with EC2 is largely transparent to your underlying Grails application– once you get everything configured correctly that is… after which deployment is a snap
• This transparency is huge. I did nothing, repeat nothing, to alter the underlying application for suitability with EC2. This is NOT the case with Google’s AppEngine, as will be shown in the next post in this series.
• I don’t doubt that using the CloudFoundry management console makes deployment that much easier, without the need of a plugin. I plan to play with this more later, so expect a blog post on this as well.

Cons:

• It’s not free. We deployed the twodo application in an absolutely bare bones configuration– A single AMI which costs US$0.10 per/hour –> US$2.40 per/day –> ~US$72 per/month. While not catastrophically expensive, if you don’t need the ability to infinitely scale on a moments notice, you can certainly find cheaper hosting options.
• We currently have no persistent storage configured. Yes, we have a MySql instance running, but for all practical purposes it might as well be an in memory database. If the AMI ever goes away, our DB is slicked as well. A ‘real’ Cloud Computing solution would require the use of something like Amazon’s Elastic Block Storage Service (EBS) ( which is also not free)
• While it was not a _ton_ of work, it did take some sleuthing to get things finally working. I am assuming that this plugin is likely to no longer be actively supported, as (presumably) the CloudFoundry system effectively replaces it. Not a con exactly, but I think I would investigate what CloudFoundry has to offer before choosing this plugin as my definitive approach to getting an app into the Cloud.

That’s it for now. The next post in the series will document the deploying our application to Google’s AppEngine, so check back often/subscribe!

Technorati Tags: amazon ec2, cloud computing, grails, groovy, java, software development

To recap briefly our progress to date:

• Part 1: We introduce the basic challenge: Build a non-trivial app in about 40hours.
• Part 2: We talk about the basic app concept.. a todo list manager called (cleverly) twodo. We have decided on the general technical framework (Grails, etc)
• Part 3: The first ‘technical’ post. JSecurity setup, initial Domain object created.
• Part 4: The UI is refined, and we start using the GrailsUI plugin to populate a YUI DataTable via AJAX.

So, although this post it titled #5, its really only the third technical post. In the last post we made our UI a bit more pleasing (or at least more intentional), and introduced the GrailsUI plugin into the mix for purposes of using AJAX to dynamically update a YUI DataTable. In this post we will put the finishing touches on the core application, which primarily involves getting the GrailsUI DataTable to work the way we want to.

This post will be a bit on the long side, because I want to finish up the core application so the next couple of posts can deal with deploying the twodo application into a Computing Cloud like Google’s AppEngine or Amazon’s EC2.

DataTable and Cell Editing

In the last post we populated a single simple YUI data table with test data using the GrailsUI, and it did exactly what we asked of it. However, our current implementation produced a simple read-only table. I envisioned using a DataTable construct to not only display the data, but to allow the user to dynamically update it via AJAX– sorta like an online spread sheet type application (in a similar fashion to the YUI example provided here.)

To this end, I initially installed the latest (v1.0.4) version of the plugin as per standard instructions, and it worked fine for our initial tests– however, it did not allow for the editing of the table cells. As it turns out this is a known limitation of the current stable version of the plugin. So now what? Well, working The Google On The Internet Machine revealed that the 1.1-SNAPSHOT has inline cell editing capabilities so I downloaded the 1.1-SNAPSHOT version from the plugin site and installed it from my local drive (grails install-plugin /path/to/download/grails-grails-ui-1.1-SNAPSHOT.zip)

Recall that in the last post we used the gui:datatable tag to construct a simple DataTable like so:

1
2
3
4
5
6
7
8
9

<gui:dataTable id="dt_1"
    columnDefs="[
        [id:'ID', formatter:'text', hidden:true],
        [status:'Status', formatter:'text'],
        [description:'Description', formatter:'text'],
        [est:'Est', formatter:'text'],
    ]"
...
/>

The snapshot version of the tag supports additional editor configuration parameter to be passed along as part of the columnDefs, that allow for the specification of an AJAX target to be called when the cell is edited. Lets take a few baby steps and modify the ’subscription’ column to allow for editing:

1
2
3
4
5
6
7
8
9
10

<gui:dataTable id="dt_1"
    columnDefs="[
      [id:'ID', formatter:'text', hidden:true],
      [status:'Status', formatter:'text'],
      [description:'Description', formatter:'text',
        editor:[controller:'home', action:'tableChange']],      [est:'Est', formatter:'text'],
    ]"
...
/>

Note the additional code on line 6 where we indicate that changes to a cell in the ‘description‘ column are to be handled by making an AJAX call to the ‘tableChange‘ action in the ‘home‘ controller. Let’s point our browser at the page (http://localhost:8080/blogtwodo/home/datatable), and see what happens when we try to edit a cell in the description column.

click to enlarge

OK. As the screen shot above indicates, when we click the cell to edit it, some JavaScript automagically renders a little dialog box for us with some basic editing options. Perfect.

But when we continue and try to actually change the value of the cell and save it by clicking on the ‘Save’ button, we get the following ugly error message:

click to enlarge

Hey! Whats with the error? Well, the DataTable is attempting to make an AJAX call to the tableChange action– which does not exist yet. So let’s remedy that right now:

package com.alterlabs.twodo
class HomeController {
  ...
  /**
    * AJAX Call
    */
  def tableChange = {
 
    // TODO: Verify that this task belong to the current user!
    def task = Task.get(params.id)
 
    if ( params.field == 'description') {
      task.description = params.newValue
      task.save()
    }
    render "table changed"
  }
}

Our initial cut at the tableChange method is about as straight forward as it gets. We load the Task by id (without even checking to see if it belongs to our user! Yikes) and save the new value. Simply reloading our home page after saving verifies that our changes to the description field were indeed saved.

Now that we have the basic approach down, we can use what we have learned as a basis for constructing the real code that will be used to manipulate the two task lists in the application.
We can now add the additional configuration to provide editing for the other columns in the DataTable, as well as corresponding logic in the controller to handle the updates, like so:

1
2
3
4
5
6
7
8
9
10
11
12
13

<gui:dataTable id="dt_1"
  columnDefs="[
    [id:'ID', formatter:'text', hidden:true],
    [status:'Status', formatter:'text',
      editor:[controller:'home', action:'tableChange', 
        type:'dropdown', config:[dropdownOptions:['open','complete','delete'],disableBtns:true]]],    [description:'Description', formatter:'text',
      editor:[controller:'home', action:'tableChange']],
    [est:'Est', formatter:'text',
      editor:[controller:'home', action:'tableChange']],
  ]"
...
/>

Notice that in addition to the basic cell editing, we also specify a bunch of other nifty options to the to the g:datatable control that help customize the behavior of the editor. On line 6 we indicate that the status column is to use a dropdown list for editing (with the list values of ‘open‘, ‘closed‘ and ‘delete‘– we will use the ‘delete’ option later), and that the default save/cancel buttons are to be hidden. In the tableChange action we make the obvious/appropriate corresponding changes to handle the edits to the additional fields.

Up and until now we have been fooling around with the DataTable in a test page. We want the contents of our two task lists to be rendered as the users home page after they successfully log in, so its about time we reincorporate what we have learned back into the main application (views/home/index.gsp) Taking the sample code above and embellishing it a bit with CSS gets us something like this:

click to enlarge

So we have basic editing, but its still lacking a few fundamentals– notably callback notification on success/failure of the edit. We will address that in a bit below…

Adding Rows

So far all of the data we have been manipulating had been manually added as fixture data in the last post. Obviously a task management application needs a way to add new tasks so lets tackle that now.

Normally adding a new Task in a conventional non-AJAX fashion is trivial. (Actually, come to think of it, the standard Grails taglibs make using AJAX pretty trivial as well.)

However, accomplishing the same thing using AJAX and our DataTable, however, proves to be a tad bit trickier. Lets start again with baby steps and (reasonably) add a g:remoteForm at the bottom of our Work tasks list that submits via AJAX to an action in the home controller (once we figure it out for the Work tasks, we can replicate it for the Personal)

<g:formRemote name="worktask_form" onSuccess="alert('Success!')" 
        url="[action:'addRow',params: [ taskType:'work']]">
  <input type="text" name="newTask" id="newTask"/>
  <button type="submit">Add</button>  
</g:formRemote>

We put a simple alert() in the onSuccess event to prove things are working. Exercising our code thru the browser gives us the alert message we expect, but our DataTable is not yet updated to show the new row. If we refresh the page, we see the new row, but that’s hardly sufficient.

The obvious solution is to register a callback function that in turn forcibly refreshes the DataTable. Surprisingly, (at least according to the hours I spent trying to get this aspect of the system to work,) the YUI DataTable possesses no such straightforward function. The GrailsUI DataTable Object does, however, include a requery() function, which gets us 99% of the way there, so as a final resort I coded my own refresh() function based on this and extended the YUI DataTable Object by adding it to the javascript prototype:

1
2
3
4
5
6
7
8
9
10
11
12
13

// Used to refesh our contents after a row change. 
// Basically a copy of requery() with a new Callback
GRAILSUI.DataTable.prototype.refresh =  function() {
...
  // NOTICE: This is what has changed from the requery() function
  var oCallback = {
    success: this.onDataReturnReplaceRows,  //<<- this line    failure: this.onDataReturnSetRows,        
    scope: this,
    argument: newState 
  };
...
};

We can then make our onSuccess callback invoke this new function, and test it out:

 <g:formRemote name="worktask_form" onSuccess="GRAILSUI.dt_1.refresh();" 
         url="[action:'addRow',params: [ taskType:'work']]">

This time, adding a new row refreshed our DataTable and the new row shows up in our list– which is awesome.

Deleting Rows

Now that we can add Tasks to our list, we need to also provide a mechanism for removing them. Recall that we configured our dropdown list for the ‘status‘ column to allow the for three values: ‘open‘, ‘closed‘ and ‘delete‘. The plan is to use a cell edit on the status column to trigger the deletion of the Task. With that in mind I added the following code to the tableChange action (which is the action to be called during our cell edits):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

package com.alterlabs.twodo
class HomeController {
  ...
 
 def tableChange = {
   def profileInstance = Profile.current()
   def task = Task.get(params.id)
   ...
   if ( params.field == 'status') {
     switch (params.newValue) {
       case 'open': 
         task.completionDate = null
         task.save()
         break
       case 'closed': 
         task.completionDate = new Date()
         task.save()
         break
       case 'delete':
         // no easy way to tell what list it was from so remove from both
         profileInstance.removeFromWorkTasks(task)
         profileInstance.removeFromPersonalTasks(task)
         profileInstance.save()
         println "*** Deleted task ${task}"
         break
     } 
   }
...
}

When we change our status column to ‘delete‘ we see a rewarding message in the console, but our DataTable does not update to show any change. We have to refresh the page for the DataTable to show an updated list. Why? For that we move to the next section….

Custom GrailsUI DataTable Changes

One thing that became quickly obvious was that as great as the GrailsUI plugin is, it still lacking in a few areas. Notably there is no mechanism for notifying the UI of a successful cell edit. This is not a big deal when we are simply editing the contents of an existing row (because our browser screen is naturally kept in sync)… but it becomes a definite issue when removing a row from the DataTable.

What we need is the ability to register a callback with underlying YUI structures of a function to be called after a DataTable cell edit. If we view the browser page source and look at the underlying javascript that the g:dataTable tag generates for us we can see what is up:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

... 
var registerEditorListener = function(editor, field, url) {
  GRAILSUI.dt_1.loadingDialog.show();
  var editorCallback = {
    failure: function(o) {      // revert the cell value
      GRAILSUI.dt_1.updateCell(oArgs.editor.getRecord(), field, oArgs.oldData);
      // alert user
      alert('Recieved an error during edit: ' + o.responseText);
     }
   };
   YAHOO.util.Connect.asyncRequest('POST', url, editorCallback, 'id=' + oArgs.editor.getRecord().getData('id') + '&field=' + field + '&newValue=' + oArgs.newData);
                    });
};

From the code above we can see that the taglib creates a registerEditorListener function, and does indeed define a callback to be registered with the AJAX call. BUT it only defines what to do in case of a failure (line 5) and not what to do in cases of success. Knowing this, its easy enough to tweak the grails-app/taglib/DataTableTagLib.groovy file included in the GrailsUI plugin project to allow for us to pass in as part of the g:dataTable configuration a callback to use for cell editing. After tweaking, our augmented taglib generates something like this:

1
2
3
4
5
6
7
8
9

...
var editorCallback = {
  success: function(o) {
    ${editorSuccessCallback};
  }, 
  failure: function(o) {
    ... 
  }
};

…and we simply add the editorSuccessCallback parameter to our g:dataTable configuration data:

1
2
3
4
5

<gui:dataTable id="dt_1"
  ...  
  editorSuccessCallback="GRAILSUI.dt_1.refresh();"
  ..
]"/>

Now anytime we perform any successful cell edit (including deletes) , the DataTable.refresh() function is invoked and our table is rendered accurately. Killer.

Conclusion and Next Steps

Fundamentally the application is now functionally complete. There was plenty of mundane polishing and such that was not documented in this series, but all of the big stuff was touched upon.

The work accomplished in todays posts took a little over 20 hours to complete… the majority of the time split evenly between exploring the GrailsUI TagLib code trying to make the refresh work and tweaking CSS (which always seems to take more time than expected.) That brings our total time spent todate to about 30 hours.

The next post will (finally!) talk about deploying the twodo application into Amazon’s Elastic Compute Cloud (EC2). See you soon…

Technorati Tags: agile, grails, groovy, grailsui, java, software development

In this post we will refine the general look and feel of the User Interface, as well as start building the core list management functionality via a UI Plugin.

Look and Feel

To this point we have done almost no UI creation, and have not customized the default out-of-the-box look and feel of the application. I knew from talking to the stakeholder (in this case my business partner) that he envisioned a simple/clean interface with little clutter. The general site template was envisioned to look something like this:

Under the assumption that the look and feel, would (like all other aspects of the app) continue to evolve over the course of the development iterations, I did not want to spend too much time on polishing the initial cut at the UI template. I started with a simple raw HTML/CSS template, a bit of time in The Gimp and some CSS tweaking via Firebug until I got something that seemed initially acceptable. I added in some greeting text and voila– comparing the original/old login screen:

…and the new one:

OK. Still not award winning perhaps, but at least it looks intentional.

I don’t profess to be much an artist when it comes to web pages, and don’t intend to turn this post into a treatise on web design, but I think its worth peeking at the main views/layout/main.gsp layout file used for this project:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

<html>
   <head>
        <title><g:layoutTitle default="TwoDo" /></title>
        <link rel="stylesheet" href="${createLinkTo(dir:'css',file:'template_css.css')}" type="text/css"/>
        <link rel="stylesheet" href="${createLinkTo(dir:'css',file:'main.css')}" type="text/css"/>
        <link rel="shortcut icon" href="${resource(dir:'images',file:'favicon.ico')}" type="image/x-icon" />
 
        <g:javascript library="application" />
        <g:layoutHead />
  </head>
  <body>
    <g:render template="/layouts/includes/logo"/>
    <div id="page">
      <div id="page-bg">
        <g:render template="/layouts/includes/messages"/>
        <g:render template="/layouts/includes/errors"/>
 
        <!-- start content -->
        <div id="content">
          <g:layoutBody/> 
        </div>
        <!-- end content -->
        <div style="clear:both;">&nbsp;</div>
      </div>
    </div>
    <g:render template="/layouts/includes/footer"/>
 </body>
</html>

The main thing that is worth noting is that for every major rectangular region of the main template (as indicated on the napkin) we have further subdivided the layout into smaller templates which are then rendered from within main layout via the g:render taglib. As an example, we examine a few of the templates below.

Here is the layouts/includes/_logo.gsp template, which has been configured to:

• Show a simple text logo. We will eventually replace this with a graphic
• Display a greeting for the logged in user.
• Provide a logout link

<div id="logo">
  <h1><a href="${resource(dir:'')}">TwoDo!</a></h1>
<jsec:isLoggedIn>
  <span class="greeting" >
    Hello,<jsec:principal/> (<g:link controller="auth" action="signOut">logout</g:link>)</span>
</jsec:isLoggedIn>
</div>

And finally, a look at the layouts/includes/_messages.gsp which includes logic for the rendering of flash messages in a consistent manner:

<div id="message_wrapper" <g:if test="${!flash.message}"> style="display:none;"</g:if>>
  <div id="messageBar" <g:if test="${!flash.message}">style="display:none;"</g:if>>${flash.message}</div>
</div>

This divide-and-conquer strategy makes sense in many things software related, and the case of layout templates is no exception.

Security and Profile Management

It quickly becomes apparent that we require a mechanism for retrieving the current users Profile after a successful login. JSecurity provides a SecurityUtil class with a method that allows us to get the currently logged in Subject. From there we can get the Principle, and from the Principle we can get the Profile. I found it convenient to implement this logic in a pseudo-GORM like fashion by simply adding a static method to the existing Profile class:

1
2
3
4
5
6
7
8
9
10
11
12
13

...
import org.jsecurity.SecurityUtils
 
class Profile extends Base {
  ...
  /**
    * Return the profile associated with the currently logged in user
    */
  static Profile current() {
    def principle = SecurityUtils.getSubject().getPrincipal()
    return Profile.findByUsername( principle )
  }
}

Our controllers/whatever now simply need to call Profile.current() to get the currently logged in user.

GrailsUI Plugin

As per our sitemap in the last post, we only have a handful of pages in the entire application. Once a user logs in, they will be at their home (views/home/index.gsp) page, which will display their current list of tasks side by side. The manipulations of these lists seemed a prime candidate for some fancy UI and AJAXy goodness. In particular I envisioned a dynamic table that would use AJAX to manipulate the contents of the task lists. Rather than roll it all from scratch we will enlist the help of a the Yahoo UI libraries by way of the GrailsUI plugin. I had used the YUI library on prior projects and, wanting to maximize my productivity, so chose it over other competing frameworks (jQuery, etc.)

The GrailsUI plugin basically provides us with a wrapper taglib that encapsulates the underlying YUI javascript. In particular we are interested in the gui:dataTable tag, which wraps a YUI DataTable Control. Installing the GrailsUI Plugin automagically installs its YUI dependencies.

Before we tackle using the taglib/control however, lets look ahead, and address our data testing needs.

From reading the associated documentation, I knew that our GrailsUI DataTable was eventually going be populated by an AJAX call that will return its data in JSON format– which is all well and good. So the first order of business was to create some fixture data with which to test. To this end I added a simple action to the default Task controller we generated in the last post to create a few tasks for purposes of testing:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

package com.alterlabs.twodo
 
class TaskController {
...
  def data = {
    def p = Profile.current()
    for ( i in 0..5 ) {
      p.addToWorkTasks(new Task(description:"Work Task #${i}", estimate:60))
      p.addToPersonalTasks(new Task(description:"Personal Task #${i}", estimate:60))
    } 
    flash.message = p.save()?"Added fake tasks to profile":"Failed to add fake tasks"
    redirect(action:list)
  }
...
}

The code above grabs the Profile of the current user then creates and associates some dummy Tasks with it. Pointing our browser at http://localhost:8080/blogtwodo/task/data invokes the action and allows us to visually verify the data by listing all the Tasks in the system.

OK. Now that we have some data, lets work on making a DataTable retrieve it via AJAX. The plugin home page has some helpful information about using the DataTable, as does the authors blog. As per usual I started with a crude test to work the kinks out, with the assumption that it could be suitably refined later. By following the examples provided in the plugin documentation I quickly put together a simple test page that looked like so:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

<html>
  <head>
    <gui:resources components="['dataTable']" />
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <meta name="layout" content="main" />
    <title>DataTable Test</title>
  </head>
  <body>
    <div>  
      <h1>Work DataTable:</h1> 
      <gui:dataTable id="dt_1"
         columnDefs="[
            [id:'ID', formatter:'text', hidden:true],
            [status:'Status', formatter:'text'],
            [description:'Description', formatter:'text'],
            [est:'Est', formatter:'text'],
         ]"
         sortedBy="id"
         allowExclusiveSort='true'
         draggableColumns="false"
         controller="home" action="tasks" params="[taskType:'work']"
         rowsPerPage="10" 
         paginatorConfig="[
             template:'{PreviousPageLink} {NextPageLink} {CurrentPageReport}',
             pageReportTemplate:'{totalRecords} total records'
         ]"/>
    </div>
  </body>
</html>

In a nutshell, the above code:

1. Includes the resources for the GrailsUI DataTable (line:3)
2. Makes a DataTable with 4 columns (‘id’, ’status’, ‘description’ and ‘est’). (lines:11-17)
3. Note: The ‘id’ column is strictly for sorting purposes and we don’t want it shown on the screen, so we mark it as ‘hidden’. (line:13)
4. Wires the DataTable to populate itself from the action ‘tasks‘ on the ‘home‘ controller and also passes the parameter taskType set to ‘work‘ (line:21)

The other configuration attributes are reasonably self explanatory, or will get additional treatment in later posts. The g:dataTable tag does all the (considerable) heavy lifting for us and imports all the UI javascript files, configures the underlying YUI DataSource and associated response schema, as well as setting up various AJAX callbacks on our behalf. Great stuff.

We have configured our datatable tag to point at a ‘tasks‘ action, so lets take a look at that now. Its pretty basic:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

package com.alterlabs.twodo
import grails.converters.JSON
 
class HomeController {
 ... 
  def tasks = {
    def list = []
    //
    // Get the current user profile
    // 
    def profileInstance = Profile.current()
    def tasks = (params?.taskType == 'work')?profileInstance.workTasks:profileInstance.personalTasks
 
    tasks.each {
       list << [
           id: it.id,
           status: it.completionDate?"done":"front burner",
           est: it.estimate,
           description: it.description,
       ]
    }
    def data = [
      totalRecords: list.size(),
      results: list
    ]
    render data as JSON
  }
  ...
}

The GrailsUI DataTable expects its response schema to be fed to it in JSON format. Our tasks action basically:

1. Gets the current Profile the logged in user (line:11 )
2. Looks at the params to see which list of tasks to work on (line:12 )
3. Iterates over the task list, adding a map of the Task attributes to a list (lines:14-20
4. Create a Map that holds the new list, as well as the total number of records. This is required by the response schema (lines:22-25)
5. Render the Map as JSON back to our view (line:26)

Pointing our browser to http://localhost:8080/blogtwodo/home/datatable should get us the following:

OK, it may not look like much now, but it proves out the underpinnings. We will use it as a starting point, and continue to refine it in later posts.

Next Steps

Alright. That’s about it for this post. To recap we have:

• Established a candidate look and feel for our application, and have broken out layout in to modular templates
• Provided a means to conveniently get the Profile of the currently logged in user.
• Installed the GrailsUI plugin and used it to populate a crude DataTable via AJAX

The above took about 1/2 a day, or 5 hours or so to make happen (most of which was spent wrangling with HTML & CSS,) which puts our running total of ~7 hours spent so far with ~33 remaining in the kitty.

In the next post we will continue to refine the overall UI and get our DataTable control to allow cell editing of our tasks. See you then!

Technorati Tags: agile, grails, groovy, grailsui, java, software development

This longish post will primarily concern itself with some initial setup and design and the inclusion and integration of JSecurity into our application.

The Vision

We know that our application is going to be a simple task management system called ‘twodo‘– the intent of which is to allow the user to balance their work to-do’s with their personal to-do’s.

When my partner (aka Stakeholder) and I were discussing the application, he drew a simple diagram on a napkin that showed two lists presented side-by-side, the leftmost list for work tasks and the rightmost list for personal tasks. This seemed as good of a place to start with regard to general design, and this simple vision drove much of the basic site layout and internal constructs.

After a few minutes of noodling, the following simple high-level sitemap emerged and seemed reasonable:

Basic Setup

I created a new project (grails create-app blogtwodo) and configured my conf/DataSource.groovy to point at a locally running MySql instance with a newly created/empty ‘twodo’ database.

Basic Domain Model

As stated above, the basic function of the app is to manage two sets of task lists– one list for work, and one list for personal tasks. To this end I created a Profile domain class and a Task domain class.

The Profile will represent a registered user within the system, and will contain the account information and some basic configuration settings. The Profile will also hold a reference to the lists of Tasks, one list for each task type.

package com.alterlabs.twodo
 
class Profile extends Base {
  String username
  String email
  String firstName
  String lastName
 
  int  hoursInWorkWeek = 40
  int  hoursInPersonalWeek = 10  
 
  static hasMany = [workTasks:Task, personalTasks:Task]
 
  static constraints = {
    username(unique:true, size:5..15)
    email(email:true,nullable:false,blank:false,unique:true)
    firstName(nullable:true)
    lastName(nullable:true)
  }
}

Similarly the Task class holds the task details, as well as a reference back to its parent Profile:

package com.alterlabs.twodo
 
class Task extends Base {
  String description
  Date completionDate
  int  estimate = 0  // minutes to complete task
 
  static belongsTo = Profile
}

You may have noticed that the Task and Profile Domain classes inherit from a Base class. From past experience I have found it handy to have a simple Abstract Base Class from which all Domain classes inherit. For now we will use this Base class to enable auto-timestamping for our Domain classes:

package com.alterlabs.twodo
 
abstract class Base {
  Date dateCreated  // grails will auto timestamp
  Date lastUpdated  // grails will auto timestamp
 
  static constraints = {
     dateCreated(editable:false)
     lastUpdated(editable:false)
  }
}

I figured the above Domain objects were good enough to start with, and I could worry about constraints, etc. as the design progressed. I used Grails to generate the controllers and scaffolding (grails generate-all), fired up the dev server and verified my efforts via the browser. Navigating to http://localhost:8080/blogtwodo presented me the expected default Grails welcome screen and listed the Controllers for my Task and Profile classes. (These generated CRUD interfaces are handy for initial testing and once the app progresses in functionality we can remove them.)

So far so good.

Security Setup

Our application does not have any exceptional security requirements– we want users to be able to able to start using the system with a minimum of fuss immediately after a simple registration process. We won’t need any roles for this application– users are either anonymous, or registered. I have used the Grails JSecurity Plugin on past projects and knew it would be more than sufficient for our needs.

To get the ball rolling I followed the steps listed in the JSecurity Quick Start Guide to install the plugin and autogenerate some basic underlying constructs.

The quick start procedure creates a bunch of Jsec-prefixed Domain classes and an AuthController into the default package. I like things a bit more organized so I moved both the Domain classes and the Controller into a com.alterlabs.sec package. As a result of moving these classes we also need to add an import statement to the generated grails-app/realms/JsecDbRealm.groovy file (import com.alterlabs.sec.*) so that it can find the various needed JSecurity objects at runtime.

As per the quick start guide I created a simple conf/SecurityFilters.groovy file. The following code requires authorization to access anything other than the (yet to be created) public controller:

import com.at.sec.*
class SecurityFilters {
  def filters = {
  // Ensure that all controllers and actions require an authenticated user,
  // except for the "public" controller
 
    auth(controller: "*", action: "*") {
      before = {
        // Exclude the "public" controller.
        if (controllerName == "public" || controllerName == null) {return true}
 
        // This just means that the user must be authenticated. He does
        // not need any particular role or permission
        accessControl { true }
      }
    }
  }
}

For purposes of initial testing, I also added some temporary code to the BootStrap.groovy file that creates a hardcoded user (user ‘don’, password ‘twodo’). We will remove this code later as the application progresses.

import org.jsecurity.crypto.hash.Sha1Hash
import com.alterlabs.sec.*
 
class BootStrap {
  def init = { servletContext ->
     // TODO: Remove this stuff later
    def role = new JsecRole(name: "User").save()
    def user = new JsecUser(username: "don",
            passwordHash: new Sha1Hash("twodo").toHex()).save()
    new JsecUserRoleRel(user: user, role: role).save()
  }
 
  def destroy = {
   }
}

To test it I fired up the app and was presented the generated list of Controllers, including an additional entry for the newly added com.alterlabs.sec.AuthController. Clicking on the Profile controller presents the Quick Start generated login page, and I can successfully login with the ‘don’ account created during the BootStrapping. Awesome.

Login and Registration UI

Recall that we want to allow for a simple registration process. Once a user has registered he/she should be able to login and start using the application. Right or wrong, I find that many agile based development efforts derive requirements from the imagined user interface, and that is the direction we will take here. I envisioned another napkin with as screen that presents the user with two basic panels, one for new user registration, one for existing users to login:

Anonymous users will only have access to the public space, and once authenticated they will be sent to a home controller, where we eventually hope to add the meat of our application logic and content. With that in mind I next created both a public and home controller (grails create-controller)

I replaced the default view/index.gsp page that Grails provides with a few simple lines that uses the JSecurity taglibs to redirect to either the home controller or the public controller depending on whether the user is authenticted.

${ response.sendRedirect("/blogtwodo/home") }
 
${ response.sendRedirect("/blogtwodo/public") }

Lets create a crude page, as per our napkin above that will allow the user to either register or login. We want this page to be publicly accessible so this was created in views/public/index.gsp:

 
<!--
#login_or_register {margin:10px;width:500px;}
#login_box {border:#006DBA 1px solid;float:left;width:220px;padding: 10px;}
#register_box {border:#006DBA 1px solid;float:right;width:220px;padding:10px;}
-->
<div class="errors"></div>
<h1>Welcome to TwoDo!</h1>
Register now to get started, or login if you have an account
<div id="login_or_register">
<div id="login_box">
 
      <label for="username">Username:</label>
<input id="username" name="username" size="20" type="text" />
      <label for="password">Password:</label>
<input name="password" size="20" type="password" />
      <button>Login</button></div>
<div id="register_box">
 
      <label for="username">Username:</label>
      <span class="value ${hasErrors(bean:profileInstance,field:'username','errors')}">
<input id="username" name="username" size="20" type="text" value="${fieldValue(bean:profileInstance,field:'username')}" />
      </span>
      <label for="email">Email:</label>
      <span class="value ${hasErrors(bean:profileInstance,field:'email','errors')}">
<input id="email" name="email" size="20" type="text" value="${fieldValue(bean:profileInstance,field:'email')}" />
      </span>
      <label for="password">Password:</label>
<input name="password" size="20" type="password" />
      <label for="vpassword">Confirm:</label>
<input name="vpassword" size="20" type="password" />
      <button>Register</button></div>
</div>

Pointing our browser tohttp://localhost:8080/blogtwodo renders the above (thanks to our redirect) as shown below:

OK, so we won’t win any design awards. It’s ugly but it will serve for now– we will worry about the aesthetics later. Basically we have a page with two forms that submit to actions within two different controllers.

The login form is trivial– it’s simply configured to submit to the existing signIn action within the AuthController that was created as part of the JSecurity Quick Start. Once again easily tested via the browser.

The first thing we need to do is provide the association between the JSecurity constructs and our own Profile object. To do this we simply add an association to the JsecUserRoleRel object to our Profile. The JsecUserRoleRel holds the relationship between a JsecUser and a JsecRole and has references to both– which in turn gives our Profile object access to both. (Note: Yeah I know I said we had no need of formal roles in our application, but it seems silly to paint ourselves into a corner.)

After adding this association to the Profile class now looks like this:

package com.alterlabs.twodo
import com.alterlabs.sec.JsecUserRoleRel
 
class Profile extends Base {
  ...
  JsecUserRoleRel  principle
  ...
}

The second form in the above GSP submits to the save action of the public controller for registration. The registration process needs to take care of the following:

1. Create and persist a new JSecurity JsecUser object
2. Create a new Profile object
3. Associate the JSecUser with the Profile and persist
4. Log the new user in
5. Send the authenticated user on to the home page

Our public controller now looks like this:

package com.alterlabs.twodo
import com.alterlabs.sec.*
import org.jsecurity.crypto.hash.Sha1Hash
 
class PublicController {
  def securityService
 
  def index = { }
 
  def save = {
    def profileInstance = new Profile(params)
    def role = JsecRole.findByName("User")
    def ju = new JsecUser(username: profileInstance.username, passwordHash: new Sha1Hash(params.password).toHex()).save()
    profileInstance.principle = new JsecUserRoleRel(user: ju, role: role).save()
 
    if(!profileInstance.hasErrors() && profileInstance.save()) {
      flash.message = "Profile ${profileInstance.id} created"
      // OK, lets log them in and take them to their home page...
      try {
        securityService.authenticate(params.username, params.password)
        redirect(controller:'home',action:'index')
      }
      catch (Exception ex){
        // Authentication failed
        flash.message = message(code: "login.failed")
        def m = [ username: params.username ]
        redirect(action: 'index', params: m)
      }
    }
    else {
      render(view:'index',model:[profileInstance:profileInstance])
    }
  }
}

NOTE: I have encapsulated the authentication logic from the JSecurity Quick Start AuthController into a SecurityService, which is called on line 21 above. (This is left as an exercise to the reader :) )

Once again we can verify our progress by using a browser to explicitly try out the registration process (and in this case I also created a few unit tests.)

Next Steps

OK, not a bad start. At this stage I had spent perhaps 1-2 hours on the effort, with a decent amount to show for it. We have created our core Domain objects, wired in security and have created the self registration process. The next post we will look at designing the View layouts, refining the User Interface, and adding some of the core functionality into the mix.

Technorati Tags: agile, grails, groovy, jsecurity, software development

An Act of Congress to get this info?

Most organizations we’ve worked with have some tool they track issues with: jira, bugzilla, trak, good-old-excel. You name it.  So, in general, the mechanism for collecting issues should not be a big stretch.

On the other hand, the process of being diligent enough to log issues is another matter. From our experience in the Agile setting, those issues which get ‘parked’  by project members during a stand-up/scrum for further discussion are obviously clear candidates for further analysis and tacking. Clearly, during the scrum is not the only time that issues arise. The key point here is to be diligent about tracking issues. That being said, one of the key roadblocks to being diligent is insisting on excessive details about the ‘attributes’ of an issue. Now, there is always the respectable goal of having super fine-grained data that you can report and track progress against, but we all  know this fine-grained reporting rarely happens. Which leads us to …
Issue “Priority”

Many teams  go overboard on trying to assign priority to an issue. ‘Low, Medium, Medium-High, High, Super Duper High, Nuclear’ and variants are not uncommon to see in a priority field. At the end of the day, these designations tell us little-to-nothing about the significance of the issue. In our opinion its more fruitful to use a few meaningful priorities that convey the impact of an issue. For instance: full stop (all work being blocked), progress slowed (work continues, but decisions are needed fast), work continues, impedes/complicates some other function outside of development. Again, from the Agile lens, we care about things that are going to affect progress (i.e. velocity). There’s no need to split hairs by having umpteen variants of “work is being impacted.”  If its in the sprint plan, its important; period.

Issue “Statuses”

Status is another area where teams tend to get lost in trying to convey information that is — well — better left for a status report. From an Agile perspective, an issue is either: impacting this sprint, will impact the next sprint, will impact some other future sprint, is resolved, is reopened. So if an issue has priority of ‘full stop’ with a status of ‘impacting this sprint,’ its logical that it should get a lot of attention because you can be guaranteed that velocity is being affected as you speak.

So, when its all said an done, the attributes associated with an issue should basically boil down to: who’s responsible, which sprint, the open date, the close date, the priority, the status, and description. Anything else that is made a requirement for entry is either ‘nice-to-have’ or is getting in the way of effective issue management.

So, what to measure?

Okay, “step down from the sopabox, Sunjay.” So, what is it that we’re analyzing. Well, there are fancy names (in bold) for these things, and there are common sense (in italics) descriptions that help understand why its important to track. We’ll give you both.

• Issue Resolution Velocity – how fast issues are being closed: what’s the sprint/project average, mean, moving average? What is the open count of high priority issues versus all logged to date?
• Issue Leakage – are issues like zombies — being re-opened?: what’s the sprint/project average, mean, moving average?
• Issue Growth Rate – how is the log growing/shrinking?: how many new? how many outstanding, what’s the ratio of new to open?

So at the end of the day, the ‘issue’ category and the data that comes from analyzing it provide important fodder for the agile retrospective. Its a leading indicator that can be used to assess why velocity is slipping or what might happen with velocity in the future.  If you can, its probably a good idea to add it to your toolkit.

As we continue to work with our partners, competitors, and clients, I am continually stunned by how companies quantify the business value gained from application development and integration work they perform. Typically, its an after-the-fact analysis on profitability/revenue derived from a Line of Business and its applications. On the surface and at the gross level, this makes sense.

However, in our opinion, there is a more fine-grained and informed method for making decisions around what to build.
No organization is perfect — that’s for sure; including us [&& See below]. That being said, we are of the opinion that there is a better way for companies to decide where to spend their application development budgets in close partnership with IT. In many places this negotiation and collaboration between IT and business has been unnecessarily adversarial and characterized by mistrust born from past disappointments from both sides. Business sins include: Feature creep, ill-formed requirements, un-responsive stakeholders, constantly shifting priorities. IT sins include: cost overruns, 80 hour work-weeks, stop-start work efforts, no sense of accomplishment or finality. And its not just the project sponsor and development teams that deal with the fallout. Whether companies “fly the flag” and live by the agile manifesto or are more traditional in their processes, CSR teams, marketing teams, and other operational groups operating in fits-and-starts to support a technology deployment that is ‘a few weeks away’ are still common in the landscape.

This does not have to continue to be the case. There are no silver bullets for the werewolf because, frankly, that’s why consulting companies like ours exist. However, we do think there is a key to making this process more collaborative, informed, fast, and ultimately valuable to the business. Agile Methodology [INSERT APPROACH OF THE DAY] aside, companies still have to decide on what they want to spend time and money. And, the faster that business and IT can enter conversations about the cost and projected return of features that are going to be built the better. Okay, this sounds great, but how does one implement it?
Here is one recipe:

• Ingredient 1: Get high-level requirements quickly and efficiently — for the vast majority of application development efforts, you can and should be able get to a body of requirements that can be used for budgeting and ROI purposes in a few hours … Yes, really — in a handful of meetings and, ultimately, a few hours.

• Ingredient 2: Use a flexible and repeatable framework for estimation that allows you to model ‘what-if’ scenarios in collaboration with business stakeholders quickly and effectively. Moreover, use a framework that allows you to do this on a feature-by-feature basis. In our experience, this is crucial. Why? because Line of Business owners think about the work in terms of trade-offs and options that allow them to drive more money to the top and bottom line. Do these sound familiar: What if I add more people? I don’t really need that feature, what if we remove it? What if the team works 50 hours a week? What if I make that feature less complex?

[&& We were/are/will not be perfect. That's one of the reasons we developed a tool, Planix, to make Ingredient 2 easier to accomplish. This is a shameless self-promotion and yes, there is a free version]

• Ingredient 3: Encourage the business to provide the expected “X” — revenue, profit, widgets, karma points, clams, etc. — that is expected to be generated by the features estimated in (2). Encouragement is key, but at some point, the executives, IT, and the business must insist that projected X be rationalized and estimated in order to justify a capital budget to build the next shiny new application.

• Ingredient 4: Divide Ingredient 3 by Ingredient 2 to determine what your return is going to be from this application/IT investment

• Ingredient 5: Discuss, negotiate, repeat Ingredients 1 through 4 until you have a set of features that business and IT believe are important

Sounds like a lot doesn’t it? It can be a lot in terms of calendar time without the right sponsorship; that is for sure. For our part, we’ve seen the process take 3 months for a 2000 hour, 3 person, 50 feature effort. We’ve also seen it take 5 days for a 21,000 hour, 10-person, 600 feature effort and everything in between.

The important thing, in our opinion, is changing the dialogue … instead of only answering the question “what were the costs to build my application and what did my Line of Business bring in,”we should be collaborating on the question “what features bring the most business value; what will it cost me to gain this value?”

Technorati Tags: estimation, budgeting, ROI, project management, user stories, use cases, agile, ROI

TrueLicense Library Collection(TLC) is a collection of Java packages to securely create, install and verify license for closed source products. TLC is rich in features – License can be perpetual or temparary, bound to users, systems or other entity, free trial period implementation, privacy of license content using password based encryption provided by Java Cryptography Extension (JCE), authenticity of license using digital signature mechanism provided by Java Security API etc.

How to implement License management

What you (Developer) need to do:

1. You already have your web application (say HelloWorld) and you would like to implement License Management. If this is your case, read on.
2. You may want to add a new page for licene management. The client should be able to see the validity of the current license, and instal new license from this page.
3. Write an ascpect to verify license. The pointcut will specify the critical areas of the application. The right approach is to have the licence verification done in every critical functionality of the application. License verification is a ‘cross-cutting’ concern, and hence we can effectively use aspects to weave license verification into our code.
4. Generate private and public key using keytool command.
5. Generate license file. A license file contains encrypted, digitally signed information.
6. Use AspectJ compiler to compile your application.
7. Use a good Code Obfuscator, such as Proguard to guard against Decompilation tools. This is a very important step in real life situtation. If this step is skipped, hackers could exchange your public key and KeyStoreParam implementation with their their own code and bypass the license verification. Since this is a demo app and I’m lazy, this step is skipped. :)
8. Build your application along with the public key.
9. Distribute application and license file to your client. (NEVER give your private key)

What your client needs to do:

1. Pays $$$ and buys your HelloWorld application
2. Fires up the app, navigates to the licenese management page, instals the license file.
3. Uses the app. If the license is expired or not installed, he will not be able to use the application.

Demo Application
The demo application can be used with Tomcat. Download LicenseDemo. Please refer README before using the demo app.

How to use this Demo

1. Unzip HelloWorld-Licensed.zip
2. Run ‘ant clean generateLicense’ from /buildtools folder. This creates a file called sample.lic
3. Run ‘ant clean dist’. This creates HelloWorld-Licensed.war in dist folder.
4. Drop the HelloWorld-Licensed.war file in your webapps folder of Tomcat installation. You are all done!
5. Access the application by http://localhost:8080/HelloWorld-Licensed

First, access the Greetings page without installing license. The error message should be dislayed.
Next, instal sample.lic from the License management page and then access Greetings page. Later, allow the license to expire and access the Greetings page again.

Technorati Tags: java, license, management, aspect, aop, tomcat

Representational State Transfer (REST) is an architectural style for distributed hypermedia systems. REST relies on a single application protocol (HTTP), universal resource indicators (URI) and standardized data formats, through XML. It employs established HTTP methods such as GET and POST to direct applications.

I’m not attempting to explain the concept of REST, there are already enough stuff all over the Internet. You may have noticed that most of the Internet giants offer their services as REST web services. They may be good APIs but not RESTful.

What is not REST?

• APIs that involve Plain Old XML over HTTP are not necessarily REST.

• Using GET for all operations – including insert, updates etc is not REST

Creating a new Person resource:

• Not having distinct URI for first class objects is also a violation of the REST way.

Retrieving a Person resource

A RESTful service has use http methods such as POST, GET, PUT and DELETE to achieve the basic CRUD operations (create, read, update, and delete). All resources are identified using distinct URIs.

Framework in Java

I found Restlet to be a good framework. Restlet ensures that your application can easily be developed in a RESTful manner. Each concept of REST has a corresponding Java interface or class in the Restlet API. Another advantage is the ability to work with multiple protocols (HTTP, JDBC, SMTP, etc.) using the exact same API.

However, I could not find a good example for restlet + spring + tomcat integration, I took the liberty of writing one. You can download the restdemo application here. Just run ant, drop the WAR in your webapps folder – you are all set.

Download RestDemo ZIP

Technorati Tags: rest, web service, java

OK. Lets take a simple example to further illustrate the point at hand. Lets say we have a simple web based form with the following basic fields: name, address, phone, email, and age.

This form will submit to an action which then immediately delegates to a trivial business service.

Lets first define the Business Service in question. It looks like this:

import com.alterthought.train.ui.forms.SimpleForm;

public interface SimpleBizIfc
{
    public void doSomeBusiness(SimpleForm a_form);
}

Thats it. We have a single method that takes a single argument. One thing to note is that for brevity, general laziness and simplification, our business service takes a SimpleForm object, which in turn is derived from a Struts Form bean. THIS IS NOT RECOMMENED FOR REAL-WORLD STUFF. It too closely ties your business interfaces to Struts. I am doing it here for convienence. It should also be noted that many people now favor DynaForms, but once again, I am doing an example here…

Here is the Struts form bean in question:

public class SimpleForm extends ValidatorForm
{
    private String name;
    private String address;
    private String phone;
    private String email;
    private Integer age;

  // Accessors here... snipped for space saving...
    ...
}

The Old Crusty Way

A standard (non-spring) strutsy implementation where this form is validated using the Struts Validator might look as follows:
struts-config.xml file:

<form-beans>
  ...
  <form-bean name="simpleForm" type="com.alterthought.train.ui.forms.SimpleForm"/>
  ...
</form-beans>
...
<action-mappings>
  ...
  <!--  Basic Struts action. No Spring awareness whatsoever. -->
  <action path="/simpleConventionalAction"
      type="com.alterthought.train.ui.actions.SimpleConventionalAction"
      name="simpleForm"
      scope="request"
      validate="true"
      input="/validator.do" >
    <forward name="success" path="/success.jsp" />
  </action>
  ...

Note that we request that our form be validated by setting validate to true. I'm not going to include it here, but we would obviously need the appropriate entry in our validation.xml file that maps to our form name (simpleForm) that indicates what sort of field level validation we wish to perform.

OK, and (finally) now the SimpleConventionalAction class that we are going to submit our form to:

public class SimpleConventionalAction extends Action
{

    public ActionForward execute(
        ActionMapping mapping,
        ActionForm form,
        HttpServletRequest request,
        HttpServletResponse response) throws Exception
    {

      /*
       * Conventional (old-fashioned) way of obtaining handle to business service.
       */
      SimpleBizIfc svc = new SimpleBizImpl();

      SimpleForm sf = (SimpleForm)form;

      svc.doSomeBusiness(sf);
      return mapping.findForward ("success");
    }
}

The above should be pretty digestible; as we are not doing anything out of the ordinary. We instantiate a new Business service by directly constructing it, and then we pass the form bean to the business method in question. No Spring or any fanciness.

The Shiny New Way

OK. Now that we have the old-fangled approach, lets first Spring-ify struts and get to business. There are a bunch of good articles on how to get Struts and Spring running together, and as such I am not going spend alot of time aspect of this effort. For more info try here or here.

Our solution will use Spring to wire up our Struts Action implementation to a ProxyBean which can transparently intercept our method calls and perform validation. The Validator integration itself is courtesy of the SpringModules project, and this is what our interceptor will use to perform the service side validation.

First add the following to the appropriate section in your web.xml file:

...
<!-- Spring params -->
<context-param>
  <param-name>contextConfigLocation</param-name>
  <param-value>classpath:/applicationContext.xml </param-value>
</context-param>

<listener>
  <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
</listener>
...

And then we need the applicationContext.xml file that is referenced in the contextConfigLocation parameter (this will live in your wars WEB-INF/classes dir). Rather then give you the whole file at once, I will present excerpts of it inline, so we can talk about the various sections.

Lets start with the Struts action itself. We will add a new entry to the struts-config.xml file that delegate the Action implemention to Spring.

<action path="/spring_betterSpringAction"
    type="org.springframework.web.struts.DelegatingActionProxy"
    name="simpleForm"
    scope="request"
    validate="true"
    input="/validator2.do" >
  <forward name="success" path="/success.jsp" />
</action>

And now lets look at the relevant entries to our applicationContext.xml file:

<bean name="/spring_betterSpringAction"
    class="com.alterthought.train.ui.actions.BetterSpringAction">
  <property name="svc">
      <ref bean="validatingBizSvc"/>
  </property>
</bean>

There are a couple of interesting things about the above entrys:

1. We set our action type to be a Spring DelegatingActionProxy. This is the glue between the Struts controller and Spring.
2. We use the simple (and aribtrary) naming convention of preappending 'spring_' to the action path to reenforce in our example that this Action is delegated to Spring.
3. The path defined for the action in the struts-config.xml file needs to exactly match the corresponding Spring bean name entry in the applicationContext.xml file. Notice I said name and not the more typical bean id. This is because the slash character (/) is illegal in the id field.
4. The Spring bean itself can be seen to use a validatingBizSvc bean, which we will talk about shortly.

And here is the straight forward implementation of our business service (the only difference between this one and the initial ConventionalSpringAction is that we use Spring/IoC to wire a handle to our service delegate):

public class BetterSpringAction extends Action
{
    private AnotherSimpleIfc svc;

    public ActionForward execute(
        ActionMapping mapping,
        ActionForm form,
        HttpServletRequest request,
        HttpServletResponse response) throws Exception
    {

        SimpleIBizIfc svc = getSvc();

        SimpleForm sf = (SimpleForm)form;
        svc.doSomeBusiness(sf);
        return mapping.findForward ("success");
    }

    public AnotherSimpleIfc getSvc()  {   return svc;}

    public void setSvc(AnotherSimpleIfc svc) { this.svc = svc;}
 }

Now lets take a look at the validatingBizSvc bean entry from the applicationContext.xml file:

<bean id="validatingBizSvc" class="org.springframework.aop.framework.ProxyFactoryBean">
  <property name="target">
    <ref local="validatingBizSvcTarget" />
  </property>
  <property name="proxyInterfaces">
    <value>com.alterthought.train.biz.SimpleBizIfc</value>
  </property>
  <property name="interceptorNames">
    <list>
      <value>validationInterceptor</value>
    </list>
  </property>
</bean>

<bean id="validatingBizSvcTarget" class="com.alterthought.train.biz.ValidatingBizImpl"/>

<bean id="validationInterceptor" class="com.alterthought.train.sys.ValidationInterceptor">
  <property name="validator" ref="beanValidator"/>
</bean>

The validatingBizSvc bean is in fact a Proxy that targets the validatingBizSvcTarget bean and implements the underlying SimpleBizIfc interface, and uses the bean validationInterceptor as its sole interceptor. Huh? Say what? Come again? What we are doing is using Spring to dynamically introduce a Proxy that masquerades as our Business Service. We can then set up an interceptor stack (among other things) that allows us to introspect the method invocation before (and after) we ever even call the real business service implementation. Remember the client asked for transparent validation. Using a method interceptor we can inject our own logic with the business service being none the wiser. We will scutinize the implementation of the ValidatingInterceptor in a bit.

The last remaining bits are the beans used by our interceptor for the validation itself. Once again, back to the applicationContext.xml file:

<bean id="validatorFactory"
    class="org.springmodules.validation.commons.DefaultValidatorFactory">
  <property name="validationConfigLocations">
    <list>
      <value>/WEB-INF/spring-validator-rules.xml</value>
      <value>/WEB-INF/validation.xml</value>
    </list>
  </property>
</bean>

<bean id="beanValidator"
    class="org.springmodules.validation.commons.DefaultBeanValidator">
  <property name="validatorFactory" ref="validatorFactory"/>
</bean>

Some further explaination on the above:

1. The beanValidator bean uses the SpringModules project to integrate with the Commons Validator.
2. We tell the validatorFactory where to find its configuration data and we feed it two files
3. MUY IMPORTANTE: The first file contains the validator rules themselves and is NOT the same as the standard Struts validator-rules.xml file. It has been altered to use the SpringModules stuff. You can get this file here
4. The second file has the form validation mappings and IS the exact same file that the Struts Validator uses. This is how we reuse our validation rules between the presentation and the business tiers.

The only thing left is to check out the implementation of our ValidationInterceptor:

import org.aopalliance.intercept.MethodInterceptor;
import org.aopalliance.intercept.MethodInvocation;
import org.apache.struts.action.ActionForm;
import org.springframework.validation.BindException;
import org.springmodules.validation.commons.AbstractBeanValidator;

public class ValidationInterceptor implements MethodInterceptor
{

    private AbstractBeanValidator validator;

    /**
     * Interceptor to validate our form. This is a quick and dirty impl.. there
     * are numerous issues with it, not the least of which is the tight coupling
     * between Struts ActionForms and a Business Interceptor.
     *
     */
    public Object invoke(MethodInvocation invocation) throws Throwable
    {
        Object args[] = invocation.getArguments();

        /**
         * OK. This is obviously fairly brittle and niave, for example
         * the form name is hardcoded here... etc.
         */
        for (int i = 0; i < args.length; i++) {
            Object object = args[i];

            if ( object instanceof ActionForm ) {
                ActionForm form = (ActionForm)object;
                BindException errors = new BindException(form, "simpleForm");
                validator.validate(form, errors);
                if(errors.hasErrors()) {
                    System.err.println("Validation failed");
                    throw new RuntimeException(errors);
                }
            }
        }
        return invocation.proceed();
    }

    public AbstractBeanValidator getValidator() { return validator;}
    public void setValidator(AbstractBeanValidator validator) { this.validator = validator;}

}

The ValidationInterceptor.invoke() is called before our actual business service method... thereby allowing us to perform some additional logic as we see fit. In this case we simply iterate over the passed arguments looking for an ActionForm. If it finds one it submits it to the validator. Assuming there are no errors, we allow the method invocation to continue by calling proceed()

Wrap Up

So to recap, we use delegate our Struts Action to a Spring enabled implementation, which in turn uses a Proxy to intercept the business method calls. Our interceptor transparently performs Validation on the passed data using the same Validation configuration that was used by the Presentation tier. Viola!

Technorati Tags: java, spring, struts, validator

So I was the unfortunate victim a harddrive hardware failure. Repeated WindowsXP CHKDSK runs did nothing to fix the problem and I came to the painful realization that I needed a new drive. Bad sectors you see. I contacted Dell, and since my laptop is under warranty, they shipped me an identical 30g drive.

OK, so now the dilemma: How to best recover? My critical data was backed up and current, but the thought of reinstalling 2 years worth of applications, development tools, setting, etc. brought me to tears.

Enter dd. The Un*x/Linux dd (‘direct dump’? idunno) command can take raw data dumps from one file to another, and given that all devices in Unix are represented as files, from device to device. To put it another way, it is basically a lowlevel copy utility capable of backing up entire devices.

Here are the steps I followed to recover my system:

1. Formated the new drive (A) using Windows install disks. This low-level formatting remaps bad sectors and prepares the disk.
2. Installed drive A in my laptop as the internal drive.
3. Configured my laptop BIOS to boot from CDROM.
4. Installed the old/busted drive(B) into an external USB case, and plugged the USB drive into a free USB port on the laptop.
5. Booted the laptop with a Linux installation CDROM Disk 1 (RedHat WS, but it does not really matter which distribution/release)
6. At the ‘what do you want install prompt’, I ran ‘linux rescue’ to load a mini kernel into RAM; which drops me into a shell.
7. Run the following command
   

   dd if=/dev/sda1 of=/dev/hda1 bs=1024 conv=noerror

8. Came back the next day and rebooted my laptop in to Windows (woo-hoo!), logged in, and _immediately_ ran the full CHKDSK on the drive.
9. kaPOW! We are in business.

A few notes on the above command invocation.

• The basic dd form is dd if={infile} of={outfile} bs={blocksize}
• In Linux, devices are accessible as files.
• Our infile (if) in this case is drive A, which mapped to the first IDE device of /dev/hda1
• Our outfile (of) in this case is drive B, the external USB drive. Linux distros map USB storage devices as SCSI devices, which is why we use /dev/sda1 to access the USB drive.
• We specify a blocksize (bs) of 1024. I am a little unclear here as to what repesents an optimal blocksize. I errored on the size of small.
• The conv=noerror option tells dd to ignore any I/O errors. This is critical, since we know we will have some, after all, our drive is shot.
• Here is link to the man page for dd.

That’s it! Good luck!

Technorati Tags: linux, dd, drive recovery, corrupted hardrive