This entry is a part of the ‘What can you build in 40hrs‘ series– which, in a nutshell, involved an experiment to see what we could build in about typical weeks worth of work. In the last post we effectively completed the basic functional requirements of the application, and now have a working system that can be used to manage two sets of to-do lists as part of a task management application.

Cloud Computing

As part of this effort, we wanted to try something ‘new’ as part of the application initiative. We decided to make one of the system requirements of the application involve its deployment into a Cloud Computing infrastructure– the leading candidates at time of construction being Amazon’s Elastic Cloud Computing (EC2) and Google’s App Engine.

While both Amazon and Google offer similar infrastructure platforms, as it shall be seen, the level of effort necessary to deploy a stock Grails application into these infrastructures varies greatly.

This entry concerns itself with the deployment of the TwoDo application into Amazon’s EC2 infrastructure. A subsequent entry will detail the efforts necessary to get the same application running on Google’s AppEngine platform. (At the time I started this effort Google’s AppEngine had only recently released its Java API, and was dribbling out access to its infrastructure in a controlled fashion (FIFO?) As such, I did not initially have the rights to deploy an application to AppEngine– hence the decision to go with EC2 first.)

Cloud Tools Plugin

We are going to leverage the CloudTools Plugin for all the heavy lifting aspects of getting our application deployed into Amazon. Cloud Tools was developed by the founder of CloudFoundry– a service designed to deploy Java/Grails apps to Amazon’s EC2. I suppose I could have gone with the CloudFoundry as part of my deployment efforts (and indeed, will probably try it out) but for whatever reason chose to start at a lower level by using the plugin.

Getting Started

A prerequisite for using the CloudTools Plugin is to have an existing Amazon AWS account. The setup of which is easy, and well documented outside of this blog… so we aren’t going to concern ourselves with much of it here. If nothing else simply go to the main site and click the ‘Sign Up for Amazon EC2′ button

The first basic step was to install the plugin. I downloaded the plugin from the main CloudTools site and installed it from my local drive:

grails install-plugin /path/to/downloads/grails-cloud-tools-0.6.zip

The installation process creates a conf/CloudTools.groovy file which, in theory, could be simply added to the list of configuration files that Grails should slurp during boot time via an explicit specification in the conf/Config.groovy file like so:

...
grails.config.locations = ["classpath:CloudTools.groovy"]
..

However, I could never get this file to be located during the deployment process, so I eventually punted and copied the contents of the conf/CloudTools.groovy file directly into conf/Config.groovy

Once the contents of the conf/CloudTools.groovy file are pasted into the standard conf/Config.groovy file it needs to be edited with the specifics of our application.

Database Setup

The CloudTools plugin will create a MySql schema for us, as well as a user account used to access the DB as part of the deployment process. All we need to to is edit the parameters passed to the schema() method of the ClusterSpec() object that we pasted into our Config.groovy file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

cloudTools {
...
  clusterSpec = new ClusterSpec()
      .tomcats(1)
      .topology("SingleInstanceTopology")
       .instanceType(EC2InstanceType.SMALL)
       .slaves(0)
       .bucketPrefix(System.getProperty("grails.env"))
       .catalinaOptsBuilder("""{builder, databasePrivateDnsName, slaves ->
            builder.arg("-Xmx500m")
	    builder.prop("com.sun.management.jmxremote")
	    builder.prop("com.sun.management.jmxremote.port", 8091)
	    builder.prop("com.sun.management.jmxremote.authenticate", false)
	    builder.prop("com.sun.management.jmxremote.ssl", false)
	    builder.prop("ptrack.application.environment", "ec2")
	    builder.prop("log4j.configuration", "log4j-minimal.properties")
	    builder.prop("jdbc.db.server", databasePrivateDnsName)}""")
        .schema("blogtwodo", ["twodouser": "password"], [])

On line 18 above we configure the ClusterSpec to create the ‘blogtwo‘ database and the ‘twodouser‘, the specifications of which should match the contents of the conf/DataSource.groovy file.

CloudTools AWS Properties File

We need to indicate the location of our Amazon Web Service property file, which is set by editing the value for the awsPropertiesFile property within the conf/Config.groovy file.

...
cloudTools {
  awsPropertiesFile = "etc/ec2.properties"  ...
}

As you can see we have created a new grails-app/etc directory to hold this new file. The properties file is a CloudTools specific artifact that contains the relevant configuration information required by Amazon to host our application. In order to deploy into Amazon’s platform we are going to need provide several bits of information by way of this properties file. Let’s take a look at the contents of ours and dissect it line-by-line:

1
2
3
4
5
6
7
8
9

imageId.m1.small=ami-6f2cc906
imageId.m1.large=ami-0129cc68
imageId.m1.xlarge=ami-0129cc68
accountId=XXXX-XXXX-XXXX
accessKey=XXXXXXXXXXXXXXXXXXX
secretKey=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
keyName=blogtwodo
keyPairFile=/Users/don/DevelopmentWork/blogtwodo/etc/blogtwodo.pem
sshDir=/usr/bin

AMI Identification
An application deployed into the EC2 will reside in Amazon Machine Image (AMI). Think of this as a identifier for the base OS image/VPS. In our case (lines 1-3) we will simply use the ones specified by the plugin documentation:

1
2
3

imageId.m1.small=ami-6f2cc906
imageId.m1.large=ami-0129cc68
imageId.m1.xlarge=ami-0129cc68

Security Credentials
The next few lines contain part of the security credentials we need to provide to Amazon. From within the Amazon Web Services Console, we navigate to ‘Your Account -> Security Credentials‘. The accountId (line 4) is your 12-digit account displayed on this page under the ‘Your AWS Account ID, E-mail Address and Password‘ section.

While within the ‘Access Keys‘ tab, I clicked ‘Create a New access Key‘ and used the resulting generated information to fill in the accessKey (line 5) and secretKey (line 6) properties.

4
5
6

accountId=XXXX-XXXX-XXXX
accessKey=XXXXXXXXXXXXXXXXXXX
secretKey=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Amazon’s site does a good job describing the purpose of the above access credentials:

Access Key ID
Your Access Key ID identifies you as the party responsible for service requests. Use your Access Key ID as the value of the AWSAccessKeyId parameter in requests you send to Amazon Web Services (when required).

Secret Access Key
Each Access Key ID has a Secret Access Key associated with it. Use your Secret Access Key to calculate a signature to include in requests to web services that require authenticated requests. Your Secret Access Key is a secret, and should be known only by you and AWS. You should never include your Secret Access Key in your requests to AWS. You should never e-mail your Secret Access Key to anyone. It is important to keep your Secret Access Key confidential to protect your account.

Key Pair
In addition to the above credentials we need to provide a public/private key pair in the form of an PEM file. Keys are created from within the EC2 dashboard and I created a new keypair with the application name of ‘blogtwodo’ by logging into the ‘Amazon EC2 dashboard -> Key Pairs‘ and clicking the ‘Create Key Pair‘ link. The resulting blogtwodo.pem file was saved locally and moved into the grails-app/etc directory, where its configured by lines 7 and 8 below:

7
8

keyName=blogtwodo
keyPairFile=/Users/don/DevelopmentWork/blogtwodo/etc/blogtwodo.pem

Finally we need to indicate the path to an ssh client (line 9) to be used to securely upload our application to Amazon. I am using OSX on a Mac which includes this natively in the /usr/bin directory:

9

sshDir=/usr/bin

Deployment

OK. With our configuration out of the way lets try and deploy this thing. From the command line we invoke the gant deployment target added by the CloudTools plugin:

grails cloud-tools-deploy

… which promptly reports the following error:

Error executing script CloudToolsDeploy: No such property: DEFAULT_J5_DEPS for class: ExplodedWar_groovy

Whats the deal? Well, the CloudTools plugin was developed pre Grails v1.1.1, as it turns out there are a few incompatibilities that need to be addressed in order to get our application deployed.

This error is addressed by adding the following highlighted code (line 10) to the $GRAILS_HOME/scripts/_GrailsWar.groovy

1
2
3
4
5
6
7
8

...
DEFAULT_DEPS = [
    "ant-*.jar",
     ...
     ...
]
 
DEFAULT_J5_DEPS = DEFAULT_DEPS

OK. Lets try to deploy again….. this time, although we get farther, we eventually fail with the error:

ssh: connect to host ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com port 22: Operation timed out

Drat. Now what? After a little digging it’s determined that the problem is that we have not allowed the SSH protocol through the built in firewall associated with our application. This is fixed by navigating from within the EC Dashboard to ‘Security Groups -> default group’ and adding the SSH protocol to the table of connection methods. Let’s try our deployment again.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/Users/don/DevelopmentWork/blogtwodo/etc/blogtwodo.pem' are too open.
It is recommended that your private key files are NOT accessible by others.
This private key will be ignored.

Sigh. OK. Trying again after chmod‘ing the PEM file to 400 (chmod 400 blogtwodo.pem).

After several minutes (and I do mean several.. deployment took close to 10 minutes) this time we get:

...
...
Location: http://localhost/blogtwodo/ [following]
--10:34:55--  http://localhost/blogtwodo/
Connecting to localhost|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 404 /blogtwodo/
10:34:55 ERROR 404: /blogtwodo/.
 
Finished deploying cluster

Hmm. The ERROR 404 does not look good, but it does look like the application technically deployed. A check on the EC2 console confirms that we have a single running instance, and clicking on it drills down to a panel that shows us the public DNS address we can use to test it out with a browser (running grails cloud-tools-describe gets us the same information from the command line.) When I visit the base address I get a generic Apache test page, which is encouraging:

But, sure enough, I am greeted with a 404 error when I try to hit our applications URL with a browser.

So now what? Well, when in doubt, go for the log files. We can ssh directly into our running instance using our PEM file for authentication:

ssh -i etc/blogtwodo.pem root@ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com

A bit of poking around/exploring our VPS leads me to a promising log file: /var/log/tomcat5/catalina.out with a suspicious stacktrace:

...
java.lang.NoClassDefFoundError: net.chrisrichardson.ec2deployer.cluster.ClusterSpec
        at Config$_run_closure3.class$(Config.groovy)
        at Config$_run_closure3.$get$$class$net$chrisrichardson$ec2deployer$cluster$ClusterSpec(Config.groovy)
        at Config$_run_closure3.doCall(Config.groovy:87)
        at Config$_run_closure3.doCall(Config.groovy)
        at Config.run(Config.groovy:76)
...

It would seem that the CloudTools jars don’t appear to be packaged into our deployment war file. After a bit of investigation, this problem turns out to be another issue caused by behavior changes to Grails v1.1.1– notably in the management of plugins. The CloudTools plugin expects to find stuff located local to the grails project directory and the new Grails v1.1.1 paradigm moves all that stuff to centralized ~/.grails (on a Mac anyway) location. My hack to get things working involved creating a symbolic link so that CloudTools would deploy all the necessary bits (for all our other plugins as well):

ln -s ~/.grails/1.1.1/projects/blogtwodo/plugins ./plugins

OK. Surely that must be the last step. After deploying once again, we again get the 404 error. Now what? Well checking the logfile again reveals this:

org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Access denied for user 'twodouser'@'localhost' (using password: YES))

I don’t doubt that there are better solutuions, but simply I punted and simply forcibly granted our ‘twoduser’ access to the database via the mysql command line:

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 6 to server version: 5.0.22-log
 
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
 
mysql> use blogtwodo
Database changed
mysql> GRANT USAGE ON *.* TO 'twodouser'@'localhost' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.00 sec)
 
mysql> GRANT ALL PRIVILEGES ON `blogtwodo`.* TO 'blogtwodo'@'localhost' WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)

This time we want to make sure that we redeploy the application to the same AMI where we just made our database changes, so we deploy the app using the cloud-tools-redeploy gant target:

grails cloud-tools-redeploy

At last we get something that appears to be a clean deployment:

Connecting to localhost|127.0.0.1|:8080... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: http://localhost:8080/blogtwodo/ [following]
--10:48:48--  http://localhost:8080/blogtwodo/
Connecting to localhost|127.0.0.1|:8080... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: http://localhost:8080/blogtwodo/public [following]
--10:48:49--  http://localhost:8080/blogtwodo/public
Connecting to localhost|127.0.0.1|:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [text/html]
200 OK
 
Finished redeploying cluster

Pointing the browser at the site confirms it as working:

click to enlarge

Awesome.

Conclusion and Wrapup

All told the above took about 2 days (16 hours) to get working. Our total number of hours spent prior to this was about 30 hours, which brings our total to ~46 hours… just over our 40 hour goal. Still not too bad IMO. Let’s survey the pros and cons about the approach we used today:

Pros:

• Integration with EC2 is largely transparent to your underlying Grails application– once you get everything configured correctly that is… after which deployment is a snap
• This transparency is huge. I did nothing, repeat nothing, to alter the underlying application for suitability with EC2. This is NOT the case with Google’s AppEngine, as will be shown in the next post in this series.
• I don’t doubt that using the CloudFoundry management console makes deployment that much easier, without the need of a plugin. I plan to play with this more later, so expect a blog post on this as well.

Cons:

• It’s not free. We deployed the twodo application in an absolutely bare bones configuration– A single AMI which costs US$0.10 per/hour –> US$2.40 per/day –> ~US$72 per/month. While not catastrophically expensive, if you don’t need the ability to infinitely scale on a moments notice, you can certainly find cheaper hosting options.
• We currently have no persistent storage configured. Yes, we have a MySql instance running, but for all practical purposes it might as well be an in memory database. If the AMI ever goes away, our DB is slicked as well. A ‘real’ Cloud Computing solution would require the use of something like Amazon’s Elastic Block Storage Service (EBS) ( which is also not free)
• While it was not a _ton_ of work, it did take some sleuthing to get things finally working. I am assuming that this plugin is likely to no longer be actively supported, as (presumably) the CloudFoundry system effectively replaces it. Not a con exactly, but I think I would investigate what CloudFoundry has to offer before choosing this plugin as my definitive approach to getting an app into the Cloud.

That’s it for now. The next post in the series will document the deploying our application to Google’s AppEngine, so check back often/subscribe!

Technorati Tags: amazon ec2, cloud computing, grails, groovy, java, software development

To recap briefly our progress to date:

• Part 1: We introduce the basic challenge: Build a non-trivial app in about 40hours.
• Part 2: We talk about the basic app concept.. a todo list manager called (cleverly) twodo. We have decided on the general technical framework (Grails, etc)
• Part 3: The first ‘technical’ post. JSecurity setup, initial Domain object created.
• Part 4: The UI is refined, and we start using the GrailsUI plugin to populate a YUI DataTable via AJAX.

So, although this post it titled #5, its really only the third technical post. In the last post we made our UI a bit more pleasing (or at least more intentional), and introduced the GrailsUI plugin into the mix for purposes of using AJAX to dynamically update a YUI DataTable. In this post we will put the finishing touches on the core application, which primarily involves getting the GrailsUI DataTable to work the way we want to.

This post will be a bit on the long side, because I want to finish up the core application so the next couple of posts can deal with deploying the twodo application into a Computing Cloud like Google’s AppEngine or Amazon’s EC2.

DataTable and Cell Editing

In the last post we populated a single simple YUI data table with test data using the GrailsUI, and it did exactly what we asked of it. However, our current implementation produced a simple read-only table. I envisioned using a DataTable construct to not only display the data, but to allow the user to dynamically update it via AJAX– sorta like an online spread sheet type application (in a similar fashion to the YUI example provided here.)

To this end, I initially installed the latest (v1.0.4) version of the plugin as per standard instructions, and it worked fine for our initial tests– however, it did not allow for the editing of the table cells. As it turns out this is a known limitation of the current stable version of the plugin. So now what? Well, working The Google On The Internet Machine revealed that the 1.1-SNAPSHOT has inline cell editing capabilities so I downloaded the 1.1-SNAPSHOT version from the plugin site and installed it from my local drive (grails install-plugin /path/to/download/grails-grails-ui-1.1-SNAPSHOT.zip)

Recall that in the last post we used the gui:datatable tag to construct a simple DataTable like so:

1
2
3
4
5
6
7
8
9

<gui:dataTable id="dt_1"
    columnDefs="[
        [id:'ID', formatter:'text', hidden:true],
        [status:'Status', formatter:'text'],
        [description:'Description', formatter:'text'],
        [est:'Est', formatter:'text'],
    ]"
...
/>

The snapshot version of the tag supports additional editor configuration parameter to be passed along as part of the columnDefs, that allow for the specification of an AJAX target to be called when the cell is edited. Lets take a few baby steps and modify the ’subscription’ column to allow for editing:

1
2
3
4
5
6
7
8
9
10

<gui:dataTable id="dt_1"
    columnDefs="[
      [id:'ID', formatter:'text', hidden:true],
      [status:'Status', formatter:'text'],
      [description:'Description', formatter:'text',
        editor:[controller:'home', action:'tableChange']],      [est:'Est', formatter:'text'],
    ]"
...
/>

Note the additional code on line 6 where we indicate that changes to a cell in the ‘description‘ column are to be handled by making an AJAX call to the ‘tableChange‘ action in the ‘home‘ controller. Let’s point our browser at the page (http://localhost:8080/blogtwodo/home/datatable), and see what happens when we try to edit a cell in the description column.

click to enlarge

OK. As the screen shot above indicates, when we click the cell to edit it, some JavaScript automagically renders a little dialog box for us with some basic editing options. Perfect.

But when we continue and try to actually change the value of the cell and save it by clicking on the ‘Save’ button, we get the following ugly error message:

click to enlarge

Hey! Whats with the error? Well, the DataTable is attempting to make an AJAX call to the tableChange action– which does not exist yet. So let’s remedy that right now:

package com.alterlabs.twodo
class HomeController {
  ...
  /**
    * AJAX Call
    */
  def tableChange = {
 
    // TODO: Verify that this task belong to the current user!
    def task = Task.get(params.id)
 
    if ( params.field == 'description') {
      task.description = params.newValue
      task.save()
    }
    render "table changed"
  }
}

Our initial cut at the tableChange method is about as straight forward as it gets. We load the Task by id (without even checking to see if it belongs to our user! Yikes) and save the new value. Simply reloading our home page after saving verifies that our changes to the description field were indeed saved.

Now that we have the basic approach down, we can use what we have learned as a basis for constructing the real code that will be used to manipulate the two task lists in the application.
We can now add the additional configuration to provide editing for the other columns in the DataTable, as well as corresponding logic in the controller to handle the updates, like so:

1
2
3
4
5
6
7
8
9
10
11
12
13

<gui:dataTable id="dt_1"
  columnDefs="[
    [id:'ID', formatter:'text', hidden:true],
    [status:'Status', formatter:'text',
      editor:[controller:'home', action:'tableChange', 
        type:'dropdown', config:[dropdownOptions:['open','complete','delete'],disableBtns:true]]],    [description:'Description', formatter:'text',
      editor:[controller:'home', action:'tableChange']],
    [est:'Est', formatter:'text',
      editor:[controller:'home', action:'tableChange']],
  ]"
...
/>

Notice that in addition to the basic cell editing, we also specify a bunch of other nifty options to the to the g:datatable control that help customize the behavior of the editor. On line 6 we indicate that the status column is to use a dropdown list for editing (with the list values of ‘open‘, ‘closed‘ and ‘delete‘– we will use the ‘delete’ option later), and that the default save/cancel buttons are to be hidden. In the tableChange action we make the obvious/appropriate corresponding changes to handle the edits to the additional fields.

Up and until now we have been fooling around with the DataTable in a test page. We want the contents of our two task lists to be rendered as the users home page after they successfully log in, so its about time we reincorporate what we have learned back into the main application (views/home/index.gsp) Taking the sample code above and embellishing it a bit with CSS gets us something like this:

click to enlarge

So we have basic editing, but its still lacking a few fundamentals– notably callback notification on success/failure of the edit. We will address that in a bit below…

Adding Rows

So far all of the data we have been manipulating had been manually added as fixture data in the last post. Obviously a task management application needs a way to add new tasks so lets tackle that now.

Normally adding a new Task in a conventional non-AJAX fashion is trivial. (Actually, come to think of it, the standard Grails taglibs make using AJAX pretty trivial as well.)

However, accomplishing the same thing using AJAX and our DataTable, however, proves to be a tad bit trickier. Lets start again with baby steps and (reasonably) add a g:remoteForm at the bottom of our Work tasks list that submits via AJAX to an action in the home controller (once we figure it out for the Work tasks, we can replicate it for the Personal)

<g:formRemote name="worktask_form" onSuccess="alert('Success!')" 
        url="[action:'addRow',params: [ taskType:'work']]">
  <input type="text" name="newTask" id="newTask"/>
  <button type="submit">Add</button>  
</g:formRemote>

We put a simple alert() in the onSuccess event to prove things are working. Exercising our code thru the browser gives us the alert message we expect, but our DataTable is not yet updated to show the new row. If we refresh the page, we see the new row, but that’s hardly sufficient.

The obvious solution is to register a callback function that in turn forcibly refreshes the DataTable. Surprisingly, (at least according to the hours I spent trying to get this aspect of the system to work,) the YUI DataTable possesses no such straightforward function. The GrailsUI DataTable Object does, however, include a requery() function, which gets us 99% of the way there, so as a final resort I coded my own refresh() function based on this and extended the YUI DataTable Object by adding it to the javascript prototype:

1
2
3
4
5
6
7
8
9
10
11
12
13

// Used to refesh our contents after a row change. 
// Basically a copy of requery() with a new Callback
GRAILSUI.DataTable.prototype.refresh =  function() {
...
  // NOTICE: This is what has changed from the requery() function
  var oCallback = {
    success: this.onDataReturnReplaceRows,  //<<- this line    failure: this.onDataReturnSetRows,        
    scope: this,
    argument: newState 
  };
...
};

We can then make our onSuccess callback invoke this new function, and test it out:

 <g:formRemote name="worktask_form" onSuccess="GRAILSUI.dt_1.refresh();" 
         url="[action:'addRow',params: [ taskType:'work']]">

This time, adding a new row refreshed our DataTable and the new row shows up in our list– which is awesome.

Deleting Rows

Now that we can add Tasks to our list, we need to also provide a mechanism for removing them. Recall that we configured our dropdown list for the ‘status‘ column to allow the for three values: ‘open‘, ‘closed‘ and ‘delete‘. The plan is to use a cell edit on the status column to trigger the deletion of the Task. With that in mind I added the following code to the tableChange action (which is the action to be called during our cell edits):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

package com.alterlabs.twodo
class HomeController {
  ...
 
 def tableChange = {
   def profileInstance = Profile.current()
   def task = Task.get(params.id)
   ...
   if ( params.field == 'status') {
     switch (params.newValue) {
       case 'open': 
         task.completionDate = null
         task.save()
         break
       case 'closed': 
         task.completionDate = new Date()
         task.save()
         break
       case 'delete':
         // no easy way to tell what list it was from so remove from both
         profileInstance.removeFromWorkTasks(task)
         profileInstance.removeFromPersonalTasks(task)
         profileInstance.save()
         println "*** Deleted task ${task}"
         break
     } 
   }
...
}

When we change our status column to ‘delete‘ we see a rewarding message in the console, but our DataTable does not update to show any change. We have to refresh the page for the DataTable to show an updated list. Why? For that we move to the next section….

Custom GrailsUI DataTable Changes

One thing that became quickly obvious was that as great as the GrailsUI plugin is, it still lacking in a few areas. Notably there is no mechanism for notifying the UI of a successful cell edit. This is not a big deal when we are simply editing the contents of an existing row (because our browser screen is naturally kept in sync)… but it becomes a definite issue when removing a row from the DataTable.

What we need is the ability to register a callback with underlying YUI structures of a function to be called after a DataTable cell edit. If we view the browser page source and look at the underlying javascript that the g:dataTable tag generates for us we can see what is up:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

... 
var registerEditorListener = function(editor, field, url) {
  GRAILSUI.dt_1.loadingDialog.show();
  var editorCallback = {
    failure: function(o) {      // revert the cell value
      GRAILSUI.dt_1.updateCell(oArgs.editor.getRecord(), field, oArgs.oldData);
      // alert user
      alert('Recieved an error during edit: ' + o.responseText);
     }
   };
   YAHOO.util.Connect.asyncRequest('POST', url, editorCallback, 'id=' + oArgs.editor.getRecord().getData('id') + '&field=' + field + '&newValue=' + oArgs.newData);
                    });
};

From the code above we can see that the taglib creates a registerEditorListener function, and does indeed define a callback to be registered with the AJAX call. BUT it only defines what to do in case of a failure (line 5) and not what to do in cases of success. Knowing this, its easy enough to tweak the grails-app/taglib/DataTableTagLib.groovy file included in the GrailsUI plugin project to allow for us to pass in as part of the g:dataTable configuration a callback to use for cell editing. After tweaking, our augmented taglib generates something like this:

1
2
3
4
5
6
7
8
9

...
var editorCallback = {
  success: function(o) {
    ${editorSuccessCallback};
  }, 
  failure: function(o) {
    ... 
  }
};

…and we simply add the editorSuccessCallback parameter to our g:dataTable configuration data:

1
2
3
4
5

<gui:dataTable id="dt_1"
  ...  
  editorSuccessCallback="GRAILSUI.dt_1.refresh();"
  ..
]"/>

Now anytime we perform any successful cell edit (including deletes) , the DataTable.refresh() function is invoked and our table is rendered accurately. Killer.

Conclusion and Next Steps

Fundamentally the application is now functionally complete. There was plenty of mundane polishing and such that was not documented in this series, but all of the big stuff was touched upon.

The work accomplished in todays posts took a little over 20 hours to complete… the majority of the time split evenly between exploring the GrailsUI TagLib code trying to make the refresh work and tweaking CSS (which always seems to take more time than expected.) That brings our total time spent todate to about 30 hours.

The next post will (finally!) talk about deploying the twodo application into Amazon’s Elastic Compute Cloud (EC2). See you soon…

Technorati Tags: agile, grails, groovy, grailsui, java, software development

In this post we will refine the general look and feel of the User Interface, as well as start building the core list management functionality via a UI Plugin.

Look and Feel

To this point we have done almost no UI creation, and have not customized the default out-of-the-box look and feel of the application. I knew from talking to the stakeholder (in this case my business partner) that he envisioned a simple/clean interface with little clutter. The general site template was envisioned to look something like this:

Under the assumption that the look and feel, would (like all other aspects of the app) continue to evolve over the course of the development iterations, I did not want to spend too much time on polishing the initial cut at the UI template. I started with a simple raw HTML/CSS template, a bit of time in The Gimp and some CSS tweaking via Firebug until I got something that seemed initially acceptable. I added in some greeting text and voila– comparing the original/old login screen:

…and the new one:

OK. Still not award winning perhaps, but at least it looks intentional.

I don’t profess to be much an artist when it comes to web pages, and don’t intend to turn this post into a treatise on web design, but I think its worth peeking at the main views/layout/main.gsp layout file used for this project:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

<html>
   <head>
        <title><g:layoutTitle default="TwoDo" /></title>
        <link rel="stylesheet" href="${createLinkTo(dir:'css',file:'template_css.css')}" type="text/css"/>
        <link rel="stylesheet" href="${createLinkTo(dir:'css',file:'main.css')}" type="text/css"/>
        <link rel="shortcut icon" href="${resource(dir:'images',file:'favicon.ico')}" type="image/x-icon" />
 
        <g:javascript library="application" />
        <g:layoutHead />
  </head>
  <body>
    <g:render template="/layouts/includes/logo"/>
    <div id="page">
      <div id="page-bg">
        <g:render template="/layouts/includes/messages"/>
        <g:render template="/layouts/includes/errors"/>
 
        <!-- start content -->
        <div id="content">
          <g:layoutBody/> 
        </div>
        <!-- end content -->
        <div style="clear:both;">&nbsp;</div>
      </div>
    </div>
    <g:render template="/layouts/includes/footer"/>
 </body>
</html>

The main thing that is worth noting is that for every major rectangular region of the main template (as indicated on the napkin) we have further subdivided the layout into smaller templates which are then rendered from within main layout via the g:render taglib. As an example, we examine a few of the templates below.

Here is the layouts/includes/_logo.gsp template, which has been configured to:

• Show a simple text logo. We will eventually replace this with a graphic
• Display a greeting for the logged in user.
• Provide a logout link

<div id="logo">
  <h1><a href="${resource(dir:'')}">TwoDo!</a></h1>
<jsec:isLoggedIn>
  <span class="greeting" >
    Hello,<jsec:principal/> (<g:link controller="auth" action="signOut">logout</g:link>)</span>
</jsec:isLoggedIn>
</div>

And finally, a look at the layouts/includes/_messages.gsp which includes logic for the rendering of flash messages in a consistent manner:

<div id="message_wrapper" <g:if test="${!flash.message}"> style="display:none;"</g:if>>
  <div id="messageBar" <g:if test="${!flash.message}">style="display:none;"</g:if>>${flash.message}</div>
</div>

This divide-and-conquer strategy makes sense in many things software related, and the case of layout templates is no exception.

Security and Profile Management

It quickly becomes apparent that we require a mechanism for retrieving the current users Profile after a successful login. JSecurity provides a SecurityUtil class with a method that allows us to get the currently logged in Subject. From there we can get the Principle, and from the Principle we can get the Profile. I found it convenient to implement this logic in a pseudo-GORM like fashion by simply adding a static method to the existing Profile class:

1
2
3
4
5
6
7
8
9
10
11
12
13

...
import org.jsecurity.SecurityUtils
 
class Profile extends Base {
  ...
  /**
    * Return the profile associated with the currently logged in user
    */
  static Profile current() {
    def principle = SecurityUtils.getSubject().getPrincipal()
    return Profile.findByUsername( principle )
  }
}

Our controllers/whatever now simply need to call Profile.current() to get the currently logged in user.

GrailsUI Plugin

As per our sitemap in the last post, we only have a handful of pages in the entire application. Once a user logs in, they will be at their home (views/home/index.gsp) page, which will display their current list of tasks side by side. The manipulations of these lists seemed a prime candidate for some fancy UI and AJAXy goodness. In particular I envisioned a dynamic table that would use AJAX to manipulate the contents of the task lists. Rather than roll it all from scratch we will enlist the help of a the Yahoo UI libraries by way of the GrailsUI plugin. I had used the YUI library on prior projects and, wanting to maximize my productivity, so chose it over other competing frameworks (jQuery, etc.)

The GrailsUI plugin basically provides us with a wrapper taglib that encapsulates the underlying YUI javascript. In particular we are interested in the gui:dataTable tag, which wraps a YUI DataTable Control. Installing the GrailsUI Plugin automagically installs its YUI dependencies.

Before we tackle using the taglib/control however, lets look ahead, and address our data testing needs.

From reading the associated documentation, I knew that our GrailsUI DataTable was eventually going be populated by an AJAX call that will return its data in JSON format– which is all well and good. So the first order of business was to create some fixture data with which to test. To this end I added a simple action to the default Task controller we generated in the last post to create a few tasks for purposes of testing:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

package com.alterlabs.twodo
 
class TaskController {
...
  def data = {
    def p = Profile.current()
    for ( i in 0..5 ) {
      p.addToWorkTasks(new Task(description:"Work Task #${i}", estimate:60))
      p.addToPersonalTasks(new Task(description:"Personal Task #${i}", estimate:60))
    } 
    flash.message = p.save()?"Added fake tasks to profile":"Failed to add fake tasks"
    redirect(action:list)
  }
...
}

The code above grabs the Profile of the current user then creates and associates some dummy Tasks with it. Pointing our browser at http://localhost:8080/blogtwodo/task/data invokes the action and allows us to visually verify the data by listing all the Tasks in the system.

OK. Now that we have some data, lets work on making a DataTable retrieve it via AJAX. The plugin home page has some helpful information about using the DataTable, as does the authors blog. As per usual I started with a crude test to work the kinks out, with the assumption that it could be suitably refined later. By following the examples provided in the plugin documentation I quickly put together a simple test page that looked like so:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

<html>
  <head>
    <gui:resources components="['dataTable']" />
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <meta name="layout" content="main" />
    <title>DataTable Test</title>
  </head>
  <body>
    <div>  
      <h1>Work DataTable:</h1> 
      <gui:dataTable id="dt_1"
         columnDefs="[
            [id:'ID', formatter:'text', hidden:true],
            [status:'Status', formatter:'text'],
            [description:'Description', formatter:'text'],
            [est:'Est', formatter:'text'],
         ]"
         sortedBy="id"
         allowExclusiveSort='true'
         draggableColumns="false"
         controller="home" action="tasks" params="[taskType:'work']"
         rowsPerPage="10" 
         paginatorConfig="[
             template:'{PreviousPageLink} {NextPageLink} {CurrentPageReport}',
             pageReportTemplate:'{totalRecords} total records'
         ]"/>
    </div>
  </body>
</html>

In a nutshell, the above code:

1. Includes the resources for the GrailsUI DataTable (line:3)
2. Makes a DataTable with 4 columns (‘id’, ’status’, ‘description’ and ‘est’). (lines:11-17)
3. Note: The ‘id’ column is strictly for sorting purposes and we don’t want it shown on the screen, so we mark it as ‘hidden’. (line:13)
4. Wires the DataTable to populate itself from the action ‘tasks‘ on the ‘home‘ controller and also passes the parameter taskType set to ‘work‘ (line:21)

The other configuration attributes are reasonably self explanatory, or will get additional treatment in later posts. The g:dataTable tag does all the (considerable) heavy lifting for us and imports all the UI javascript files, configures the underlying YUI DataSource and associated response schema, as well as setting up various AJAX callbacks on our behalf. Great stuff.

We have configured our datatable tag to point at a ‘tasks‘ action, so lets take a look at that now. Its pretty basic:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

package com.alterlabs.twodo
import grails.converters.JSON
 
class HomeController {
 ... 
  def tasks = {
    def list = []
    //
    // Get the current user profile
    // 
    def profileInstance = Profile.current()
    def tasks = (params?.taskType == 'work')?profileInstance.workTasks:profileInstance.personalTasks
 
    tasks.each {
       list << [
           id: it.id,
           status: it.completionDate?"done":"front burner",
           est: it.estimate,
           description: it.description,
       ]
    }
    def data = [
      totalRecords: list.size(),
      results: list
    ]
    render data as JSON
  }
  ...
}

The GrailsUI DataTable expects its response schema to be fed to it in JSON format. Our tasks action basically:

1. Gets the current Profile the logged in user (line:11 )
2. Looks at the params to see which list of tasks to work on (line:12 )
3. Iterates over the task list, adding a map of the Task attributes to a list (lines:14-20
4. Create a Map that holds the new list, as well as the total number of records. This is required by the response schema (lines:22-25)
5. Render the Map as JSON back to our view (line:26)

Pointing our browser to http://localhost:8080/blogtwodo/home/datatable should get us the following:

OK, it may not look like much now, but it proves out the underpinnings. We will use it as a starting point, and continue to refine it in later posts.

Next Steps

Alright. That’s about it for this post. To recap we have:

• Established a candidate look and feel for our application, and have broken out layout in to modular templates
• Provided a means to conveniently get the Profile of the currently logged in user.
• Installed the GrailsUI plugin and used it to populate a crude DataTable via AJAX

The above took about 1/2 a day, or 5 hours or so to make happen (most of which was spent wrangling with HTML & CSS,) which puts our running total of ~7 hours spent so far with ~33 remaining in the kitty.

In the next post we will continue to refine the overall UI and get our DataTable control to allow cell editing of our tasks. See you then!

Technorati Tags: agile, grails, groovy, grailsui, java, software development

This longish post will primarily concern itself with some initial setup and design and the inclusion and integration of JSecurity into our application.

The Vision

We know that our application is going to be a simple task management system called ‘twodo‘– the intent of which is to allow the user to balance their work to-do’s with their personal to-do’s.

When my partner (aka Stakeholder) and I were discussing the application, he drew a simple diagram on a napkin that showed two lists presented side-by-side, the leftmost list for work tasks and the rightmost list for personal tasks. This seemed as good of a place to start with regard to general design, and this simple vision drove much of the basic site layout and internal constructs.

After a few minutes of noodling, the following simple high-level sitemap emerged and seemed reasonable:

Basic Setup

I created a new project (grails create-app blogtwodo) and configured my conf/DataSource.groovy to point at a locally running MySql instance with a newly created/empty ‘twodo’ database.

Basic Domain Model

As stated above, the basic function of the app is to manage two sets of task lists– one list for work, and one list for personal tasks. To this end I created a Profile domain class and a Task domain class.

The Profile will represent a registered user within the system, and will contain the account information and some basic configuration settings. The Profile will also hold a reference to the lists of Tasks, one list for each task type.

package com.alterlabs.twodo
 
class Profile extends Base {
  String username
  String email
  String firstName
  String lastName
 
  int  hoursInWorkWeek = 40
  int  hoursInPersonalWeek = 10  
 
  static hasMany = [workTasks:Task, personalTasks:Task]
 
  static constraints = {
    username(unique:true, size:5..15)
    email(email:true,nullable:false,blank:false,unique:true)
    firstName(nullable:true)
    lastName(nullable:true)
  }
}

Similarly the Task class holds the task details, as well as a reference back to its parent Profile:

package com.alterlabs.twodo
 
class Task extends Base {
  String description
  Date completionDate
  int  estimate = 0  // minutes to complete task
 
  static belongsTo = Profile
}

You may have noticed that the Task and Profile Domain classes inherit from a Base class. From past experience I have found it handy to have a simple Abstract Base Class from which all Domain classes inherit. For now we will use this Base class to enable auto-timestamping for our Domain classes:

package com.alterlabs.twodo
 
abstract class Base {
  Date dateCreated  // grails will auto timestamp
  Date lastUpdated  // grails will auto timestamp
 
  static constraints = {
     dateCreated(editable:false)
     lastUpdated(editable:false)
  }
}

I figured the above Domain objects were good enough to start with, and I could worry about constraints, etc. as the design progressed. I used Grails to generate the controllers and scaffolding (grails generate-all), fired up the dev server and verified my efforts via the browser. Navigating to http://localhost:8080/blogtwodo presented me the expected default Grails welcome screen and listed the Controllers for my Task and Profile classes. (These generated CRUD interfaces are handy for initial testing and once the app progresses in functionality we can remove them.)

So far so good.

Security Setup

Our application does not have any exceptional security requirements– we want users to be able to able to start using the system with a minimum of fuss immediately after a simple registration process. We won’t need any roles for this application– users are either anonymous, or registered. I have used the Grails JSecurity Plugin on past projects and knew it would be more than sufficient for our needs.

To get the ball rolling I followed the steps listed in the JSecurity Quick Start Guide to install the plugin and autogenerate some basic underlying constructs.

The quick start procedure creates a bunch of Jsec-prefixed Domain classes and an AuthController into the default package. I like things a bit more organized so I moved both the Domain classes and the Controller into a com.alterlabs.sec package. As a result of moving these classes we also need to add an import statement to the generated grails-app/realms/JsecDbRealm.groovy file (import com.alterlabs.sec.*) so that it can find the various needed JSecurity objects at runtime.

As per the quick start guide I created a simple conf/SecurityFilters.groovy file. The following code requires authorization to access anything other than the (yet to be created) public controller:

import com.at.sec.*
class SecurityFilters {
  def filters = {
  // Ensure that all controllers and actions require an authenticated user,
  // except for the "public" controller
 
    auth(controller: "*", action: "*") {
      before = {
        // Exclude the "public" controller.
        if (controllerName == "public" || controllerName == null) {return true}
 
        // This just means that the user must be authenticated. He does
        // not need any particular role or permission
        accessControl { true }
      }
    }
  }
}

For purposes of initial testing, I also added some temporary code to the BootStrap.groovy file that creates a hardcoded user (user ‘don’, password ‘twodo’). We will remove this code later as the application progresses.

import org.jsecurity.crypto.hash.Sha1Hash
import com.alterlabs.sec.*
 
class BootStrap {
  def init = { servletContext ->
     // TODO: Remove this stuff later
    def role = new JsecRole(name: "User").save()
    def user = new JsecUser(username: "don",
            passwordHash: new Sha1Hash("twodo").toHex()).save()
    new JsecUserRoleRel(user: user, role: role).save()
  }
 
  def destroy = {
   }
}

To test it I fired up the app and was presented the generated list of Controllers, including an additional entry for the newly added com.alterlabs.sec.AuthController. Clicking on the Profile controller presents the Quick Start generated login page, and I can successfully login with the ‘don’ account created during the BootStrapping. Awesome.

Login and Registration UI

Recall that we want to allow for a simple registration process. Once a user has registered he/she should be able to login and start using the application. Right or wrong, I find that many agile based development efforts derive requirements from the imagined user interface, and that is the direction we will take here. I envisioned another napkin with as screen that presents the user with two basic panels, one for new user registration, one for existing users to login:

Anonymous users will only have access to the public space, and once authenticated they will be sent to a home controller, where we eventually hope to add the meat of our application logic and content. With that in mind I next created both a public and home controller (grails create-controller)

I replaced the default view/index.gsp page that Grails provides with a few simple lines that uses the JSecurity taglibs to redirect to either the home controller or the public controller depending on whether the user is authenticted.

${ response.sendRedirect("/blogtwodo/home") }
 
${ response.sendRedirect("/blogtwodo/public") }

Lets create a crude page, as per our napkin above that will allow the user to either register or login. We want this page to be publicly accessible so this was created in views/public/index.gsp:

 
<!--
#login_or_register {margin:10px;width:500px;}
#login_box {border:#006DBA 1px solid;float:left;width:220px;padding: 10px;}
#register_box {border:#006DBA 1px solid;float:right;width:220px;padding:10px;}
-->
<div class="errors"></div>
<h1>Welcome to TwoDo!</h1>
Register now to get started, or login if you have an account
<div id="login_or_register">
<div id="login_box">
 
      <label for="username">Username:</label>
<input id="username" name="username" size="20" type="text" />
      <label for="password">Password:</label>
<input name="password" size="20" type="password" />
      <button>Login</button></div>
<div id="register_box">
 
      <label for="username">Username:</label>
      <span class="value ${hasErrors(bean:profileInstance,field:'username','errors')}">
<input id="username" name="username" size="20" type="text" value="${fieldValue(bean:profileInstance,field:'username')}" />
      </span>
      <label for="email">Email:</label>
      <span class="value ${hasErrors(bean:profileInstance,field:'email','errors')}">
<input id="email" name="email" size="20" type="text" value="${fieldValue(bean:profileInstance,field:'email')}" />
      </span>
      <label for="password">Password:</label>
<input name="password" size="20" type="password" />
      <label for="vpassword">Confirm:</label>
<input name="vpassword" size="20" type="password" />
      <button>Register</button></div>
</div>

Pointing our browser tohttp://localhost:8080/blogtwodo renders the above (thanks to our redirect) as shown below:

OK, so we won’t win any design awards. It’s ugly but it will serve for now– we will worry about the aesthetics later. Basically we have a page with two forms that submit to actions within two different controllers.

The login form is trivial– it’s simply configured to submit to the existing signIn action within the AuthController that was created as part of the JSecurity Quick Start. Once again easily tested via the browser.

The first thing we need to do is provide the association between the JSecurity constructs and our own Profile object. To do this we simply add an association to the JsecUserRoleRel object to our Profile. The JsecUserRoleRel holds the relationship between a JsecUser and a JsecRole and has references to both– which in turn gives our Profile object access to both. (Note: Yeah I know I said we had no need of formal roles in our application, but it seems silly to paint ourselves into a corner.)

After adding this association to the Profile class now looks like this:

package com.alterlabs.twodo
import com.alterlabs.sec.JsecUserRoleRel
 
class Profile extends Base {
  ...
  JsecUserRoleRel  principle
  ...
}

The second form in the above GSP submits to the save action of the public controller for registration. The registration process needs to take care of the following:

1. Create and persist a new JSecurity JsecUser object
2. Create a new Profile object
3. Associate the JSecUser with the Profile and persist
4. Log the new user in
5. Send the authenticated user on to the home page

Our public controller now looks like this:

package com.alterlabs.twodo
import com.alterlabs.sec.*
import org.jsecurity.crypto.hash.Sha1Hash
 
class PublicController {
  def securityService
 
  def index = { }
 
  def save = {
    def profileInstance = new Profile(params)
    def role = JsecRole.findByName("User")
    def ju = new JsecUser(username: profileInstance.username, passwordHash: new Sha1Hash(params.password).toHex()).save()
    profileInstance.principle = new JsecUserRoleRel(user: ju, role: role).save()
 
    if(!profileInstance.hasErrors() && profileInstance.save()) {
      flash.message = "Profile ${profileInstance.id} created"
      // OK, lets log them in and take them to their home page...
      try {
        securityService.authenticate(params.username, params.password)
        redirect(controller:'home',action:'index')
      }
      catch (Exception ex){
        // Authentication failed
        flash.message = message(code: "login.failed")
        def m = [ username: params.username ]
        redirect(action: 'index', params: m)
      }
    }
    else {
      render(view:'index',model:[profileInstance:profileInstance])
    }
  }
}

NOTE: I have encapsulated the authentication logic from the JSecurity Quick Start AuthController into a SecurityService, which is called on line 21 above. (This is left as an exercise to the reader :) )

Once again we can verify our progress by using a browser to explicitly try out the registration process (and in this case I also created a few unit tests.)

Next Steps

OK, not a bad start. At this stage I had spent perhaps 1-2 hours on the effort, with a decent amount to show for it. We have created our core Domain objects, wired in security and have created the self registration process. The next post we will look at designing the View layouts, refining the User Interface, and adding some of the core functionality into the mix.

Technorati Tags: agile, grails, groovy, jsecurity, software development

The high-level constraints governing the experiment were as follows:

• The application should be web-based and ‘modern’ (AJAX, friendly URL’s, etc)
• It should be reasonably non-trivial.
• It should be developed as it were going to someday become a Software As A Service type of application.
• It should involve some new/untried/unfamiliar technology– just to make things more realistic/interesting
• Where possible, we’d explore some geeky ‘bells and whistles’

My biggest concern was the time constraint. 40 hours is not a heck of a lot of time, and given the other requirement of having to deal with some other unfamiliar technology I needed something that would allow for RAD. The resulting architecture for the application ended up looking something like this:

• The application was designed as a Grails (v1.1.1) application. No surprises here. I have used Grails on a number of prior efforts, and its my framework of choice.
• The more sophisticated/AJAXy UI components were provided courtesy of the excellent GrailsUI Plugin- which in turn is based on the excellent Yahoo UI JavaScript library.
• We decided to play around with deploying the application into a Cloud Computing infrastructure like Google’s AppEngine or Amazon’s Elastic Compute Cloud (EC2) platform. I had no experience with either prior to this effort.

The Cloud Computing aspect was the real twist — In one fell swoop, we attempted to meet the requirements for SaaS, ‘new or unfamiliar technology’ and high geek factor. As a bonus it could provide an online presence so that readers could visit and play with the app.

This post and a series of subsequent ones to follow, attempt to document in reasonable chronological order, the details associated with realizing this challenge. That is, the information is presented in the blog posts in the approximate order it was experienced in ‘real-life’– with a couple of notable exceptions.

In broad brushstrokes the application development cycle looked something like this:

1. Build a standard Grails application locally that meets the basic functional requirements.
2. Take the prototype from Step#1 and port it to Amazon’s EC2
3. Take the prototype from Step#1 and port it to Google’s AppEngine

The above approach seemed reasonable, and at the time I started this endeavor, I had signed up for Google’s AppEngine Java SDK access, but had not yet been granted it– so implementing an EC2 version of the application first seemed only reasonable. More on this in later posts.

OK! this is the last of the non-technical intro posts. The next post in the series will talk about the design and construction of the basic/local application.

Technorati Tags: agile, grails, cloud computing, appengine, ec2, software development

Integration testing can be painful enough under the best of circumstances. Confusion around where data comes from, when it should be deleted, rolled back or committed can derail the even the best laid plans. (This is especially true in more complicated projects that involve multiple datasources, that are managed outside the GORM.)

To better manage testing, we (ALTERthought) classify the persistent data to be used in our integration tests into two categories: Fixture Data and Dependent Data …

Fixture Data is data that is pertinent to the test case in particular. Data is considered to be Fixture Data if the following core conditions are true:

• The data is represented by a Domain Object in your system
• The data is writable (create/update/delete)

Fixture Data should be provisioned in one of the following ways:

1. Through a provisioning tool like DBUnit
2. Explicitly in the TestCase setUp()

NOTE: The decision as to whether we provision data using a tool like DBUnit or ‘locally’ in the TestCase setUp() is largely based on the complexity of Domain relationships. The more complicated the Domain Object graph, the more likely we are to use DBUnit to set up the data relationships. This has proven particularly true on projects where we must integrate with legacy databases that are highly normalized, and/or we have multiple datasources. Test fixtures for simple data can easily be provisioned in the setUp() method.

Upon test case completion, any Fixture Data that was added to the system should be removed, leaving the database in a clean state (see The Clean Slate Principle), containing only Dependent Data. Data should be removed in one of the following ways:

1. Through DBUnit processing
2. In the TestCase tearDown()
3. As part of the transactional TestCase rollback

NOTE: It is generally discouraged to explicitly remove the Dependent Data in the test method themselves! Unexpected test case errors can leave your DB in an unstable state with partially populated data.

Dependent Data is data that must be present in order for your test environment to simulate the real world, but is largely tangential to the test cases at hand. Ex: A code table containing the 50 US states, valid zip codes, etc.

Data is considered dependent data if all of the following conditions are true:

• The data is tangential to the test cases at hand
• The data in question is read only. Your test cases will never update or delete the data in question.
• The data can safely live in the DB between test cases.

Dependent data should be provisioned in SQL scripts that are run prior to any tests as part of the environment setup. Once it has been provisioned it need not be re-provisioned for every individual test case.

OK, that’s it! It may seem like obvious advice, but establishing a few simple rules can go a long way towards making testing of your next Grails app much less painful. Subsequent posts will talk a bit about using tools like dbunit to help achieve your testing goals, testing with multiple datasources, as well as advice/hints on testing the various components of a Grails app.

Technorati Tags: grails tests, integration testing, unit testing

Features

• Generation of a terracotta configuration file
• Generation of terracotta enabled start up scripts for containers

Installation
grails install-plugin terracotta

Usage
Install the plugin
Run the start up script generation task
Run the terracotta configuration generation task
generate a war for the application and deploy in a container
start the terracotta server(s)
start the application servers

Components
Generate<ContainerName>Script: Generate the start up script for an application server
GenerateTcConfig: Generate the tc-config.xml file for terracotta

Generate<ContainerName>Script
Currently only 2 containers are supported: jboss and tomcat.

Edit TerracottaConfig.properties to set the parameters for the start up scripts:
The terracotta install directory on the target jboss or tomcat server:
terracotta.install.dir
The path place where the tc.config.xml will be made available to on the servers (this plugin does not remote copy it you have to do that):
terracotta.config.path

Run:
grails generate-jboss-scripts

Copy the generated scripts in the startup script folders of the containers. You will later start the containers using these scripts instead of standard ones.

GenerateTcConfig
Generates the tc-config.xml required to run a terracotta enabled container.
It enables the clustering of all the domain classes defined in the project and allows to add any additional classes to be made clusterable by including a user defined xml segment.
It also clusters the http sessions across the container instances for your application.

Edit TerracottaConfig.properties to set the host names and ports for the terracotta servers to be used by the application.

Edit CustomIncludes.xml to specify any additional terracotta include rules required by your application.

Run:
grails generate-tc-config

Copy the generated file on each container server, in the location specified when generating the container start up scripts.

Running the Clustered Application
Deploy your application on all the container instances.
Start the terracotta server(s)
Start the containers using the generated start up scripts. (Note that you will need a form of load balancing to witness the effects of the clustering, I used Apache with mod_jk)

Fun Note
Start one only one node of your cluster. Start a session (log in your app, …). Stop that server instance, then start another one. Continue using your application without losing your session!

Looking Ahead
Future enhancements include support for more containers (easy), and support for running terracotta in development mode (more complex).

Technorati Tags: Terracotta, Grails, Scalability, Java, Clustering

Its happened to all of us, a test that was working just fine, suddenly starts failing because the test database was left in an unstable state. Enter the Clean Slate Principle.

Grails integration tests should endeavor to follow a simple rule:

Leave the database in the state in which you found it.

In other words, if your tests perform any write operations on your datasource(s) (Create/Update/Delete) they are responsible for undoing said operation upon test completion.

This ‘Clean Slate Principle’ applies to Test Cases as a we as Test Methods within Test Cases… i.e., if a Test Method inserts a row into the DB as part of a test, it should be removed upon completion (regardless of success or failure).

Subsequent posts will discuss approaches for accomplishing this, as well as the types of pre-existing test data and techniques for provisioning. Until then, keep your slate clean!

Technorati Tags: grails tests, integration testing, unit testing

controller.metaClass.redirect = { Map args -> return args}

Note that the action of the redirect will not be invoked. Also note that I am MOPing an instance, not the Class.

Technorati Tags: grails tests, groovy tests, grails controller, unit testing

What I found is that it could have helped us in at least 4 different ways.

Session Clustering
The lowest hanging fruit is the ability of terracotta to be used as a session clustering solution. The beauty of it is that it can be done with NO change to the application code. I was able to enable terracotta management of the sessions for our Oyogi application without changing a line of its source code. In fact I did not even recompile it, I only created the terracotta configuration and edited the the JBoss start scripts as described in the Quick Start.
Terracotta can be used in that way for all the major web containers: Tomcat, Jetty, JBoss, BEA WebLogic, IBM WebSphere, Glassfish and Geronimo.

Global Singleton
Terracotta provides the ability to turn traditional (single JVM) singletons into terracotta cluster wide singletons. Those are not “real” singletons as one instance exists on each JVM clustered by terracotta, but they can behave in all manners just like if there was really a single instance shared by all the JVMs. This can also be easily applied to Spring beans thanks to the Terracotta for Spring.

Keep Transient Data Transient
This is where I believe terracotta can have the most unique impact, but it also is where the application structure is directly impacted by the usage of Terracotta. In many applications data that is transient in nature but needs to be shared between JVM instances ends up being stored in the database. Terracotta allows us to keep this data in the Java heap and cluster it. This approach can also in some cases replace the usage of queuing infrastructure (such as JMS) between a producer and a consumer.
Taking full advantage of this requires some changes in the application code, but those changes result in simpler solution. The code is modified to use objects in the heap directly instead of retrieving data from a database (and write to objects in the heap instead of update/insert in a database). A core benefit of this approach is to offload work from the databases and let them do what they are designed to do: store and retrieve persistent data.

Hibernate Second Level Cache
Terracotta provide support for Hibernate from multiple perspectives. First it supports basic Hibernate object clustering: a data object retrieved through hibernate can have it’s lazily read attributes read from any node in the Terracotta cluster. Second it provides clustering of Hibernate detached objects (data objects no kept after the end of the database session). Finally and most importantly it provides support for clustering the use of EHCache as a Hibernate Second Level Cache.
Like the session clustering solution this usage of Terracotta can be completely transparent to the application. An single JVM application should be possible to deploy across multiple JVMs clustered by Terracotta with no changes to the application itself.

Technorati Tags: Terracotta, Hibernate, Scalability, Java, Clustering