Clojure’s great fun, but how can I increase its chances of use at work?

From personal experience and feedback from others, I don’t see big rewrites of systems being a viable way of introducing new technology; who can take the risk of spending 6 months to simply end up with the same business features again? That doesn’t deliver value for the users.

I much more subscribe to the approach of incremental improvement through an approach such as the strangulation pattern. The question was, could I incrementally introduce Clojure?

The server application I work on is for the operations part of the financial Fund of Fund business; financial transactions, valuations and planning. Its a Java application using Spring MVC to expose the API via HTTP.

I was first thinking I needed to use Clojure’s Ring library for the HTTP part which then lead me to the ring-java-servlet project for adding a new Servlet to our existing web.xml.

Upon reflection, this didn’t feel the right way to go. Thinking a bit more, couldn’t I write a Spring MVC controller in Clojure?

The short answer is yes!

Sorry if the colours appear crappy, but somethings changed with code snippets in Octopress and I want to get the article out rather than delay publishing - I will tidy up afterwords!

We use Spring MVC’s annotation-based configuration to find controllers in the application’s classpath. This means we need to use Clojure’s AOT (Ahead Of Time) compilation so that there is a class to find at start-up time.

This was the first time I’ve used Clojure’s support for generating a class, which wasn’t difficult. What was more challanging was getting the annotations correct.

Clojure has support for adding metadata to symbols and collections. This approach is used for adding Java annotations to generated classes and methods. The tricky bit is working out what value to give the annotation (it’s a map so every annotation needs a value).

If there’s no value for the annotation, you can use an empty map. The empty string given to the Controller annotation is because you can give an optional name.

In Java I’d use a single string to add the URL template to the RequestMapping annotation, but if you read the source code for the annotation you’ll notice that it is actually an array. I’m not sure if the ability to add a single string is some syntactic sugar of the Java compiler, but you can’t do this from Clojure; you need to make it a list.

For building, I used Leiningen via Eclipse’s CounterClockWise Clojure plug-in. You need to remember to tell Leiningen to do the AOT build as well otherwise the gen-class declaration is ignored.

Now I just need to find a new part of the system to implement where choosing Clojure would be the smart thing to do…

Getting back into web development is an overwhelming experience as I recounted in my last entry. After perusing this interesting comparison of the various MVC Javascript frameworks, I decided to look further at AngularJS (I’ll just call it Angular from here on out).

Why Angular? I liked Angular’s approach to HTML templating and the emphasis on testing. Only after someone else mentioned it, did I realise that Angular imposes no particular implementation on the model; many of the other frameworks impose an inheritence model.

Based on the quantity of links and Github watchers, Backbone.js appears to be the leader in this space, though I think the comparison might be rather superficial as Backbone is more limited in scope (by design) than Angular. Backbone is more at the library end of the scale and Angular is towards the framework end.

As part of my evaluation of Angular, I wanted to build some very minimal working code. I have a lot of experience with Java, but find Java (the language) frustrating for modern development. To increase the fun, I built the backend with Grails; the Java world’s answer to Ruby on Rails. Grails will be a topic of another blog entry.

The Domain

I work in finance, so the demo was around CRUD (I only did the RU bit for Angular) for managing trades, essentially the buying and selling of funds. It could be about CRUD for any thing as finance only impacted the name of the fields I was capturing, not the functionality of the demo.

Learning

I find Angular’s documentation adequate - nothing more. The API guide is complete, but often missing the ‘why’ and the ‘how’ of a feature. Experimentation is the name of the game - something I assume anyone looking at an early-stage technology (in any area of IT) would be comfortable with.

I think the tutorial is the weakest part. I can’t quite put my finger on why the tutorial doesn’t work for me, but I didn’t like it and skipped it.

Much more useful was the basic Angular template project. Clone it from Github and you’re good to go with a minimal, working Angular app. Now you can experiment and see the results.

There’s a Google group and IRC chat channel as well. I had the occasional hit on the Google group when searching for answers, but I never tried the IRC channel for help.

There’s a good number of videos on Youtube from the Angular team from various presentations they’ve given. I’d also recommend looking at some of the shorter and more focused videos on implementing Angular, especially from John Lindquist - as a bonus you’ll see John using Webstorm, a pretty nice IDE for web development that I switched to half way through my two week investigation.

What I Implemented

Because I don’t have enough new moving parts in my exploration, I deployed the end result to AWS (Amazon Web Services) using a custom domain and added HTTPS support.

I’d already used Grails to generate scaffolding for a simple CRUD app for trade entry using server-side rendering, so I decided to replace the RU bit with an Angular implementation.

I’m assuming readers are at least vaguely familiar with the MVC pattern (and its many variants). At the end of the day I didn’t have much code to get Angular using the JSON-based Grails server. A big part of this is Angular’s resource abstraction that makes it simple to bind CRUD functionality to a REST-flavoured server.

One area Angular is not opinionated is the file structure of your project. Being recently returned to JavaScript, I don’t currently have a good feel for what makes a logical and maintainable file layout for Angular’s routes, controller, services and models. Looking at how others organise their projects will be an interesting learning experience.

Security

This more a general topic for the SPA (Single Page Application) style of application. With server-side rendering of web pages, it’s easy secure the HTTP end-points, but how to secure the XHR calls? Ultimately, this appears to be going the way of something like OAuth2 with tokens added to every call, but I didn’t want to add yet another new topic to my investigation. Maybe I could continue to use session-based security I get for free with Grails?

The problem: requesting a secured URL triggers an HTTP 302 Found response that is automatically handled by the browser and does the redirection behind-the-scenes. You asked for a list of trades in JSON? Have an HTML page for the login form instead!

The redirect-to-login-page-on-insecure-access pattern doesn’t seem like a good fit for SPA apps. Looking at the HTTP responses, 401 Unauthorized seems a more fitting response. But how to handle this in Angular?

The Interwebs to the rescue! You can find a discussion and code for a solution here and here. Thanks to Witold Szczerba for sharing his solution. The solution uses Angular’s interception feature to:

1. fire an event when we receive a HTTP 401 response and remember what we were trying to request
2. Other code listening for the event will dynamically show a login form
3. authenticate and fire a success event
4. request the original URL

It took me a few hours to get this working in my app, but it’s always a nice feeling when you get the code to do what you want it to.

Summary

All-in-all, I enjoyed my experiment with Angular.

What I liked:

• Declarative nature with the HTML fragments
• Lack of code to write
• REST/resource abstraction

What needs some work:

• Documentation
• Examples
• Composable views

Ever since I did front-end development with C & X Windows in the mid 90’s I’ve stuck to server-side development as a preference. Some ASP (no, not the nicer ASP.NET, but the original - *shudder*) in the late 90’s was a brief detour, but it’s been solidly server-side since.

We’ve all seen the trend towards applications migrating to the browser and the related resurgence of Javascript as a language.

Related to this, I’ve been working on a large application with a legacy Java Swing client for the last number of years and our attempts to modernise this with a service layer and a more ambitious change of client technology.

For various reasons we started with Flex at the tail-end of 2010, and for non-technical reasons, this approach was killed in late 2011 (no, not related to Adobe’s foot-in-mouth moment of appearing to shoot Flex in the head).

If I was to make the decision today, I’d go with a pure web approach. While we don’t have the luxury of doing the re-write at the moment, I do want to be ready with a recommendation on what we could use.

This leads to the TL;DR bit - holy shit! There’s a lot of frameworks and libraries for doing modern web stuff. It’s damn confusing and I’m sure there’s plenty of right answers.

We want to build a web application, not a web site, so I was leaning towards the single-page application style. Gmail is a classic example of this style. I’ve also never been a fan of server-side templating - this maybe a side-effect of doing ASP all those years ago. I’d prefer to keep the services as a pure API unencumbered with presentation stuff.

I’ll end this piece with a list of interesting libraries and frameworks I came across in my research and follow-up on what came next:

• Extjs
• Smartclient
• Ember.js
• Backbone.js
• Knockoutjs
• AngularJS

Further to my earlier blog post on the Clojure Programming book, I decided to give myself a good kickstart by attending the Euroclojure training and conference in London at the end of May 2012.

I was disappointed to learn that only seven people were taking the opportunity for the training, especially since Relevance were they guys giving the course. On the other hand, it did mean everyone could engage with the two instructors, Stuart Sierra and Luke VanderHart (and yes, that is a capital H in the middle of his surname) at leisure.

At this stage of Clojure’s growth, who invests in a training course? We had a couple of trainers from Ireland, a startup founder from Denmark and two guys from the CNRS, the French National Center for Scientific Research. All keen technologists, as you’d expect. Investing in Clojure from a career point-of-view is not a sure-fire winner at this point-in-time.

The course was over three days and alternated between theory and some of the most challanging labs I’ve had in language training. It wasn’t always possible to complete the labs in the time allowed (which was at least an hour), but one could always come back to them in quiet moments or at the end-of-the-day. Luke and Stuart alternated between the topics and the class had a good back-and-forth discussion with both the trainers.

Of the lab sessions, I most enjoyed solving the how-to-score-a-hand-of-poker problems. This involved lots of higher-order function use and I needed an additional hour in the evening to finish it, but it did give a sense of accomplishment when I got the final code to run.

For the afternoon of the last day we had an introduction to ClojureScript. This was the one part of the training that didn’t work for me. To give some context, I mostly do server side development and last did web development back in 1999 with a mixture of ASP, VBScript and Javascript. We used the ClojureScriptOne application as the starting point of wiring up a web application to rock-paper-scissors game we implemented in an earlier lab.

This involved too much magic for me and I couldn’t work out what needed to be done for most of the lab exercises and needed to look at the answers. So while I’m interested in ClojureScript, this lab didn’t improve my understanding.

For the final hour of the training Stuart suggested we work through some code examples as a group to give a more interactive approach. This worked well and helped me get on track as my solo efforts to solve the last problem were going in the wrong direction.

All-in-all I thoroughly enjoyed the training and would recommend it to anyone interested in getting their feet wet with Clojure. Thanks go to Stuart and Luke for an entertaining three days.

No, not a review of a Clojure book you’ve never heard of: Dive into Clojure, but more a play on words as I spent quite a bit of time reading Clojure Programming (not to be confused with Programming Clojure) while on a diving holiday in Egypt’s Red Sea.

Before getting to the book I want to say I think O’Reilly’s Safari Online is pretty terrible. Difficult to navigate and always pushing for you to take a paid subscription. It compares badly to The Pragmatic Programmer guys in managing your books online. I was annoyed that the epud version wasn’t available until the book was finished (I regularly use the early access feature to technical books).

I did very little functional programming at University and what I did (ML), I didn’t like and/or appreciate. Since then I’ve been programming in a variety of languages including C/C++, Visual Basic and, for the last 10 years, Java. All firmly in the imperative style.

I am always looking for tools to make my programming life easier and I’ve become increasingly frustrated with Java’s verbosity and clumsy support for meta-programming.

Of the newer languages on the block, I’ve dabbled with Python, Ruby, Groovy, Scala and now Clojure. So far, Groovy had been the the alternative language that was usable in my day job as it can work as a companion to your Java code whereas I see the other languages as a complete replacement (though they can all run on the JVM).

Why Clojure? Why not? Clojure had 2 points that drew me to it: it’s a Lisp and that has an air of mysticism attached to it since it comes from the dawn of the computer age and it’s a functional language. Scala pitches itself as a functional/OOP hybrid; you’re not forced to go functional and I felt I would benefit from using a functional-only style language.

Of the three authors (Chas Emerick, Brian Carper and Christophe Grand), I had come across Chas and Christophe in my web reading around Clojure.

The first part of the book goes into detail on the nuts and bolts of Clojure before getting to the functional and concurrency parts of the language.

The one thing that sold me on the book and spending more time with Clojure is the example of a functional approach to implementing Conway’s Game of Life.

Wow.

This was a real eye opener for me on the power of a functional style. The elegance of the solution is something for me to aspire to as I learn more of Clojure.

This is the first time I’ve explored macros in Clojure. Previously I suffered the double trouble of hearing that macros are complicated and a mind poisoned by the C/C++ usage of the word ‘macro’. While I’m sure there’s plenty still to learn about them, I’m starting to see the power and their usage.

I enjoyed the fact that the book goes into detail on practical things, such as database access, web programming and how to deploy things. All essential features of getting stuff working in the real world.

I heartily recommend the book.

This is a small follow-up on my first post onJaCoCo & Gradle that shows how to collate and report the code coverage for a multi-project build.

We’ve already seen the basics in part one. We still run JaCoCo during the unit testing, but now the unit testing happens once per Java project.

I decided to collect together the generated JaCoCo coverage files for generating the report, but I think this step could be cut out, but I haven’t tried this simplification yet. Here’s the code coverage run per project and the results copied to the root project’s output directory using the projects’ names as part of the new coverage file names:

Now we can generate the report from the individual JaCoCo data files and include the source code for each project:

The final part is to wire up the dependencies. We can’t generate the report unless all the sub-projects have finished copying the JaCoCo data files:

We’ve all experienced the ‘pleasure’ of Yak shaving. This is the tale of my most recent one.

In porting our build to Gradle we recently found we’d missed one project as part of the multi-project build. Easy, we just add the project to Gradles settings.gradle and all will be well.

Nope.

Another project in the build fails and it has no obvious relation to the newly re-added project. WTF? Some error about groovyc (the Ant task for building Groovy code) has failed.

A bit of googling shows we’ve run up against the command line limitation on Windows (8K for XP onwards, only 2K for older versions). Bummer, but we can change the default Gradle behaviour to run the Groovy compilation in-process rather than spawning the compiler from the command line and encountering the problem.

That should work. Nope.

Now Spock is complaining that we’ve failed to process the annotations on all tests using Spock. This must be because we’re running in-process and Spock’s annotation processor isn’t on the classpath.

Maybe adding Spock to Gradle’s classpath fixes this new problem? Nope.

OK, back to googling around. Stack Overflow shows a workaround for problems of classpaths that are too long. Basically, replace the long classpath with an empty JAR containing a MANIFEST.MF that defines the real classpath. Sounds easy enough.

Does the pathing JAR solve the problem? Nope. The first problem with this solution is the lack of a clear description with examples of how this works. The classpath. The paths should be relative URLs (absolute paths appear to work on Mac OS X, but not Windows) from the point-of-view of the pathing JAR. I did this, but none of my classpath dependencies were found and no obvious reason why.

Argh! Back to Google. I eventually found a recent bug about the command line length problem with Groovy and a proposed patch (not yet accepted). I also raised a problem with the Gradle guys to see if anyone else could point out a workaround.

After giving it a day and no simple workaround appearing, I decided to try and apply the patch myself and build my own local Groovy. I used the 1.8.5 version of Groovy and manually applied the patch in Eclipse as the patch line numbers didn’t match up to the source. Groovy recently moved to building with Gradle, so after cleaning up the docs.gradle file (it had diff markings in it, meaning it didn’t work), I was able to build with Gradle. The build did report an error right at the end, but this was after thegroovy and groovy-all JARs were produced, so that wasn’t a problem.

Using the locally built Groovy did have one downside: I needed to provide the needed dependencies explicitly rather than relying on the transitive dependencies automatically being resolved, but that’s a small price to pay.

So at last, the build works and I can leave that particular Yak alone.

There’s a new kid on the block when it comes to code coverage in the Java world: JaCoCo (Java Code Coverage).

It comes from the guys who did the EclEmma plug-in for Eclipse after they decided it wasn’t worthwhile to evolve the Emma code coverage tool (they are not the original developers). Instead they decided to start over and JaCoCo is the (ongoing) result.

We originally used Emma for the code coverage, but this stopped working when we started using Lombok to generate all the boilerplate code that Java needs. The next step was Cobertura. I’ve worked with Cobertura in the past and found the build integration brittle; you end up with either 0% or 100% coverage and no obvious errors. Luckily one of the other guys set Cobertura up with our Ant build so I didn’t need to do it again.

Now I’m moving our build over to Gradle and wanted to look at how JaCoCo was progressing. I was pleasently surprised to find they’ve released, so I went looking for documentation on how to get this working with Gradle. No such luck and no-one to my knowledge has (yet) written a Gradle plug-in. After this, maybe I will.

JaCoCo does ship with Ant tasks and since Gradle includes Ant, it is very easy to use the tasks to get the work done.

My first problem was working out where to get JaCoCo. Maven Central seemed the logical source, but that version appeared to be missing the Ant task and wouldn’t work with my Gradle script. I went for downloading JaCoCo and adding the Jar to my build project to solve the problem.

One difference with JaCoCo from Cobertura is it uses a Java agent to instrument the classes on loading; no longer do we need to post-compile instrument the class files. This seems a much cleaner way of doing things.

A quick bit of googling turned up the solution for specifying the agent argument when executing our tests in the Gradle build:

The first thing I do is add the JaCoCo jar to a new Gradle configuration, codeCoverage. This allows me to keep the JaCoCo Jar seperate from other compile and test dependencies.

I’m passing a number of arguments to the agent to control JaCoCo’s behaviour:

1. ${configurations.codeCoverage.singleFile}
2. destfile=${buildDirName}/coverage-results/jacoco.exec
3. sessionid=HSServ
4. append=false

Taking each of these lines in turn, we have:

1. Here I make use a new configuration, codeCoverage, to resolve the location of the JaCoCo Jar file.

2. JaCoCo writes the code coverage metrics to a binary file we specify here.

3. Identify the code coverage session. I haven’t had any success in seeing this in the results.

4. Re-create the output file each time rather than appending to an existing file.

This is enough to generate the statistics, but it doesn’t generate the report. For the report you need to do more work:

Phew! There’s a lot going on here. This is where we make use of the Ant tasks that come with JaCoCo. Let’s break it down a bit:

lines 3 - 5

Define an Ant task for the JaCoCo reporting task.

line 7

The output directory needs to exist before the report is run.

lines 10 - 14

Point JaCoCo to the binary file generated from the test run.

lines 16 - 26

The JaCoCo report needs access to the class and source files.

line 28

Create an XML output file (this isn’t needed if you’re only interested in the HTML output).

line 29

Create HTML report.

I link the HTML output in our Jenkins build (I’m not aware of a Jenkins plug-in for JaCoCo yet).

The Gradle build where I use JaCoCo is a multi-project build which gives us the added challenge of combining the results from each project into a single report, but that’s for another blog entry…