I once again had the pleasure of talking at Skillsmatter in early May, and this time I presented “Cloud Developer’s DHARMA: Redefining ‘done’ for Cloud applications”. I wrote about this on my company’s blog the night after I delivered the talk, but I’ve just realised I didn’t post anything here – therefore here we are. The synopsis for the talk can be found below.

As is always the case with giving a presentation at Skillsmatter, I very much enjoyed the experience, and there were some great questions and chat in the pub afterwards. Many thanks to all who attended – your comments and feedback are very much appreciated!

Skillsmatter have very kindly recorded the session, and you can watch my full talk here. You can also find a link to the slides on slideshare below.

Cloud Dharma Talk at Skillsmatter - Daniel Bryant

 

Talk synopsis:

Building applications for the IaaS Cloud is easy, right? “Sure, no problem – just lift and shift!” all the Cloud vendors shout in unison. However, the reality of building and deploying Cloud applications can often be different. This talk will introduce lessons learnt from the trenches during two years of designing and implementing cloud-based Java applications, which we have codified into our Cloud developer’s ‘DHARMA’ rules; Documented (just enough); Highly cohesive/loosely coupled (all the way down); Automated from code commit to cloud; Resource aware; Monitored thoroughly; and Antifragile.

We will look at these lessons from both a theoretic and practical perspective using a real-world case study from Instant Access Technologies (IAT) Ltd. IAT recently evolved their epoints.com(http://epoints.com/) customer loyalty platform from a monolithic Java application deployed into a data centre on a ‘big bang’ schedule, to a platform of loosely-coupled JVM-based components, all being continuously deployed into the AWS IaaS Cloud

If you have any questions then please do get in touch via the usual methods!

After attending Sam Newman’s microservice talks at Geecon last week I started to think more about what is most likely an essential feature of service-oriented/microservice platforms for monitoring, reporting and diagnostics: correlation ids. Correlation ids allow distributed tracing within complex service oriented platforms, where a single request into the application can often be dealt with by multiple downstream service. Without the ability to correlate downstream service requests it can be very difficult to understand how requests are being handled within your platform.

I’ve seen the benefit of correlation ids in several recent SOA projects I have worked on, but as Sam mentioned in his talks, it’s often very easy to think this type of tracing won’t be needed when building the initial version of the application, but then  very difficult to retrofit into the application when you do realise the benefits (and the need for!). I’ve not yet found the perfect way to implement correlation ids within a Java/Spring-based application, but after chatting to Sam via email he made several suggestions which I have now turned into a simple project using Spring Boot to demonstrate how this could be implemented.

Why?

During both of Sam’s Geecon talks he mentioned that in his experience correlation ids were very useful for diagnostic purposes. Correlation ids are essentially an id that is generated and associated with a single (typically user-driven) request into the application that is passed down through the stack and onto dependent services. In SOA or microservice platforms this type of id is very useful, as requests into the application typically are ‘fanned out’ or handled by multiple downstream services, and a correlation id allows all of the downstream requests (from the initial point of request) to be correlated or grouped based on the id. So called ‘distributed tracing’ can then be performed using the correlation ids by combining all the downstream service logs and matching the required id to see the trace of the request throughout your entire application stack (which is very easy if you are using a centralised logging framework such as logstash)

The big players in the service-oriented field have been talking about the need for distributed tracing and correlating requests for quite some time, and as such Twitter have created their open source Zipkin framework (which often plugs into their RPC framework Finagle), and Netflix has open-sourced their Karyon web/microservice framework, both of which provide distributed tracing. There are of course commercial offering in this area, one such product being AppDynamics, which is very cool, but has a rather hefty price tag.

Creating a proof-of-concept in Spring Boot

As great as Zipkin and Karyon are, they are both relatively invasive, in that you have to build your services on top of the (often opinionated) frameworks. This might be fine for some use cases, but no so much for others, especially when you are building microservices. I’ve been enjoying experimenting with Spring Boot of late, and this framework builds on the much known and loved (at least by me :-) ) Spring framework by providing lots of preconfigured sensible defaults. This allows you to build microservices (especially ones that communicate via RESTful interfaces) very rapidly. The remainder of this blog pos explains how I implemented a (hopefully) non-invasive way of implementing correlation ids.

Goals

  1. Allow a correlation id to be generated for a initial request into the application
  2. Enable the correlation id to be passed to downstream services, using as method that is as non-invasive into the code as possible

Implementation

I have created two projects on GitHub, one containing an implementation where all requests are being handled in a synchronous style (i.e. the traditional Spring approach of handling all request processing on a single thread), and also one for when an asynchronous (non-blocking) style of communication is being used (i.e., using the Servlet 3 asynchronous support combined with Spring’s DeferredResult and Java’s Futures/Callables). The majority of this article describes the asynchronous implementation, as this is more interesting:

The main work in both code bases is undertaken by the CorrelationHeaderFilter, which is a standard Java EE Filter that inspects the HttpServletRequest header for the presence of a correlationId. If one is found then we set a ThreadLocal variable in the RequestCorrelation Class (discussed later). If a correlation id is not found then one is generated and added to the RequestCorrelation Class:

The only thing is this code that may not instantly be obvious is the conditional check currentRequestIsAsyncDispatcher(httpServletRequest), but this is here to guard against the correlation id code being executed when the Async Dispatcher thread is running to return the results (this is interesting to note, as I initially didn’t expect the Async Dispatcher to trigger the execution of the filter again?)

Here is the RequestCorrelation Class, which contains a simple ThreadLocal<String> static variable to hold the correlation id for the current Thread of execution (set via the CorrelationHeaderFilter above)

Once the correlation id is stored in the RequestCorrelation Class it can be retrieved and added to downstream service requests (or data store access etc) as required by calling the static getId() method within RequestCorrelation. It is probably a good idea to encapsulate this behaviour away from your application services, and you can see an example of how to do this in a RestClient Class I have created, which composes Spring’s RestTemplate and handles the setting of the  correlation id within the header transparently from the calling Class.

Making this work for asynchronous requests…

The code included above works fine when you are handling all of your requests synchronously, but it is often a good idea in a SOA/microservice platform to handle requests in a non-blocking asynchronous manner. In Spring this can be achieved by using the DeferredResult Class in combination with the Servlet 3 asynchronous support. The problem with using ThreadLocal variables within the asynchronous approach is that the Thread that initially handles the request (and creates the DeferredResult/Future) will not be the Thread doing the actual processing.

Accordingly, a bit of glue code is needed to ensure that the correlation id is propagated across the Threads. This can be achieved by extending Callable with the required functionality: (don’t worry if example Calling Class code doesn’t look intuitive – this adaption between DeferredResults and Futures is a necessary evil within Spring, and the full code including the boilerplate ListenableFutureAdapter is in my GitHub repo):

And there we have it – the propagation of correlation id regardless of the synchronous/asynchronous nature of processing!

You can clone the Github report containing my asynchronous example, and execute the application by running mvn spring-boot:run at the command line. If you access http://localhost:8080/externalNews in your browser (or via curl) you will see something similar to the following in your Spring Boot console, which clearly demonstrates a correlation id being generated on the initial request, and then this being propagated through to a simulated external call (have a look in the ExternalNewsServiceRest Class to see how this has been implemented):

Conclusion

I’m quite happy with this simple prototype, and it does meet the two goals I listed above. Future work will include writing some tests for this code (shame on me for not TDDing!), and also extend this functionality to a more realistic example.

I would like to say a massive thanks to Sam, not only for sharing his knowledge at the great talks at Geecon, but also for taking time to respond to my emails. If you’re interested in microservices and related work I can highly recommend Sam’s Microservice book which is available in Early Access at O’Reilly. I’ve enjoyed reading the currently available chapters, and having implemented quite a few SOA projects recently I can relate to a lot of the good advice contained within. I’ll be following the development of this book with keen interest!

If you have any comments or thoughts then please do share them via the comment below, or feel free to get in touch via the usual mechanisms!

References

I used Tomasz Nurkiewicz’s excellent blog several times for learning how best to wire up all of the DeferredResult/Future code in Spring:

http://www.nurkiewicz.com/2013/03/deferredresult-asynchronous-processing.html

 

So, I’m currently travelling back from my first Geecon conference in Krakow, Poland, and I must say it was an awesome experience! Firstly, I have to say a massive congratulations and thanks to all of the organisers and volunteers, and especially to Andrzej (@ags313), Konrad (@ktosopl), Adrian (@adrno) and Adam (@maneo). These guys and girls work tirelessly throughout the year in order to run Geecon, and when I heard that they do everything themselves (without paid-for project managers etc.,) I was even more impressed!

I learned so much at the conference that I’m going to try and do a couple of blog posts to brain-dump my thoughts (although I know from past experience I know that this may not happen, and so I’ll focus on the one post at the moment :-) ). I’ll start with a review of the conference itself, move on to key memes (that I saw), and then briefly outline the sessions and social activities that I attended:

Conference Overview

  • The organisation of the conference is superb, right from the location (an out-of-town Cinema), the sessions, the speakers, the attendees, the food, to the amazingly helpful volunteers. My only small minus (and it’s really a symptom of the great success of the conference) was that the corridors can get super crowded during break times and lunch. I was chatting to Andrzej about this, and he mentioned that so many people wanted to come to Geecon that he had to stop accepting registrations after the initial 1200+…wow!
  • Krakow is an awesome city for the conference. It has some amazing sight-seeing and historical opportunities (Old Town, Castle, Salt mines, Auschwitz). Everyone I interacted with in the city was very friendly, and the vast majority also spoke superb English (which again puts the UK’s language skills to shame!)
  • Geecon is not just a Java conference, and this makes it all the better. It may be primarily focused on Java and the JVM, but you can easily pick sessions to avoid this (if you really wanted to?), and I managed to get a nice blend of different languages over the three days
  • Geecon is at the cutting-edge of thought-leadership within software development. There aren’t many conferences that can claim a perfect mix of sessions on programming fundamentals (often ignored), DDD, crafted design and architecture, Agile good practices, Java 8, JEE 7, Spring boot, Groovy, Scala, JavaScript modularity, microservices, DevOps, Open Source Software, low latency performance and more (except perhaps DevoxxUK, but I could be biased ;-) )
  • The social events and after parties are amazing – you’ll probably learn as much at the parties as you will in the main sessions :-)
  • Get plenty of sleep before the event, and also expect your brain to hurt after the three days of intensive knowledge injection. You know the part of The Matrix films where Neo ‘downloads’ knowledge on Kung-fu while plugged in to chair? Well Geecon is pretty much the beta version of this (fortunately without the wires plugging into your brain)

Key Memes

  • Never stop learning, and make sure you keep revising the fundamentals as well as innovating.
  • Jurgen Appelo’s awesome keynote set the tone for the first part of the above point perfectly, and I would recommend that all developers watch this talk when it is released on video by the Geecon team. The key takeaways for me were; have a goal, work relentlessly towards it (evaluating progress all the way, and adjusting were necessary); daily habits and discipline are key; be open to other people’s ideas; read more; read more; read more… ;-)
  • The ever-inspiring Kevlin Henney also did a superb job at both of his talks on Thursday. The first was a call to action for avoiding the common mistakes that many of us make when writing code, and also a reminder that style and layout matter. This can be summed up perfectly by one of his quotes “Style and layout matter in programming for the same reason they matter in writing” Amen to that… The second of Kevlin’s presentations was a homage to the ‘worse is better’ work by Richard Gabriel, and reminded us all of the fundamental’s of agile development, and that often less is more when it comes to software development. Key takeaways here were strive for simplicity, completeness, correctness, consistency
  • Java 8 really is making a difference to developers. I realise that many of the people presenting at the conference are thought-leaders, and also at the cutting-edge of software design, but I could clearly see the impact of Java 8′s new Lambdas and Streams. For a start, it made a lot of the code on slides much more readable (no more crazy anonymous inner Class boilerplate), and also more expressive, which allowed for key memes to be demonstrated in a much more understandable way (for example, in the great talk about Netflix’s RxJava library by Tomasz Kowalczewski)
  • Java EE 7 is also making a clear difference to developers. Many of the JEE sessions I have attended in conferences over the last few years have been about work-arounds for JEE versions <= 6, or the promised benefit of JEE 7, but at Geecon we actually got to see clear improvements from within the developer trenches (for example, Adam Bien’s and Arun Gupta’s sessions)
  • JavaScript really is growing-up. I know that this observation is not particularly new, and the JavaScript language (and tooling) has been developing in leaps and bounds over the past few years, but again at Geecon I saw proper evidence of this. In particular, the session by Paul Bakker and Sander Mak on JavaScript modularity and the proposed enhancements in ECMAScript 6 look awesome.
  • Microservices are at the tipping-point for mainstream adoption (and are in danger of being the ‘next big thing’). Both of Sam Newman’s talks about microservices were superb, and clearly demonstrated that Sam and the guys/girls at Thoughtworks have been doing this stuff in the trenches for quite some time. Sam is clearly ahead of the game in many areas of service design, implementation and delivery that I was very pleased to hear he was writing a book with O’Reily (which I’ve now bought in Early Access format here). Key takeaway’s from Sam’s talks included; software architects should be more like town-planners than building architects, especially when designing with microservices; be flexible with the implementation of microservices, but standardise the stuff ‘in between’ – monitoring, interfaces, deployment, architectural safety; strive for resource (not data or procedure) oriented designs; distributed txns are hard and should be avoided (see CAP theorem :-) ); distributed tracing and correlation IDs are very valuable (I’ll second this piece of advice!); get the testing strategy correct (think Mike Cohn’s test pyramid); make thing more ‘production-like’ close to the development (vagrant, docker, packer are useful here); consumer-driven contracts are very useful for testing; semantic monitoring is a great technique for testing in production.
  • The need for well-crafted design and architecture is still as important (maybe more) as it ever was. Sandro Mancuso’s excellent ‘Crafted Design’ talk provided clear evidence for this. Building heavily on DDD-based principles, Sandro proposed a new architecture and package design for Java applications that allows more cohesive modelling of the problem domain, and promotes clear separation between the Model Domain and delivery and infrastructure mechanisms. I wouldn’t do the proposal justice if I tried to describe it here, but check out the slides on Slideshare. It’s seriously worth spending time looking at this, as creating the ideal package/module structure that represents the problem domain is somewhat the holy grail of Java developers. It’s also worth checking out Simon Brown’s related work in this field (of which I am a big fan), and although he wasn’t at Geecon you will be able to catch him at Devoxx UK in June. I chatted to Sandro at the conference, and we both agreed that it would be great if he and Simon could get together sometime for a meeting of minds!
  • Several of the emerging languages and tooling are becoming a lot more opinionated, which I welcome. It’s great to have an uber-flexible language (Scala I’m looking at you here), but the flexibility and lack of constraints can confuse novices, and also stifle the creative challenge. Ken Sipe did an amazing job of introducing Google’s (very opinionated) Go language in an hour session, and this has inspired my to further look into this language as perhaps an alternative to some of the Python I write. Ken also did a superb advanced session on the testing framework Spock. I could see some people’s brains melting at some of the content (and mine got quite hot), but some of the power demonstrated looked simply awesome. Both of Ken’s talks demonstrated to me that the developers of the language/tool had clearly surveyed their respective fields and picked what they considered best for inclusion into their offering. This obviously could be hit-and-miss, but with Go lang and Spock I think we have a couple of winners.

That’s all for the moment, but if you ever get the chance to attend Geecon then make sure you do – you’ll learn lots, and have a great time doing it! if you have any comments then please let me know!

Just another piece of shameless self promotion, but you can catch me at the Geecon conference in Krakow, Poland next week (May 14th-16th), where I’ll be joining Heather Vancura, Richard Warburton and Arun Gupta on a panel discussing the adoption of OpenJDK, the Java Community Process (JCP) and the Java Specification Request (JSR).

You can find out more about the session and the speakers here. I look forward to seeing you at Geecon!

 

Java 8 Lambdas – by Richard Warburton

5_star

In a nutshell: If you are looking for a concise and high-impact guide to the core features of Java 8, then you’ve found the right book! Whether you’re just getting started with Java, or are a seasoned programmer, this book will help you get to grips with the new JDK 8 essentials such as Lambdas and Streams

I’m sure many people browsing this book will have heard about the recent (March ’14) release of Java JDK 8 and all of the associated excitement about ‘the biggest language changes’ since JDK 5, and no doubt you are looking to see if any of the current JDK 8 books are worth an addition to your programming bookshelf. My answer to that question for this book is a definite yes.

The book starts with a whistle-stop tour of the influences for the new language additions, such as Lambdas and Streams (with a nod to the much-vaunted ‘functional programming’), and then launches into explaining and demonstrating these key features in a well-paced and logical fashion. The stand-out chapters for me are 3, 4 and 5, in which you get to see many practical examples of where and how the new syntax (and new way of thinking) can save a lot of boilerplate typing and can also lead to much more expressive code.

As a relatively seasoned Java programmer, the examples and associated explanations had me up-and-running with the new features over the course of a weekend, and left me plenty of room (and motivation) to start experimenting in my own time with my new-found knowledge.

Data parallelism is covered well in Chapter 6, and the essential topic of testing (and a few associated gotchas with the new Lambda-driven approach) is covered nicely in the following chapter. Another excellent chapter is ‘Design and Architectural Principles’, which walks through some of the well-known design patterns and demonstrates to you that all of your current pattern knowledge doesn’t have to be thrown away when using a functional style of coding. There is also a great section in this chapter discussing how the new language features relate to Uncle Bob’s SOLID principles, and if you get chance I would also recommend searching for a companion talk by Richard on the Skillsmatter website.

Personally I think this book will end up alongside the classic Java books that I recommend to anyone looking to truly master the language, and this list includes such classics as ‘Effective Java’, ‘Java Concurrency In Practice’, ‘Java Generics and Collections’ and ‘The Well-grounded Java Developer’. Richard has done a great job of distilling the key elements of the new JDK 8 language feature, and presented them in a concise tour-de-force without skimping on detail. A job well done!

Disclaimer: I know Richard personally from his good work within the London Java Community, and was also a reviewer for an early version of this book. I have endeavoured to write an unbiased review, and would be happy to discuss any of my thoughts listed here via the review comments section below, or via personal communication. Anyone who has seen Richard present, or had the pleasure of pair-programming with him, will know he is very much the ‘real deal’ when it comes to Java wizardry, and so I whole-heartedly support his first foray into the world of publishing!

Click here to buy Java 8 Lambdas on Amazon UK

Just a brief post to mention that I will be speaking twice at Devoxx UK this year (as well as helping out within the Program Committee!), and so I would love to see as many of you as possible at the conference. Devoxx is one of the truly great developer conferences, mainly because it’s run by developers, for developers, and the entry price is a bargain too. Have a look at www.devoxx.co.uk (and the @DevoxxUK twitter feed) for more detail, and snag your ticket today!

My first talk is entitled “Moving to DevOps mode: Easy, hard or just plain terrifying”, and this will be a joint presentation with Steve Poole from IBM. Perhaps unsurprisingly, this talk will focus on the experiences of Steve and myself as we have helped transition organisations into a ‘DevOps’ mode and culture. Steve will be drawing on his plentiful experience from working within large Operations/sysadmin teams at IBM, and I will be talking about lessons learned from applying the DevOps philosophy in a small startup. You can find out more details on the Devoxx UK website here

The Second talk is entitled “Adopt OpenJDK: Looking behind the curtain of everyone’s favourite development platform” and this will be a BOF run by Martijn Verburg, Richard Warburton, Mani Sarkar and myself. In this session we will talk about all things OpenJDK-related, such as how to get started with hacking on the open source Reference Implementation of the Java JDK, how to make changes to the codebase, and how to contribute back to the community in the most effective way. If you aren interested in getting involved with the OpenJDK, then this is one session that you don’t want to miss! You can find more details of this session on the Devoxx UK website here.

Please do get your tickets for Devoxx UK now, as they are being snapped up quickly! I’m looking forward to seeing lots of people at Devoxx, both old and new friends alike! :-)

 

I'm speaking at Devoxx UK!

I was given the opportunity to present a lightning talk at the regular London Software Craftsmanship Community meetup in March at Skillsmatter here in Shoreditch. I enjoyed giving the talk, and had some great questions and banter in the pub afterwards.

The whole event was very well run and received, and you can find more details about all of the talks at the Skillsmatter website - there is some great content here about evolutionary design, and the similiarity between functional and OO programming.

I’ve included the slides below, and a link to the recorded video can be found here

Follow

Get every new post delivered to your Inbox.