Why do I contribute to the JCP (and at the same time evangelise the Spring framework)?

Josh Long, Richard Warburton and myself were having an interesting conversation on twitter about standardisation early today, specifically related to the Java Community Process (JCP), which is the mechanism for developing standard technical specifications for Java technology. Josh asked a question that I often get asked “what does JCP standardisation offer?” (I’m paraphrasing here slightly). This is a totally fair question, and I thought it deserved a little more explanation than I could craft on Twitter.

Innovation and Standardisation; Ying and Yang

The key thing to remember about the JCP process is that it is not about innovation. Quite the opposite in fact. For a standard to be created there must have an initial requirement or problem, significant innovation creating solutions, ideally some competing ideas and implementations, plenty of evaluation and discussion, and ultimately an agreed approach on how to meet the requirement. This process takes time, and it is only at the second from final point the JCP can start creating standards. This is the biggest misunderstanding I encounter when running JSR hack days around the world, particularly with junior developers, as they think the JCP is some mystical think tank who crank out the latest and greatest innovative frameworks (I appreciate calling EJB ‘latest and greatest’ is very ironic 🙂 ).

It’s also worth mentioning at this point that the work of the JCP is now undertaken in the open (I do appreciate the fact that it didn’t used to be, but JSR-348 has made great progress to abolish the ‘behind closed door’ work). This openness provides a platform that allows anyone who wants to get involved to be able to contribute opinions and ideas to the process, and if a standard will cause problems (or is evolving in a problematic fashion) then the community can rise up and publicly duke this out with the spec leads (no Duke pun intended!)

Now on the flip side to this there exists organisations like Spring.io/Pivotal who are all about innovation, and are constantly pushing the boundaries of what a language or framework can do. Personally I love this. I have an entrepreneurial background, and I thrive on innovation and playing with the latest tech and bleeding-edge frameworks as do many of the companies I work with. The Spring framework really does excel here, and this is why I made the transition to coding in Spring back when the framework was at version 1.X and I was really struggling with building J2EE applications. However, as a consultant I appreciate that not all my clients (or the industry in general) think like this, or desire this level of innovation or disruption.

Many companies are inherently risk adverse (sometimes with good reason) and they want to ensure any investment in technology or training their people in a specific technology offers a long-term return on investment (ROI). Such organisation also often desire portability of application/code, and although the practical implementation on the Java platform of this philosophy may not have been perfect in the past, I’ve personally moved several large(ish)-scale Java EE applications across differing application servers with minimal effort. In my mind this is where standardisation can offer enormous benefits, particularly if the standardisation work is undertaken out in the open. On a related note, last year within the London Java Community (LJC) we undertook a community survey of our members, and many Java developers were in favour of standards such as those offered by the JCP (check out the result here http://londonjavacommunity.wordpress.com/2013/09/16/the-java-community-process-survey/)

Horses for Courses…

I strongly believe that innovation and standardisation are far from mutually exclusive, and in fact are very much mutually beneficial (perhaps to the level where one cannot exist without the other, but this is just my opinion). Without innovation we wouldn’t be the embracing the benefits offered by the latest incantation of Service Oriented Architecture (SOA), currently being labelled as ‘microservices’, lead by the likes of Spring Boot, Dropwizard and Ratpack in the Java space. I am very much enjoying working in this space, and the fact that I don’t have to follow any kind of specification results in some very agile, flexible and effective applications.

However, you don’t have to look too far to see the problems that an absence of standardisation can surface. Earlier in the year Facebook announced that it was attempting to create a specification for PHP, as none had existed up until this point, and this made it difficult to decide what the ‘correct’ behaviour of any particular PHP runtime should be. Recently the AngularJS team announced a new version of their framework, and suggested that there will most likely be no clear migration path between the current 1.X and new 2.X versions. This will surely stifle innovation and hamper maintenance of code within companies who have invested significant resources into version Angular JS 1.X (not to mention the problem of dealing with thousands of lines of code that are currently running in production). There are a couple of other related examples that spring to mind, but I won’t mention them as I hope readers will follow my intentions. On a related topic, I’m also very interested to see what will happen with the .NET platform now that Microsoft have open sourced the underlying code with an MIT/Apache2 licence…

Summary

So in summary, I think there is most definitely a place for innovation and standardisation, and I believe both are very useful. This is why I choose to publicly evangelise the Spring platform (and write stacks of code in Spring Boot), and at the same also support the great efforts of the JCP and the OpenJDK which help to drive the future of a standards-based Java platform.

I would be keen to hear other’s thoughts, and so please feel free to comment below 🙂

Disclaimer: I am a member of the OpenJDK Adoption group, and also contribute to the excellent work undertaken within the JCP via the London Java Community JCP committee. However, in contrast 90% of the Java code I write when consulting is currently Spring-based (specifically Spring boot of late), and I publicly evangelise the superb innovation undertaken by the Spring framework team.

Implementing correlation ids in Spring Boot (for distributed tracing in SOA/microservices)

After attending Sam Newman’s microservice talks at Geecon last week I started to think more about what is most likely an essential feature of service-oriented/microservice platforms for monitoring, reporting and diagnostics: correlation ids. Correlation ids allow distributed tracing within complex service oriented platforms, where a single request into the application can often be dealt with by multiple downstream service. Without the ability to correlate downstream service requests it can be very difficult to understand how requests are being handled within your platform.

I’ve seen the benefit of correlation ids in several recent SOA projects I have worked on, but as Sam mentioned in his talks, it’s often very easy to think this type of tracing won’t be needed when building the initial version of the application, but then  very difficult to retrofit into the application when you do realise the benefits (and the need for!). I’ve not yet found the perfect way to implement correlation ids within a Java/Spring-based application, but after chatting to Sam via email he made several suggestions which I have now turned into a simple project using Spring Boot to demonstrate how this could be implemented.

Why?

During both of Sam’s Geecon talks he mentioned that in his experience correlation ids were very useful for diagnostic purposes. Correlation ids are essentially an id that is generated and associated with a single (typically user-driven) request into the application that is passed down through the stack and onto dependent services. In SOA or microservice platforms this type of id is very useful, as requests into the application typically are ‘fanned out’ or handled by multiple downstream services, and a correlation id allows all of the downstream requests (from the initial point of request) to be correlated or grouped based on the id. So called ‘distributed tracing’ can then be performed using the correlation ids by combining all the downstream service logs and matching the required id to see the trace of the request throughout your entire application stack (which is very easy if you are using a centralised logging framework such as logstash)

The big players in the service-oriented field have been talking about the need for distributed tracing and correlating requests for quite some time, and as such Twitter have created their open source Zipkin framework (which often plugs into their RPC framework Finagle), and Netflix has open-sourced their Karyon web/microservice framework, both of which provide distributed tracing [edit 27/07/14: It would appear that although distributed tracing was mentioned as an upcoming feature in the Karyon blog post, it never made it in to the public Github repo. Thanks to John Eikenberry for pointing this out in the comments below]. There are of course commercial offering in this area, one such product being AppDynamics, which is very cool, but has a rather hefty price tag.

Creating a proof-of-concept in Spring Boot

As great as Zipkin and Karyon are, they are both relatively invasive, in that you have to build your services on top of the (often opinionated) frameworks. This might be fine for some use cases, but no so much for others, especially when you are building microservices. I’ve been enjoying experimenting with Spring Boot of late, and this framework builds on the much known and loved (at least by me 🙂 ) Spring framework by providing lots of preconfigured sensible defaults. This allows you to build microservices (especially ones that communicate via RESTful interfaces) very rapidly. The remainder of this blog pos explains how I implemented a (hopefully) non-invasive way of implementing correlation ids.

Goals

1. Allow a correlation id to be generated for a initial request into the application
2. Enable the correlation id to be passed to downstream services, using as method that is as non-invasive into the code as possible

Implementation

I have created two projects on GitHub, one containing an implementation where all requests are being handled in a synchronous style (i.e. the traditional Spring approach of handling all request processing on a single thread), and also one for when an asynchronous (non-blocking) style of communication is being used (i.e., using the Servlet 3 asynchronous support combined with Spring’s DeferredResult and Java’s Futures/Callables). The majority of this article describes the asynchronous implementation, as this is more interesting:

• Spring Boot asynchronous (DeferredResult + Futures) communication correlation id Github repo

The main work in both code bases is undertaken by the CorrelationHeaderFilter, which is a standard Java EE Filter that inspects the HttpServletRequest header for the presence of a correlationId. If one is found then we set a ThreadLocal variable in the RequestCorrelation Class (discussed later). If a correlation id is not found then one is generated and added to the RequestCorrelation Class:

	public class CorrelationHeaderFilter implements Filter {
	//…
	@Override
	public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain filterChain)
	throws IOException, ServletException {
	final HttpServletRequest httpServletRequest = (HttpServletRequest) servletRequest;
	String currentCorrId = httpServletRequest.getHeader(RequestCorrelation.CORRELATION_ID_HEADER);
	if (!currentRequestIsAsyncDispatcher(httpServletRequest)) {
	if (currentCorrId == null) {
	currentCorrId = UUID.randomUUID().toString();
	LOGGER.info("No correlationId found in Header. Generated : " + currentCorrId);
	} else {
	LOGGER.info("Found correlationId in Header : " + currentCorrId);
	}
	RequestCorrelation.setId(currentCorrId);
	}
	filterChain.doFilter(httpServletRequest, servletResponse);
	}
	//…
	private boolean currentRequestIsAsyncDispatcher(HttpServletRequest httpServletRequest) {
	return httpServletRequest.getDispatcherType().equals(DispatcherType.ASYNC);
	}

view raw

CorrelationHeaderFilter.java

hosted with ❤ by GitHub

The only thing is this code that may not instantly be obvious is the conditional check currentRequestIsAsyncDispatcher(httpServletRequest), but this is here to guard against the correlation id code being executed when the Async Dispatcher thread is running to return the results (this is interesting to note, as I initially didn’t expect the Async Dispatcher to trigger the execution of the filter again?)

Here is the RequestCorrelation Class, which contains a simple ThreadLocal<String> static variable to hold the correlation id for the current Thread of execution (set via the CorrelationHeaderFilter above)

	public class RequestCorrelation {
	public static final String CORRELATION_ID = "correlationId";
	private static final ThreadLocal<String> id = new ThreadLocal<String>();
	public static String getId() { return id.get(); }
	public static void setId(String correlationId) { id.set(correlationId); }
	}

view raw

RequestCorrelation.java

hosted with ❤ by GitHub

Once the correlation id is stored in the RequestCorrelation Class it can be retrieved and added to downstream service requests (or data store access etc) as required by calling the static getId() method within RequestCorrelation. It is probably a good idea to encapsulate this behaviour away from your application services, and you can see an example of how to do this in a RestClient Class I have created, which composes Spring’s RestTemplate and handles the setting of the  correlation id within the header transparently from the calling Class.

	@Component
	public class CorrelatingRestClient implements RestClient {
	private RestTemplate restTemplate = new RestTemplate();
	@Override
	public String getForString(String uri) {
	String correlationId = RequestCorrelation.getId();
	HttpHeaders httpHeaders = new HttpHeaders();
	httpHeaders.set(RequestCorrelation.CORRELATION_ID, correlationId);
	LOGGER.info("start REST request to {} with correlationId {}", uri, correlationId);
	//TODO: error-handling and fault-tolerance in production
	ResponseEntity<String> response = restTemplate.exchange(uri, HttpMethod.GET,
	new HttpEntity<String>(httpHeaders), String.class);
	LOGGER.info("completed REST request to {} with correlationId {}", uri, correlationId);
	return response.getBody();
	}
	}
	//… calling Class
	public String exampleMethod() {
	RestClient restClient = new CorrelatingRestClient();
	return restClient.getForString(URI_LOCATION); //correlation id handling completely abstracted to RestClient impl
	}

view raw

CorrelatingRestClient.java

hosted with ❤ by GitHub

Making this work for asynchronous requests…

The code included above works fine when you are handling all of your requests synchronously, but it is often a good idea in a SOA/microservice platform to handle requests in a non-blocking asynchronous manner. In Spring this can be achieved by using the DeferredResult Class in combination with the Servlet 3 asynchronous support. The problem with using ThreadLocal variables within the asynchronous approach is that the Thread that initially handles the request (and creates the DeferredResult/Future) will not be the Thread doing the actual processing.

Accordingly, a bit of glue code is needed to ensure that the correlation id is propagated across the Threads. This can be achieved by extending Callable with the required functionality: (don’t worry if example Calling Class code doesn’t look intuitive – this adaption between DeferredResults and Futures is a necessary evil within Spring, and the full code including the boilerplate ListenableFutureAdapter is in my GitHub repo):

	public class CorrelationCallable<V> implements Callable<V> {
	private String correlationId;
	private Callable<V> callable;
	public CorrelationCallable(Callable<V> targetCallable) {
	correlationId = RequestCorrelation.getId();
	callable = targetCallable;
	}
	@Override
	public V call() throws Exception {
	RequestCorrelation.setId(correlationId);
	return callable.call();
	}
	}
	//… Calling Class
	@RequestMapping("externalNews")
	public DeferredResult<String> externalNews() {
	return new ListenableFutureAdapter<>(service.submit(new CorrelationCallable<>(externalNewsService::getNews)));
	}

view raw

CorrelationCallable.java

hosted with ❤ by GitHub

And there we have it – the propagation of correlation id regardless of the synchronous/asynchronous nature of processing!

You can clone the Github report containing my asynchronous example, and execute the application by running mvn spring-boot:run at the command line. If you access http://localhost:8080/externalNews in your browser (or via curl) you will see something similar to the following in your Spring Boot console, which clearly demonstrates a correlation id being generated on the initial request, and then this being propagated through to a simulated external call (have a look in the ExternalNewsServiceRest Class to see how this has been implemented):

	[nio-8080-exec-1] u.c.t.e.c.w.f.CorrelationHeaderFilter : No correlationId found in Header. Generated : d205991b-c613-4acd-97b8-97112b2b2ad0
	[pool-1-thread-1] u.c.t.e.c.w.c.CorrelatingRestClient : start REST request to http://localhost:8080/news with correlationId d205991b-c613-4acd-97b8-97112b2b2ad0
	[nio-8080-exec-2] u.c.t.e.c.w.f.CorrelationHeaderFilter : Found correlationId in Header : d205991b-c613-4acd-97b8-97112b2b2ad0
	[pool-1-thread-1] u.c.t.e.c.w.c.CorrelatingRestClient : completed REST request to http://localhost:8080/news with correlationId d205991b-c613-4acd-97b8-97112b2b2ad0

view raw

gistfile1.txt

hosted with ❤ by GitHub

Conclusion

I’m quite happy with this simple prototype, and it does meet the two goals I listed above. Future work will include writing some tests for this code (shame on me for not TDDing!), and also extend this functionality to a more realistic example.

I would like to say a massive thanks to Sam, not only for sharing his knowledge at the great talks at Geecon, but also for taking time to respond to my emails. If you’re interested in microservices and related work I can highly recommend Sam’s Microservice book which is available in Early Access at O’Reilly. I’ve enjoyed reading the currently available chapters, and having implemented quite a few SOA projects recently I can relate to a lot of the good advice contained within. I’ll be following the development of this book with keen interest!

If you have any comments or thoughts then please do share them via the comment below, or feel free to get in touch via the usual mechanisms!

References

I used Tomasz Nurkiewicz’s excellent blog several times for learning how best to wire up all of the DeferredResult/Future code in Spring:

http://www.nurkiewicz.com/2013/03/deferredresult-asynchronous-processing.html

 

EhCache and Spring – can’t find class EhCacheManagerFactoryBean?

I’m playing around with Spring and EhCache, and having a lot of fun. This problem stumped me for a few minutes as a lot of the tutorials I was playing around with didn’t mention a key dependency!

The Error

org.springframework.beans.factory.CannotLoadBeanClassException: Cannot find class [org.springframework.cache.ehcache.EhCacheManagerFactoryBean] for bean with name 'cacheManager' defined in class path resource [application-context.xml]; nested exception is java.lang.ClassNotFoundException: org.springframework.cache.ehcache.EhCacheManagerFactoryBean
 at org.springframework.beans.factory.support.AbstractBeanFactory.resolveBeanClass(AbstractBeanFactory.java:1269)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.predictBeanType(AbstractAutowireCapableBeanFactory.java:578)

The Missing Dependency…

<dependency>
   <groupId>org.springframework</groupId>
   <artifactId>spring-context-support</artifactId>
   <version>${spring.version}</version>
</dependency>

You’ll also need two other core dependencies in addition to the standard Spring dependencies, but most tutorials/guides cover this:

<dependency>
   <groupId>com.googlecode.ehcache-spring-annotations</groupId>
   <artifactId>ehcache-spring-annotations</artifactId>
   <version>1.2.0</version>
</dependency>
<dependency>
   <groupId>net.sf.ehcache</groupId>
   <artifactId>ehcache</artifactId>
   <version>2.7.2</version>
</dependency>

As an aside, if you are looking for a great introduction and tutorial to using EhCache within Spring I can highly recommend Goyello’s Blog.

Overriding a packaged Spring application properties file via an external file

A common use case when developing a Spring application is the requirement to have multiple versions of configuration properties which vary depending on the target environment you are deploying to. For example, a database URLs or a feature flag may be environment specific, and the value you use for local (dev) testing may be significantly different than on QA or production etc.

Like most Spring development tasks, there are several way to solve the problem. My preference is based on the following assumptions and preferences:

• We create a default configuration properties file (e.g. ‘appConfig.properties’) and package this within the deployable artifact (JAR or WAR etc)
  • This file will contain a sensible set of default ‘baseline’ properties that the application requires to run successfully
  • This default configuration file will typically be used for development purposes i.e. the  appConfig.properties included within the application code will allow the application to work ‘out of the box’ on a well configured local development box
  • We are using Maven to package our application, and therefore we place the appConfig.properties file in the root level of the src/main/resources directory – this doesn’t mean you have to use Maven, but you will have to ensure that your application packaging process includes the properties files in a location that is on the classpath
• We want to override properties in the baseline appConfig.properties file via an external file located in the deployed application’s working directory
  • We typically name this file appConfigOverride.properties
  • We may not override all of the default properties, and therefore we want any properties not present in the override file to default to the values in the baseline appConfig.properties file
• It is possible to override application properties by passing parameters on the Command Line when executing the application, or setting system variables, but this is a separate topic 🙂

The Solution

We use the following structure for our application-context.xml:

	<beans xmlns="http://www.springframework.org/schema/beans&quot;
	xmlns:context="http://www.springframework.org/schema/context&quot;
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance&quot;
	xsi:schemaLocation="http://www.springframework.org/schema/beans
	http://www.springframework.org/schema/beans/spring-beans.xsd
	http://www.springframework.org/schema/context
	http://www.springframework.org/schema/context/spring-context.xsd"&gt;
	<context:property-placeholder location="file:appConfigOverride.properties" order="-1"
	ignore-unresolvable="true" ignore-resource-not-found="true" />
	<context:property-placeholder location="classpath:appConfig.properties" />
	….
	</beans>

view raw

application-context.xml

hosted with ❤ by GitHub

The key things here, are the ‘order’ attribute, which forces properties found in the appConfigOverride.properties to be used when a property is also found in another file (effectively overriding values from the other file), and the ‘ignore-unresolvable=”true” ignore-resource-not-found=”true”‘ which allows Spring to continue loading the context if it can’t find the external file (or it doesn’t contain override all of the default properties from the appConfig file)

Testing Web-based Spring Applications in 2013 (Part One)

A fellow LJCer and TDD/BDD advocate has recently started developing an application using the ever-popular Spring stack, and he asked my advice on how best to implement a Test-driven methodology within this context. I started typing an email back in response, but then it suddenly occurred to me that this might be useful for other people as well… So here it is – I hope it helps 🙂

Assumptions

The rest of this post assumes the following:

• Applications will be implemented in Java using the following components of the Spring stack: Core, MVC, Data, Security etc BUT we won’t be covering Spring Integration or Batch (although the recommended testing frameworks and tools can be quite similar I find that the techniques for testing these components can be quite different, and so this will have to wait until another blog post :))
• ‘End-users’ will interact with the developed application(s) via a web-based UI and/or HTTP-based API (typically, but not exclusively REST-like)
• For production an external datastore will be utilised to persist long-lived state within the application. We assume that you will want to test how your application interacts with this too.
• We will not cover testing via remote or embedded containers, other than using a simple in-memory servlet container such as Jetty to deploy the entire packaged application. Frameworks such as Arquillian offer some really cool features that allow Unit testing against real (embedded) containers such as Tomcat, GlassFish and JBoss Application Server. This is achieved by providing mechanisms to simulate production-like packaging (“ShrinkWrapping“) and deployment of an arbitrary sub-set of components (down to the Class-level) for testing within the embedded container, all via your favourite testing framework such as JUnit.

General Recommendations

If you’re new test-driven development I recommend the following books:

• Effective Unit Testing: A guide for Java developers – This is a great (and up to date) introduction to all key concepts and technologies within Java-based unit testing. This book not only covers motivations and methodologies, but also how to write ‘good’ tests that are expressive and maintainable etc, and also advice on choosing the most appropriate technique for each use case.
• Practical Unit Testing with JUnit and Mockito and/or Practical Unit Testing with TestNG and Mockito. These are both great books that contain a wealth of practical examples on how to test Java applications, and also discusses the why and how of TDD. Both books are very similar and differ primarily in the framework used to drive the tests. My advice is to buy the TestNG version in combination with ‘Effective Unit Testing’ (which is primarily focused on JUnit), as it’s great to know the differences and strengths/weaknesses of the two most popular Java-based testing engines.
• Growing Object-Oriented Software, Guided by Tests (Beck Signature) – this is one of the classic TDD books, and it discusses all of the high-level concepts and motivations of Test-driving application design. This book also provides a great practical example of how to use the prescribed methodology by detailing the construction and evolution of an entire application based on an auction website.
• There are a few books written about Unit testing JavaScript, and although I haven’t had chance to read them all I do recommend Testable JavaScript

Testing the Core

Unit Testing

Purpose: The core of your application models the problem you are trying to solve, and typically contains domain-specific representations and associated ‘business’ rules which you are primarily responsible for writing (i.e. the opportunity to re-use third-party code here is somewhat limited).

Accordingly it is easy to argue that this code should be well tested – not only are the components you implement here likely to be unique, but you want to be sure they behave as expected before wiring them up to frameworks that offer boilerplate functionality, such as persistence. Well written tests here can also ‘document’ your code (and associated functionality) at the micro level.

Tools:

• JUnit / TestNG – de facto frameworks for running Java tests. I like to run these as part of my Maven build process via the surefire plugin
• Spock – an awesome Groovy-based testing and specification framework. If you haven’t heard of it stop reading right now and go visit the webpage. Seriously. Now check out the Spock Basics page. How awesome is that?? 🙂 And it plays well with Spring too!
• Mockito – my favourite mocking framework. Others do exist, but I find Mockito strikes a great balance betweeen ease of use, expressivity and maintainability
• PowerMock – a great extension to Mockito for use when you face a difficult situation that Mockito can’t cope with, such as mocking an old library that relies on static methods, or mocking private or final methods, or mocking object instantiation. My advice is to always double and triple-check your component design when you start reaching for PowerMock – often the desire to use PowerMock indicates a testing ‘smell’, and may indicate that your component design should be improved, or the manner in which your code interfaces with external dependency could be enhanced. For example, rather than mocking a third-party library’s static constructor method, could you not hide this method call behind your own interface (following the decoractor/adapter pattern) which will facilitate easier mocking?
• ConcurrentUnit – a great utility framework for the difficult task of testing concurrency within components
• Make-it-easy – a great little framework for creating Test Data builder (the concept of which comes from the aforementioned ‘Growing Object-Oriented Software, Guided by Tests)‘
• junitparams – a great framework for augmenting JUnits parameterised testing (or as the creator states – ‘Parameterized tests that don’t suck’)

Do:

• Isolate Classes and components under test. Unit tests should be focused on testing small units of work, and therefore have minimal dependencies (or dependencies should be mocked)
• Create test cases that cover all paths through the code, not just the happy path
• Create tests the replicate edge cases
• Use the power of parameterised tests to facilitate rapid addition of data-driven test cases as they are discovered

Don’t:

• Write trivial tests just to increase unit test coverage
• Create brittle tests that break with the slightest change to your code i.e. if your using more than 10 lines of code to set up your mocks and expectations then you may be doing something wrong
• Test getters and setters, unless they contain logic
• Directly test any interaction with the underlying OS (file, network etc), a datastore, or any container/server. This interaction should be mocked at this level of testing

Plugging it together

Integration Tests – Persistence Layer

Purpose: Think Unit testing for your persistence or DAO layer, but here you want to simulate behaviour offered by an external resource, rather than mocking this functionality.

Tools:

• All the Unit Testing tools mentioned above plus…
• Utilise Spring’s @ContextConfiguration  to manage the wiring of dependencies together, such as an EntityManager or MongoOperations, and AbstractTransactionalJUnit4SpringContextTests to manage running the tests with well-defined transaction semantics
• Embedded (in-memory) versions of your chosen datastore e.g.
  • H2 – A great embedded SQL datastore, and in 99% of standard use cases behaves like MySQL (although Your Mileage May Vary…)
  • Embedded MongoDB – for all your MongoDB datastore needs (or for older versions of MongoDB, pre 2.9, check out another project I contributed to SDFongo)
  • Embedded Solr via ZoomInfo’s InProcessSolrServer, and updated by me (shameless self-plug! 🙂 )
  • ActiveMQ – run as embedded and non-persisting

Do:

• Assert that entities persist as you expect them. I’ve lost count of the number of times I’ve incorrectly configured XToMany JPA annotations which resulted in incorrect cascading operations to child entities.
• Aim to keep test execution time reasonably small.
• Test all public methods exposed on the DAO/repository interface
• Create harnesses that allow the loading of pre-canned test data. Think executeSqlScript() from Spring’s AbstractTransactionalJUnit4SpringContextTests
• Destroy (or delete) data at the end of each test in order to prevent previous test data from influencing results. This is especially important if your chosen testing framework doesn’t guarantee the order in which tests are executed!

Don’t

• Overly duplicate coverage from the Unit tests i.e. most business logic will call the persistence layer to load data, process this, and call the persistence layer again to save the results. Tests at this level only need to focus on the load and save
• Test every conceivable permutation of data
• Chain too many operations together – this should be covered by service-integration tests or end-to-end tests

Coming in Part 2…

I’ll try and post Part 2 soon, and this will cover topics such as Service-level Integration testing, Web-based testing and API testing. Part 3 will cover end-to-end (E2E) testing and BDD.

Testing Solr 4.3.X within Spring (An update to ZoomInfo’s InProcessSolrServer)

A recent contract I was working on had decided to use Solr to implement full-text search over a product catalogue for an e-commerce platform. Naturally we were approaching development with a TDD-mindset, and were keen to implement both Unit Tests for core business functionality, and also integration tests for for a more end-to-end style of testing. The primary application stack consists of Spring (Core, Data, MVC), MySQL and Solr 4.

Just a slight aside, but for anyone looking to implement full-text search the primary candidates are Solr and ElasticSearch. I won’t discuss the merits of either implementation further as it’s best to evaluate each in respect to your use cases (and here is an excellent resource to help you decide http://solr-vs-elasticsearch.com/) 

With our chosen frameworks and datastores we found the Unit testing relatively straight-forward, and decided to use JUnit (driven via the Maven surefire plugin), Mockito for mocking external dependencies (persistence layer, API calls etc), and PowerMock for the difficult mocking (for example, mocking static method calls of several reliable-but-decidedly-old-skool dependencies).

Integration testing was also relatively easy to setup – we chose to again drive tests via JUnit (this time via the failsafe plugin), and use Spring’s @ContextConfiguration and AbstractTransactionalJUnit4SpringContextTests to manage injected sub-components (@Autowires etc) and instantiate various parts of the application for testing, and we also ran an embedded H2 database to allow realistic simulation of a SQL datastore (just an aside, in ~99% of ‘standard’ use cases I have found H2 to behave identically to MySQL, but there are a couple of corner cases to watch out for – this will be another blog post :))

The Problem – How do we run an embedded Solr?

When we first started using Solr 4 we naturally wanted to create integration tests running against this datastore, and we wanted to run this in the same manner as we did with H2 – executing as a light-weight in-memory (embedded) process that we could create, pre-load, and destroy relatively quickly.

We soon found the EmbeddedSolrServer Class distributed within the Solr package, and although useful it didn’t fit in exactly with the way we wanted to design and deploy the Solr communication layer within our Spring application. For production use we wanted to instantiate a SolrServer bean for which we supply the target endpoint on the network (and under the hood this SolrServer bean would actually be instantiated using a custom HttpSolrServer Class). We needed a way to create an ’embedded’ version that implemented the SolrServer interface, but also allowed us to override the Solr config and data directory (to load pre-canned indexes etc)

After a fair bit of searching we stumbled over ZoomInfo’s excellent blog in which they had shared their version of an embedded SolrServer that could easily be exposed as a Spring bean. They called the Class the InProcessSolrServer

We would like to offer many thanks to ZoomInfo for sharing there great work, and this Class provided us with many months of good service. However, with the latest releases of Solr (4.2 +) ZoomInfo’s InProcessSolrServer will no longer compile due to an interface change within the Solr internals.

In the spirit of sharing the wealth I wanted to blog an update to the original ZoomInfo code, which addresses the interface change, and I’ve also included the Spring scaffolding in the gist below to give you an idea of how we run this code.

	package uk.co.taidev.solrtesting.solr;
	import com.google.common.base.Throwables;
	import com.google.common.io.Files;
	import com.iat.compassmassive.normalisedproductloader.springutils.SpringProfileName;
	import org.apache.solr.client.solrj.SolrRequest;
	import org.apache.solr.client.solrj.SolrServer;
	import org.apache.solr.client.solrj.SolrServerException;
	import org.apache.solr.client.solrj.embedded.EmbeddedSolrServer;
	import org.apache.solr.client.solrj.response.UpdateResponse;
	import org.apache.solr.common.util.NamedList;
	import org.apache.solr.core.CoreContainer;
	import org.apache.solr.core.CoreDescriptor;
	import org.apache.solr.core.SolrCore;
	import org.apache.solr.core.SolrResourceLoader;
	import org.apache.solr.schema.IndexSchema;
	import org.apache.solr.search.SolrIndexSearcher;
	import org.apache.solr.util.RefCounted;
	import org.slf4j.Logger;
	import org.slf4j.LoggerFactory;
	import org.springframework.context.annotation.Profile;
	import org.springframework.stereotype.Component;
	import java.io.Closeable;
	import java.io.File;
	import java.io.IOException;
	import java.util.Collection;
	/**
	* SolrServer sub-class that manages the life-cycle of an in-process(embedded) Solr server.
	* <p/>
	* Modified from original source provided by ZoomInfo @ http://browse.feedreader.com/c/ZoomInfo_Blog/12021683
	* <p/>
	* Required dependencies: Spring 3.2.X, Solr, 4.2.X+ (or 4.3.X), Guava 14+
	* <p/>
	* User: Daniel Bryant
	* Date: 01/07/13
	*/
	@Component
	@Profile(SpringProfileName.DEVELOPMENT)
	public class InProcessSolrServer extends SolrServer implements Closeable {
	//
	//------------------ static -------------------------
	//
	private static final Logger LOGGER = LoggerFactory.getLogger(InProcessSolrServer.class);
	private static final String DEFAULT_SOLR_HOME_DIR_PATH = "./src/test/resources/solr/";
	//
	//------------------ instance-------------------------
	//
	private File solrHomeDir = null;
	private File dataDir = null;
	private SolrServer delegate = null;
	private transient SolrCore core = null;
	//
	//------------------ constructor -------------------------
	//
	/**
	* Create an InProcessSolrServer using the default Solr Home Directory.
	*/
	public InProcessSolrServer() {
	this(DEFAULT_SOLR_HOME_DIR_PATH);
	}
	/**
	* Create an InProcessSolrServer using the specified Solr Home Directory and a Solr Data Directory placed
	* beneath the system's temporary directory (as defined by the Guava method Files.createTempDir()).
	*
	* @param solrHomeDirPath path to Solr Root Directory
	*/
	public InProcessSolrServer(String solrHomeDirPath) {
	try {
	System.setProperty("solr.solr.home", solrHomeDirPath);
	System.setProperty("solr.data.dir", Files.createTempDir().getAbsolutePath());
	CoreContainer.Initializer initializer = new CoreContainer.Initializer();
	CoreContainer coreContainer = initializer.initialize();
	delegate = new EmbeddedSolrServer(coreContainer, "");
	} catch (Exception e) {
	throw Throwables.propagate(e);
	}
	}
	//
	//------------------ public -------------------------
	//
	/**
	* This method passes all queries and indexing events on to an in-process delegate.
	*
	* @param req Solr Request
	* @return NamedList
	* @throws SolrServerException if an error occurs when processing the request
	* @throws IOException if an IOException occurs when processing the request
	*/
	@Override
	public NamedList<Object> request(final SolrRequest req) throws SolrServerException, IOException {
	try {
	return getDelegate().request(req);
	} catch (final Exception e) {
	Throwables.propagateIfInstanceOf(e, SolrServerException.class);
	Throwables.propagateIfInstanceOf(e, IOException.class);
	throw Throwables.propagate(e);
	}
	}
	/**
	* Closes the Solr Core.
	*/
	@Override
	public synchronized void close() {
	if (core != null) {
	core.close();
	core = null;
	}
	}
	/**
	* SolrIndexSearcher adds schema awareness and caching functionality over the Lucene IndexSearcher.
	* http://lucene.apache.org/solr/normalisedproductloader/org/apache/solr/search/SolrIndexSearcher.html
	*
	* @return RefCounted SolrIndexSearcher
	* @throws SolrServerException
	*/
	public RefCounted<SolrIndexSearcher> getIndexSearcher() throws SolrServerException {
	getDelegate(); // force the delegate to be created
	return core.getSearcher();
	}
	/**
	* Returns the index schema used by this Solr server.
	*
	* @return delegate SolrServer primary core IndexSchema
	* @throws SolrServerException
	*/
	public IndexSchema getIndexSchema() throws SolrServerException {
	getDelegate(); // force the delegate to be created
	return core.getSchema();
	}
	/**
	* Prepares this SolrServer for shutdown.
	*/
	@Override
	public void shutdown() {
	LOGGER.debug("shutdown entry...");
	close();
	LOGGER.debug("...core closed...");
	}
	@Override
	public UpdateResponse addBeans(Collection<?> beans) throws SolrServerException, IOException {
	UpdateResponse updateResponse = super.addBeans(beans);
	super.commit();
	return updateResponse;
	}
	@Override
	public UpdateResponse deleteByQuery(String query) throws SolrServerException, IOException {
	UpdateResponse updateResponse = super.deleteByQuery(query);
	super.commit();
	return updateResponse;
	}
	//
	//------------------ protected -------------------------
	//
	@Override
	@SuppressWarnings("FinalizeDeclaration")
	protected void finalize() throws Throwable {
	close();
	super.finalize();
	}
	//
	//------------------ private -------------------------
	//
	/**
	* This method creates an in-process Solr server that otherwise behaves just as expected.
	*/
	private synchronized SolrServer getDelegate() throws SolrServerException {
	if (delegate != null) {
	return delegate;
	}
	try {
	CoreContainer container = new CoreContainer(SolrResourceLoader.locateSolrHome());
	CoreDescriptor descriptor = new CoreDescriptor(container, "core1", solrHomeDir.getCanonicalPath());
	core = container.create(descriptor);
	container.register("core1", core, false);
	delegate = new EmbeddedSolrServer(container, "core1");
	return delegate;
	} catch (IOException ex) {
	throw new SolrServerException(ex);
	}
	}
	/**
	* Sets the Solr root directory. In Solr’s documentation, this is generally referred to as "/solr-root". The "conf"
	* directory (containing schema, stopwords, synonyms etc) will be a subdirectory of this.
	*
	* @param solrHomeDir Solr 'Home Directory'
	*/
	private void setSolrHomeDir(final File solrHomeDir) {
	this.solrHomeDir = solrHomeDir;
	System.setProperty("solr.home", solrHomeDir.getPath());
	if (this.dataDir == null) {
	setDataDir(new File(solrHomeDir, "data"));
	}
	}
	/**
	* Sets the Solr data directory. This is the parent directory of the "index" and "spellchecker" directories.
	*
	* @param dataDir Solr 'Data Directory'
	*/
	private void setDataDir(final File dataDir) {
	this.dataDir = dataDir;
	System.setProperty("solr.data.dir", dataDir.getPath());
	}
	}

view raw InProcessSolrServer.java hosted with ❤ by GitHub

I hope this helps, and if you have any questions then please feel free to comment or tweet 🙂

[Book Review] Spring Data

Spring Data – by Mark Pollack et al

TLDR:  If you are working with Spring Data on a daily basis and want a complete and thorough overview of the framework then this book is all you will need. It covers all aspects of Spring Data without being overly verbose, and even if you have used Spring Data quite a lot already (like me), then I still believe you’ll discover something useful from this book. You will also find bonus chapters in context with Spring Data on Spring Roo, the REST repo exporter (very cool!), ‘Big Data’ via Hadoop, Pig, Hive and Spring Batch/Integration, and also coverage of GemFire.

I’ve been working professionally with Spring Data for quite some time now, both for ‘old skool’ RDBMS and also a lot of NoSQL (primarily MongoDB and Redis). The company I was working for at the time the Spring Data projects were approaching release were somewhat early-adopters, and in combination with the fact that their applications were firmly rooted in Spring made the decision to use this framework an easy choice. After some initial problems, which should be expected with a new technology (such as config issues and incompatibly between libraries bundled in JARs etc), Spring Data has provided a massive boost to productivity, and it is now my de facto choice when implementing persistence within Spring.

About the book itself: The first few chapters provide a great introduction to Spring Data, and describe the key motivations and techniques behind the framework. If you are simply modifying an already configured Spring Data app then this is all you need (but please do keep reading to learn more!). The next few chapters cover integration with an RDBMS, and also the popular NoSQL implementations – MongoDB, Neo4j and Redis. If you are working in one specific technology then reading the corresponding chapter will get you up and running quickly. Although Spring Data provides a common abstraction layer, it allows datastore-specific functionality to bleed through the interfaces (which is a good thing in my opinion, as it allows you to leverage specific features and strengths of the underlying technology), and this book will provide an excellent grounding and explanation of key concepts within each underlying datastore technology so that you can become productive quickly. Of course, you can also head over to the Spring Source website to learn the really advanced stuff (if you want to).

Part 4 of the book covers several interesting advanced features of the framework, such as using Spring Roo to auto-generate repository code, and also a brief guide on how to use the REST Repository Exporter. Metaprogamming and RAD tools like Spring Roo (and web-frameworks such as Grails and Play) are becoming increasingly popular in the industry, and so this chapter is a nice addition to the book. The REST exporter is also a very cool feature, and essentially allows you to expose CRUD functionality on your repositories via a REST interface. For anyone building a SOA-based app (or using micro-services etc) then encapsulating datastores and exposing simply functionality via a well defined HTTP-based API is very cool.

The final two parts of the book provide detailed coverage of using Spring Data to work with ‘Big Data’ through the use of Apache Hadoop, Spring Batch, Spring Integration and GemFire. Although this content wasn’t relevant to my initial decision to buy the book the chapters are a complete bonus in my opinion, and upon reading them I was even more happy with my purchase. The content provided is obviously quite high-level (as Big Data is a huge topic, no pun intended :)), but has enough detail to get you up and running with some Hadoop Jobs and Hive and Pig etc, which is a great skill to add to your CV.

I chose this book over the only other real competition for Spring Data coverage, Petri Kainulainen’s Spring Data, purely because this book offered more content. Obviously the book under review has more pages, ~280 vs ~160, but more importantly it covers a greater amount of topics, and Petri’s book focuses primarily on Redis (for which I was already familiar with). My main motivation for buying a Spring Data book was to learn about the ‘tips and tricks’, and I think either book would have met this need, but the coverage of other NoSQL technologies in the book under review, and the bonus chapters on Big Data technologies swayed my final decision. Now that I’ve read the book I am very happy with the decision.

In summary: This book will be all you need to master Spring Data, from key concepts to advanced usage. You’ll learn all of the ‘tips and tricks’ along the way, and also become familiar with Spring Roo, the REST repo exporter and fundamental techniques within Spring Data’s ‘Big Data’ processing (Hadoop, Spring Batch/Integration etc). I would recommend the book to any Spring developer, even one like myself who is happy learning about Spring from the excellent Spring Source website This book is a little more ‘polished’ than the Spring Source docs, and also provides concepts in well-structured and bit-sized chunks of information.

Click Here to buy ‘Spring Data‘ on Amazon (This is a sponsored link. Please click through and help a fellow developer to buy some more books!  )

[Book Review] Spring in Practice

Spring in Practice – by Willie Wheeler and Joshua White

TLDR; This is an excellent and comprehensive guide to advanced usage of the Spring framework. For anyone who is looking to further their knowledge gained from several years of Spring development in the trenches, this book will pay dividends. Although a Spring novice may be able to learn about Spring from this book, I would recommend picking up a copy of Spring in Action first, as the ‘In practice’ books can be quite fast paced!

As a seasoned Java developer I have been working with the Spring framework for many years now. One of the first Spring books I read was Spring in Action, and in combination with Java Persistence with Hibernate  this book has helped me complete many successful projects (I seriously owe the authors a few beers!). From the grounding provided in these book, and in combination with the excellent Spring Source website, I have been able to explore and develop my skills as the Spring framework has expanded – for example, the Spring Data project is now my go-to framework for all things NoSQL related. However, I always enjoy learning from advanced Spring practitioners and also from reading stories about real-world use and abuse of the framework, and I have yet to find a good book that meets this need – until now. ‘Spring in Practice’ satisfies this gap in the market perfectly.

The book is ~500 pages, and it manages to cram in a lot of content. Advanced usage of all the main Spring components is covered, and covered well. The first nine chapters provide a great grounding and advanced look at topics such as data persistence (ORM), Spring MVC, Web Flow and Security. The remaining chapters deep-dive into topics such as Integration Testing and Enterprise Integration (REST, RabbitMQ and IMAP integration etc), and really focus on how to write good (high-quality) code for the common but difficult tasks.

As the title suggests, the book’s focus is very much about practical usage of Spring. It’s not quite in the ‘cookbook’ style you may have seen with other books, but IMHO, this book is better organised for general learning (i.e. reading the book from cover to cover). The obvious advantage with a cookbook style reference is that it’s easy to cherry-pick solutions to problems, but I find that cookbooks can be difficult to read through if you simply want to learn. ‘Spring in Practice’ is logically structured, the book is nicely paced for the advanced developer, and the discussions of real-world problems and the related code sample solutions seek to further your knowledge and encourage exploration of Spring.

As mentioned above, I have worked with Spring for several years, but this book has taught me lot of new tricks – there’s nothing like finding a section of the book that leads to a ‘no way, Spring does that?’ moment 🙂 The author’s clearly have their own style of developing in Spring, and I personally would chose to do some things differently (e.g. I code the production of XML/JSON differently), but I can’t argue that what they’ve done isn’t best practice, and with a framework as large and wide-scoped as Spring, there is bound to be many approaches to do the same thing.

In summary, this is an excellent book, and one that should be on the bookshelf of any serious Spring developer. It will help deepen knowledge gained from ‘Spring in Action’, and also help to augment skills honed from time in the development trenches. I can almost guarantee that anyone who picks up a copy of this book, no matter how advanced they are, will learn something new. As you’ve no doubt guessed by now, I highly recommend this book, and I would like to offer my congratulations to the authors and Manning for writing a book which has long been needed by advanced Spring practitioners!

Click Here to buy ‘Java Persistence with Hibernate‘ on Amazon (This is a sponsored link. Please click through and help a fellow developer to buy some more books!  )