July 29, 2015

Integration Frameworks: How Everything Works Together

Due to the diversity of this topic this will be the first post of a series. As soon as the other posts are available, they will be linked here.

These are the future topics to be covered:
  1. Integration Frameworks: general introduction and presentation of the basic concepts.
  2. Apache Camel: core concepts.
  3. Error Handling: production-ready error handling can be hard to master. We will provide solutions and concepts that are already live and have proven successful.
  4. Writing Custom Camel Components: sometimes there are use cases that are not covered by standard components. Custom Camel Components can be a solution that integrates seamlessly with the existing framework but still fits new requirements.
These topics, however, may still change and if you have any suggestions for topics that we should cover, feel free to contact us at: rene.schakmann@willhaben.at

Integration Frameworks

The integration of various services is mostly a very repetitive task. Different aspects like error handling, fault tolerance and retrying after failure have to be considered. Nevertheless, due to the huge amount of different systems and their interfaces there is no standard solution available to tackle these tasks as one. They do however share common patterns that can be extracted and used. A frequent process in Software Engineering, using patterns can help provide a common vocabulary in this domain. Enterprise Integration Patterns are a widely accepted standard in this area. They are implemented by frameworks like Apache Camel, Spring Integration or Mule ESB. Depending on the product only a part of the pattern is actually used, but they all share the common terms. Before we go into detail here, however, we should take a closer look at integration frameworks in the overall IT landscape.

Choose your tool

Depending on the size and complexity of your IT infrastructure, and mainly on your budget, there are different solutions available for system integration. The figure shown below gives a rough idea of different levels of complexity: the simplest kind of system integration, (1) a basic integration framework, (2) the more powerful Enterprise Service Bus and the most complex and powerful solution, (3) the Integration Suite. The next section will provide a short introduction to the differences between these three approaches.

different levels of system integration (higher level basing on lower level)

(1) Integration Framework

Integration Frameworks are often libraries that can be included in other projects. They provide interfaces, for example fluent APIs like Camel, to define the flow of data between systems. Due to the usage of patterns that are known by many developers they reduce complexity and provide a fast solution for integrating services. Using e.g. connectors, converts and splitters, data can be fetched from one source, transformed and sent to another service. Tasks that are similar between all kinds of integration projects (like error handling and retrying after an error) are supported without custom code. Most of these integration frameworks are open-source.

(2) Enterprise Services Bus

Normally, an ESB provides additional features for e.g. deployment and monitoring. Often, graphical editors are available to integrate systems, giving them a sort of 'drag and drop' feeling. The actual code is generated based on these graphical representations. ESB aims at a higher level of integration than integration frameworks and tends to have a strong commercial background and support.

(3) Integration Suite

The ultimate set of functionality in integration suites varies between vendors, but in general they provide features for Business Process Management and Monitoring as well as features for data management. The level of integration is even higher than with an ESB.

If you are interested in more information on this topic you can find a detailed analysis on each of the approaches here.

Apache Camel

Originally facing a variety of choices, the IT team at willhaben.at decided to use Apache Camel for their integration. Due to its great flexibility and huge amount of different adapters (e.g. HTTP, CSV, JDBC, etc) it is a perfect fit for our IT landscape. It entails a relatively steep learning curve and may not be as transparent as it should be, especially at the initial stages of usage. But learning how to use this framework pays off very soon.

Before going into further detail on the process itself, let us first look at an example that illustrates how and when Camel may come in handy. Let's start with this simple use case that is shown in the next figure:

sequential diagramme (API request)

In this example, we are requesting some weather data from a REST API framework that returns JSON data. Then we want to parse it, extract some data and store this data in the file system. Without the usage of an integration framework like Camel even a simple task like this one would require a lot of coding. Consequently, using the Apache Camel DSL, this task can simply be written as shown in the following snippet:

unmarshal().json(JsonLibrary.Jackson, WeatherInformation.class).
process(exchange -> {
   WeatherInformation weatherInformation =
   exchange.getIn().getBody(WeatherInformation.class);                 exchange.getOut().setBody(
     weatherInformation.getName() + "," +
     weatherInformation.getWind().getSpeed()  +
to("log:out", "mock:test");

A full working example of this code can be found in our willhaben github repository:

https://github.com/willhaben/blog - project camel-weatherservice

Using the timer component we start this route automatically, without any external event. We fetch the JSON from the rest API and convert the response to POJO. This way, we can simply access the properties "name" and "wind speed" and write them to an output file. As you can see, error handling is still missing in this example. However, this topic will be covered separately in a later article.

Declaring tasks using a DSL (Domain Specific Language) leads to significally reduced code and higher maintainability. Each of the used Camel components has a set of properties that can be set to adapt to the behaviour, as shown in the "fileExist=Append" property. These configuration options are well covered within the Camel documentation.

Basics Concepts of Camel

The figure belows gives a short overview of the major concepts of Apache Camel, including the used vocabulary and patterns, as well as other Integration Frameworks that implement these integration patterns). The basic idea is that a message is routed from one end point, which may be an application, to another end point. Inbetween, various tasks such as transformation or content enrichment can take place.

Source: http://www.enterpriseintegrationpatterns.com/eaipatterns.html

As you can see, Camel allows for a variety of useful operations and processes. These will be the core topic of our next blog article in this series.


Integration frameworks can tackle a lot of tasks and provide various benefits regarding system integration. In an enterprise context, where maintainability is a big issue, these frameworks can reduce software costs throughout the entire usage lifecycle: less time is needed to develop and understand already existing software based on a common framework with known patterns. As I have illustrated above, moving on from initial stages to becoming productive may take a bit of time when working with an integration framework, but it certainly pays off at the end.

1 comment: