Can application integration be pre-packaged?

The concept of pre-packaged integrations is nothing new. For years, integration vendors have tried to sell pre-packaged application integrations with various degrees of failure. You might presume that because of these failures, application integration cannot be pre-packaged and but you would be wrong.

Most of the failures were due to:

1. Complexity of underlying integration platform used

The 1.0 integration products were (are) so complex and convoluted that trying to make changes costs the same as writing from scratch. So why pre-package if it's going to cost the same amount as starting from scratch.

2. Over architecture of the solution on part of the developers

You see this over and over again. Developers with little customer implementation experience are charged with developing a pre-packaged integration solution. They start by trying to conceive every possible scenario at every customer. From there, there's no coming back. You build and build till you think you have every one covered. Only problem is that now you need a 30 day training course to understand all the moving parts.

3. Combination of 1 and 2

Integrations can absolutely be pre-packaged. The question that needs to be answered is "To what extent?". The answer is going to depend on 1st the use case, 2nd the applications being integrated, and 3rd the experience of the developer.

Some use cases lend themselves to be 90-100% pre-packaged or "canned". For example, user synchronization. The objects are usually very well defined and tend not be customized per implementation.

Some use cases reach the 70-80% level. Customer master sync is good example. For the most part the customer objects on each side will be standard but you'll find various number of custom fields that need to be included. The integration platform should provide easy method for providing the additional required mappings.

You might argue that applications like SAP which provide infinite amounts of customization capability would be the exceptions to the case but that's not necessarily true. Generally, standard objects are only modified to include new custom fields.

Most use cases can be pre-packaged to at least 50%. The trick is not to over think or over build. Trying to conceive of every scenario and account for it will be impossible. As long as the underlying integration platform provides for easy modifications at implementation time, there's no need to over architect the solution. You can deal with the one-offs at implementation time.

Don't over architect your application integrations!

Simple and agile should be your foremost design considerations in integration. Trying to get applications with different functionality and schemas to talk to each other via different protocols or APIs is complicated  enough, you don't need to add to it. Focus on the business reason why you're integrating the applications. If in doubt, ask your business user or CIO.

Keep these guidelines in mind as you embark on your next integration project:

• If point-to-point works and its all you need, don't spend time and effort designing a hub-n-spoke model and an all encompassing uber canonical intermediary model. Unless you're the fortune 100 (50?) and have multiple identical applications across your enterprise, most of your integrations will be point-to-point. That's true even if you force a messaging bus (queue) in the middle. Don't confuse your underlying software/hardware architecture with your actual integration architecture. Everything may flow through a single hub (or server) but you can still just have a bunch of point-to-point integrations. If object A from application A only needs to be integrated with object A of application B, then stop there. If you have an integration platform worth it's salt, you can always come back and evolve your integrations.
  

• Avoid custom code like your job depends on it, which it will. Custom code is the path to the dark side and the famous Gartner Spaghetti diagram. It's not point-to-point integration that gets you in trouble, it's the lack of a unifying integration platform that leads to Spaghetti. Using an integration platform will give you consistency across design and management of your integrations. If you're being tempted by the dark side to use custom code, remember that running tail on a log file doesn't consitute management and monitoring. If you're getting tempted to build your own awesome integration platform, go back to your business user or CIO and ask them what they wanted again.
  

When selecting off-the-shelf integration software, keep the following in mind:

• Pick an integration platform that can deliver 80% of what you need out-of-the-box. Most enterprise application integration problems fall into the 80% category. The other 20% edge cases may need a different solution. Platforms and vendors that focus on those edge cases to differentiate themselves tend to provide the same model for solving all problems, with great pain and difficulty!
  
• Make sure what ever solution you choose has solid coverage for the 3 basic types of application integration:

• Data Synchronization - One time migrations or ongoing synchronizations.
  
• Process Integration - Data syncs only get you so far, being able to write integration flow logic tying different business processes together is also important.
  
• UI Integration (aka Mashups) - Perhaps the most under rated method of integrating applications. If all you need is visibility to data in another system, why synchronize the data. If you don't need to use the data in the target application for processing or reporting, then don't synchronize it. Most applications these days provide browser-based interfaces with the ability to embed iframes or other methods of incorporating external data sources. "... mashups can deliver an 80% solution at 20% the cost (or less) " (source: http://blog.programmableweb.com/2009/07/23/enterprise-mashups-continue-to-gain-momentum-as-part-of-enterprise-20/).
  

• Apply standards when they make sense and avoid them when they only serve to complicate solutions. Having adopted SOA doesn't mean that every thing needs to be exposed as a service. It certainly doesn't mean that you should force adoption of overly complicated WS* "standards". Stick with the basics that you really need (vs what would be cool). You'll find varying levels of WS* "standards" adoption in the application market place. SOAP-based or REST-ful? What ever works. You'll find that you'll have to be able to support all kinds of different "standards". Make sure your integration platform is flexible.