Application Echo: Common Bi-directional Synchronization Side Effect

You set up 2 integration flows running in opposite directions that capture change events in 2 applications and replicating it to the other. Sounds easy enough, till you deploy the integrations and make a single change in one application. The 1st integration flow gets notified of the change and updates its target application. As a result, the 2nd integration flow gets notified of the change and updates its target application. Which causes the 1st integration flow to see the change....so on and so forth till you finally pull the plug and break the cycle. How do you stop this infinite application echo from happening?

You can implement a method which queries the target system 1st to see if the data in the target is different from the source and only apply the update if it is. While this seems like a good approach, it's not very efficient. It requires an additional query and field by field comparison for each integration flow which can be the cause of performance issues and can certainly put additional load on the target applications.

A simpler and less costly method would be to create specific user credential on each application that is only used by the integration flows to login. When an update triggers the flows, a simple check to see who last modified the record will let the process know to proceed or to stop.

What's a connector?

Connector is one of those over used words when talking application integration these days. Depending on who you ask, you’ll get different interpretations.

Application developers typically talk about connectors as the piece in between that connects 2 different applications together. For example, Salesforce to SAP connector which was commissioned by Salesforce to sync accounts between the 2 applications. When application developers refer to connectors, they usually are talking about specific point-to-point integrations that only implement 1 or more limited use cases.

When application integration developers talk about connectors, they are only talking about the piece that makes the calls to one application. It usually involves wrapping the APIs exposed by an application and exposing it in a uniform or standard way within the integration platform. It’s usually (but not always) use case or module  independent. In case of Salesforce and SAP, you’d have 1 Saleforce and 1 SAP connector. If you build an account to customer sync, that’s just called the integration which will use the connectors as means to get data in and out of the applications.

Connectors are some times referred to as adapters but that doesn’t change the meaning.

By the way, if you're looking for the Salesforce to SAP connector, it's no longer available from Salesforce.

Cloud-based Message Queueing and Persistence

I think given the choice, most developers rather deal with synchronous vs. asynchronous processes. It’s just much easier to wait for an immediate response and continue down your path vs. having to queue requests and set up a secondary process to poll for replies, perhaps correlate them to the original requests and then process them. However, since you can’t always control every application you need to work with, you will find that having a place to temporarily park messages for either later processing or pick up is necessary. You some times need UPS to retry delivering your package and some times you rather go pick it up at the distribution center.

As an example, if you ever worked with Salesforce’s Workflow and Outbound messaging feature, you’ve probably had to figure out a way to deal with storing the Outbound Notification from Salesforce while your Great Plains , Oracle EBS,... instance is down for maintenance. Salesforce’s Outbound messaging service will only retry a finite amount of times and then the notification is lost so having a place to temporarily store the notification until your application is back on line can be quite necessary.

For most use cases, you have plenty of options for persisting your messages, from simple database tables to more sophisticated message queueing software. If you’re looking to reduce internal IT costs and headaches, Amazon Simple Queue Service (SQS) can be a good option too. It’s simple to set up, it’s fully managed, and it’s very inexpensive. You have several choices of geographic location for the service, US East/West, EU, and APAC. The API is straightforward enough and since it uses standard Cloud-based protocols (RESTFul and SOAP WS over HTTPS), you can be up and running in no time. 

Amazon SQS uses a distributed architecture with multiple queue nodes and as most Cloud-based services go, it some times takes a little time for all the nodes to sync up and have the expected response to your request. Here are some of the behavior I noticed while playing around.

• There was almost always a delay (10-30secs) in availability of newly created queues. I would create a queue and then use the list queue service to see if it was there. Same was true when I deleted queues. They showed up in the list of queues for 10-30 seconds after I had a confirmation of deletion.
• Message retrieval seemed some what random. I would send a number of messages in a row and then try to retrieve them. I would get my messages back but in what seemed random groupings. I never lost a message so that was great news.

Overall, SQS was very reliable and as long as you don’t care about ordered delivery and you don’t that have messages that are > 8KB, it’s a viable Cloud-based choice.

Microsoft’s Azure platform offers a similar service which I haven’t played around with but it looks almost identical to Amazon’s SQS. I guess borrowing is the best form of flattery. The operations seem to be identical (1:1) to SQS and here’s a direct quote from their documentation: 

"A queue can contain an unlimited number of messages, each of which can be up to 8 KB in size. Messages are generally added to the end of the queue and retrieved from the front of the queue, although first in, first out (FIFO) behavior is not guaranteed.“ 

So pretty much the same as Amazon's SQS. It's great when your competition doubles as your PM.

If you’re looking for a Cloud-based message persistence service to complement your Cloud-based integration platform, Amazon SQS and (probably) Microsoft’s Azure both offer worthwhile solutions.

Collaboration in Web 2.0 Era is changing. Are you keeping up?

If you haven't noticed lately, the way we communicate is evolving. Phone calls and emails are losing ground to IM, SMS, Tweets, and Wall Posts. This is true in both social and business networks.

While Tweets and Wall Posts are great (only if you're insanely bored) for keeping up with every minute action of friends and family and have been used by businesses for B2C interactions, they lack the privacy and control required for inter-enterprise or B2B collaboration. 

Also, when it comes to the business communication and collaboration, we've been missing an important collaborator, namely, the business applications. That's right, we need Salesforce.com, Eloqua, SAP, Oracle Apps,.... to proactively communicate events and collaborate with us too. It's a symbiotic relationship that up until now has not been working very smoothly.

To solve this problem, Salesforce introduced their latest and greatest innovation, Chatter, at Dreamforce '09. Despite the lengthy introduction by it's passionate creator, Marc Benioff, it was clear that Salesforce is onto something that will have a big impact on how we work every day. With Chatter for Salesforce, you can keep track of any changes you subscribe to without having to set up complex workflow rules or repeatedly check the object(s) of interest for changes. Marc Benioff is admittedly copying Facebook and Twitter's model and applying it to the business world.

Seems like Lars Daalgard has his eye on the ball too. Last month, SuccessFactors acquired CubeTree, a business/social networking provider. Where as Chatter is tightly coupled and integrated into Salesforce, today, CubeTree is not application specific but offers a rich set of APIs that will allow for easy integration. I suspect that it will be natively integrated with the rest of the SFSF suite in the coming Qs.

While Chatter and (integrated) CubeTree will be great for all events occurring within their respective applications, they will require integration to other applications within the enterprise to complete the collaboration loop. And while monitoring data changes in applications can be easy straightforward, you'll need to know which corresponding object in Salesforce (as an example) to relate the change events too. Which means, you'll still need to synchronize your base objects before SAP can send you notifications via Chatter.

For business execution, you need collaboration between people and systems. For systems to collaborate, you'll need to be able to speak their language and translate to human speak.

Is Your Cloud Elastic or Plastic?

The discussions over multi vs. single tenant Cloud application design seems to be winding down with multi-tenancy clearly the winner. Although customers should frankly not care or be impacted by either architecture, vendors would be wise to follow a path that leads to easier maintenance and lower operational costs. As the trend in SaaS and Cloud computing puts downward pressure on license prices, application vendors need to adopt technologies and architectural models that result in the most cost effective manner to operate a Cloud application. That's assuming you have a for profit or at least against loss business model.

Now, it seems natural that the next big debate in Cloud computing architecture is going to be around elasticity vs plasticity. 

Is your Cloud computing infrastructure able to scale up or down based on demand? or Do you have to plan for peak demand and deploy enough hardware to make sure you have enough capacity to address your demand and SLAs?

If it's the latter, then it's like having a hotel keeping all the lights in every room on just in case some one checks in. How about turning the lights on only when you need them?! The amount of resources (aka moolah) wasted on running infrastructure constantly for peak demand can potentially negate any cost savings achieved through multi-tenancy.

The only arguments that seem to make any sense for plastic computing are with regards to SLAs and the latency involved in bringing on new hardware/software to support increases in demand. To that, I say...invest in some operational analytics. I'm sure if you can track your usage, you can see patterns of use and plan for them. For example, if you're running a sales or finance application, it seems natural that end of month/quarter would be the busiest times and beginning through mid-month, things are not so busy. Turn off the lights when you don't need them.

Can application integration be pre-packaged?

The concept of pre-packaged integrations is nothing new. For years, integration vendors have tried to sell pre-packaged application integrations with various degrees of failure. You might presume that because of these failures, application integration cannot be pre-packaged and but you would be wrong.

Most of the failures were due to:

1. Complexity of underlying integration platform used

The 1.0 integration products were (are) so complex and convoluted that trying to make changes costs the same as writing from scratch. So why pre-package if it's going to cost the same amount as starting from scratch.

2. Over architecture of the solution on part of the developers

You see this over and over again. Developers with little customer implementation experience are charged with developing a pre-packaged integration solution. They start by trying to conceive every possible scenario at every customer. From there, there's no coming back. You build and build till you think you have every one covered. Only problem is that now you need a 30 day training course to understand all the moving parts.

3. Combination of 1 and 2

Integrations can absolutely be pre-packaged. The question that needs to be answered is "To what extent?". The answer is going to depend on 1st the use case, 2nd the applications being integrated, and 3rd the experience of the developer.

Some use cases lend themselves to be 90-100% pre-packaged or "canned". For example, user synchronization. The objects are usually very well defined and tend not be customized per implementation.

Some use cases reach the 70-80% level. Customer master sync is good example. For the most part the customer objects on each side will be standard but you'll find various number of custom fields that need to be included. The integration platform should provide easy method for providing the additional required mappings.

You might argue that applications like SAP which provide infinite amounts of customization capability would be the exceptions to the case but that's not necessarily true. Generally, standard objects are only modified to include new custom fields.

Most use cases can be pre-packaged to at least 50%. The trick is not to over think or over build. Trying to conceive of every scenario and account for it will be impossible. As long as the underlying integration platform provides for easy modifications at implementation time, there's no need to over architect the solution. You can deal with the one-offs at implementation time.

Top 10 Indicators that you're a 1.0 Application Company!

10.	The last time you updated you're user interface, Windows 3.1 was your OS.
9.	Your UI is still Windows-based.
8.	You read "Only the Paranoid Survive" and have locked yourself in a room.
7.	Your API strategy is to expose your database tables and stored procedures.
6.	You think Cloud and SaaS are only being adopted by small companies with little to no money.
5.	You refer to your hosted instance of your application as SaaS.
4.	Your idea of rapid customization capability is to lock an engineer in the room with your SaaS hosted instance.
3.	You want to adopt a Cloud and SaaS strategy but are worried about eroding your high ASP.
2.	You don't have a Cloud strategy yet.
1.	You're Cloud strategy is to wait in your room till the fad is over.