ACORD is a Guideline NOT a Standard

Apologies to my readers who are not in the insurance industry and do not care about the ACORD XML "Standard". I have gabbed a lot about it lately. Perhaps this applies to other "industry standard" xml efforts. I know that it also applies to my experience with OAGIS. I also recently posted on canonical message format as well (related). I hope that this post and the comments that may ensue will put this topic to bed for a while.

While at the ACORD conference last week in Las Vegas, I challenged myself to interrogate every vendor, carrier, and ACORD employee on whether ACORD is a standard or a guideline.

To a person the consensus was that it is a guideline.

Perhaps I am just a stickler for words, but this really bothers me. I listened to executive after executive and industry analysts yapping and banging drums about "standards" and how everything is just great in the insurance world on the wire. I listened to the Gartner insurance dude spaz out about ACORD "standards" and then recommend that carriers fork it (excuse me, use ACORD as a guideline) in one sentence. The insurance analyst panel (Gartner, TowerGrouup, Celent - liked their guy Matthew Josefowicz the most)) was pretty comical - one of the more arrogant groups I have seen. Sniping at each other, challenging each other to dare to question the standards -- saying things like, "the standards have never been as close as they are today ...". They certainly all know more then I do about the insurance business, but someone needs to remind them that their PowerPoint presentations don't compile. They should take a look beneath the covers of a couple SOA projects and check to see what is really on the wire.

I talked to several aggregators and AMS reps who said that in their eyes ACORD is at best a guideline. They did say that since carrier xml is often ACORD-like and at least has some similarities, it makes things better then if everyone started from scratch. I can buy that - but that isn't standard.

I listened to a presentation from a reinsurance intermediary (Guy Carpenter) on an event driven impl. They forked the ACORD standard and have no regrets. By the way that was the best presentation of the show (even though it was called "Using ACORD XML Standards to Build Enterprise SOA"). I have a feeling that the presenter, Paul Fox didn't name it that.

I challenged some CIO level folks who were talking the standards bit. They quickly folded and agreed with me.

What bothered me the most about the ACORD conference was that there was no serious discussion about the realities of ACORD XML in practice today - no sessions, no critical thinking - zip. Everyone just seemed to wink and nod and say nice things about standards. Also, zero talk about 2.0.. Perhaps I just wasn’t in the right sessions.

Every sane person I spoke with is following the prune strategy accepted in 2.0.

I still believe that it is appropriate to use the ACORD guideline as an starting point for a insurance entity's canonical form - provided it is aggressively pruned and there is no requirement to keep up with future versions of the ACORD guideline. Most importantly, don't spread the illusion that your canonical form is "standard" -- that there will be some magic interop some day. There will not. This is ok. Owning your canonical form outright is far more important then conforming to a standard that doesn't really exist.

ACORD Conference - lots of vendor-pires & some that get it

Wow, lots of vendor-pires large and small here.

I had dinner last night with some folks from Skywire Software - we use one of their products, Insbridge.

What I like most about them is that they aren't a vendor-pires and aren't trying to be. They are SOA based and stay out of your way. I wish there were more vendors like this - that just offered service based software.

Instead, every vendor seems to want to invade your whole stack. I met with a couple of these vendors yesterday and told them straight - "I would never buy this - it will lock me in and make me miserable". Most of these vendors have crazy GUIs and scary proprietary scripties that are intended to make your life easier. "Easier" sadly also = lock in and esoteric skills that few developers want to acquire.

IMHO, SOA strategies that rely on lots of extensive clickie tool support are doomed to failure. Or at least lots of bites from various vendor-pires. I'm all for tool support - just not invasive tool support. As long as the tool isn't generating files with weird file extensions and isn't required to build a service, you are probably ok.

Insbridge. does have a clicky tool, but the way you call it at runtime is via a service. So the clicky is useful and contained - that works for me.

Update 17-JUL-06 - Now that it is possible to determine my employer by reviewing my blog posts, I must follow my employer's blog policy.

This is not an endorsement by Liberty Mutual, but my personal view. Vendors are NOT free to quote with attribution.

ACORD Conference

I, standards basher, am at the ACORD LOMA Insurance Systems Forum in Las Vegas.

The conference really starts tomorrow. I went to a technology forum run by insurance agents. It was fairly interesting. I have gone to several agencies and watched these people work. As the saying goes, "they don't call them independent agents for nothing". They all do business differently, use different tools (agency management systems (or none at all), aggregators, etc.), yet they want everything integrated and simple. Makes my life fun, what can I say.

At the agency technology forum, the word "standard" was tossed out there quite a bit. I kept muttering to myself, "what version?", "who's extensions"?

I had an interesting conversation with one of our aggregators in the conference hall. An aggregator is used by insurance agencies to do comparative rating. Basically, they just type in an insured's info once and click one button and get back rate quotes from N carriers (think Travelocity with airline tickets). If anyone should be benefiting from the industry standards that we all say have such a high payback with no questions asked, it should definitely be him. Guess what he told me? Every carrier that his system integrates with uses dramatically different XML on the wire. There is no "standard" XML that they send to each carrier.

Now I found that pretty funny.

He did say that it is nice because each carrier's XML has some resemblance, but then conceded, with some prodding, that the whole thing is b.s.

Anyway, I still hold out a slim bit of hope for Acord 2.0, but time will tell.

Don't get me wrong on standards - I think they are fine. I just get miffed when people blindly cite them as a panacea when I know under the hood what is going on isn't standard. I think we might get there someday with Acord XML - it just is going to take a long time IMHO.

EDA Lessons Learned - Performance

See EDA Lessons Learned for the list

For starters, specify a requirement or two on what acceptable performance is. Then, start measuring early. Better to know that you are way off the mark early on IMHO. Fixing services that are way too slow late in development or once they are in production often means rewriting the service from scratch in my experience (of course you still call it refactoring in meetings to make everyone feel better ;) ).

I have seen the most performance problems with Persistence.

Another thing to focus on early is how your services will scale. Our system resembles SEDA (Staged Event-Driven Architecture).

• Thread Safe - in our system that is built on top of JMS at the core, we use a JMS ServerSessionPool impl that makes adding threads (sessions) to a service configurable. This is a great way to scale and throttle a service. Sadly, we deal with some obscure stuff that isn't multithreaded - so we have to configure just one thread. For most services, however, you can have 10+ threads waiting to process events
• Competing Consumers & Shared Subscriptions - Anyone familiar with JMS understands competing consumers with Queues. We use SonicMQ, which, also offers this concept with Topics. We use these heavily. It is great because you can spread processing across machines. You can have 1 logical service running on N machines. When it is flooded with events, the processing is evenly distributed across the logical instances of the service.

Another area to focus on from the start is service granularity (the black art of SOA/EDA) - this is a lot harder to get right then a lot of people think. Especially when you are dealing with many services. Its like anything of sufficient complexity - there are 3-5 ways to do it right and 50 ways to do it wrong - pick one of the right ways.

Last thing I'll mention is async processing. This is huge in an EDA system - basically everything is async. But some developer's little minds don't like async and want to make it sync. You must find these people early and get them off the project. Just kidding (kinda). We had a couple problems with performance like this. Our login for instance was a dog at one point. This was because it was sync and the granularity of service calls was awful. So we "refactored" (where "refactor" means tore out old code and stomped on it and called it names, then wrote it again from scratch). We rewrote it in a way that was async. When a user logs in, an event is generated that the system responds to. Immediately after the login event is produced the first page of the UI is displayed. Other events in the backend are produced that the UI is interested in. So it listens for these events. The key is with the rewrite, we don't make that process sync. When the events the UI needs are produced, the UI receives them async.

EDA Lessons Learned - Scaffolding

See EDA Lessons Learned for the list

Scaffolding is useful all over the place in software. EDA and SOA really can benefit from it.

Scaffolding is especially useful in large new development efforts. Services reach code complete at different times. You are totally hamstrung if you wait until all services are complete to start testing. The scaffolding (i.e., mock services) can be used to simulate the services that are still in development.

Once developed, scaffolding can be useful when a service that is not available. If you don't have scaffolding, development and testing can grind to a halt while you wait for the service to come back up.

Scaffolding certainly doesn't come for free - you have to develop it like any other service & it can be every bit as complex as the real service. Its often worth the investment though. You don't need a scaffolding impl for every service you write. Mock services are particularly helpful for services that wrap a volatile resource or a service that is outside of your control (e.g., different org. that has a different development schedule).

A SOA or EDA that provides good location transparency makes swapping in/out scaffolding easier. For our EDA style system we just disable the real service and enable the mock service. The mock service subscribes to the same JMS Destinations, and produces the same event types. This is great because you don't have to modify anything on the clients to swap in the scaffolding. You can achieve the same thing with UDDI or RSS - I do really like the EDA style though - when you are working in an event cloud you get extreme decoupling which rocks.

SOAP vs. REST Pointless?

Steve Jones thinks that SOAP vs. REST is pointless. I assume from his post that he thinks that POX is pointless too.

I can understand that opinion. I think everyone in IT integration is frustrated that every wave that comes by that promises to give us massive reuse fails.

The good news is that IMHO services are here to stay. At the very least, they will get a fair shake. Especially with success stories like this from Amazon.

Regardless if Steve thinks the arguments are pointless or not, the WS-* standards aren't there yet. And SOAP really doesn't buy you all that much out of the box. Sure maybe if all the higher level standards amount to something it might, but I'm not holding my breath.

I think that dissent is important.

Steve argues as Anne Thomas Manes does that the tools will save us. He even cites XDoclet as an example. Funny that I use that as an example on why tools are not the answer.

Like I said, the good news is that services are here to stay. How about we just go back to the drawing board on some of the things that are broken instead of slogging on with the 60+ specs? I don't see that happening. There are too many vendor-pires too heavily invested at this point.

The only answer is enough simple OSS rebel frameworks and hopefully enlightened vendors that treat POX, REST, and whatever innovative ideas come next as first class citizens. The truth is there is massive premature standardization occurring. We might get there some day, but you can't just beat people into submission. IT Integration isn't the commodity that Steve describes. At least not where I work.

A big complaint I have against drinking the SOAP/WS-* Kool-Aid is I haven't seen any real success stories on following the WS-* approach with heavy integration. There are many around SOA, but strangely the specifics of what is on the wire are often left out. Admittedly I may be missing these - please let me know if you know of any.

In general I think that the concepts of SOA and EDA are what are really important today. If you can break things down at the proper level of granularity and use whatever you want on the wire (try a couple!), you will be fine in the end. People including me in the past do a lot of hand waving about standards, service contracts, tool support and what not which are important - there just not as important as you think - at least today.

Brighter Sun

Good to hear via James Governor that things are looking up for Sun.

I have largely made my living off of Sun's innovation (Java) for the past 10 years.

It has bugged me that they haven't open sourced Java, but sounds like they have decided to do so now eventually.

And sounds like Jonathan Schwartz is bringing a much needed new direction to the company.

I think that we all had enough of the negative Sun - I'm looking forward to a brighter Sun (impressive come back - you just have to love OSS).

Service Contracts = Hall of Mirrors

Nice post on service versioning by Dare Obasanjo. He refers to posts by Tim Ewald extensively.

I generally agree with their conclusions. Sadly, it demonstrates that "design by contract" with web services really is a hall of mirrors.

I beat the "contract first" drum as loud as anyone. The truth is, however, at runtime, the true contract between my services is very loose. This is due to lots of things - complexity of WSDL (I try to avoid at all costs), over engineering of XSD, industry standard schemas, etc.

Tool support / code gen from XSD used to be important to me. I came to services via CORBA and was enamored with the whole IDL thing. IDL in CORBA was much simpler - just basic C structs etc. It generally worked, but was certainly not without its problems. So when I first started writing web services in 2002, I dedicated myself to making WSDL an IDL (i.e., contract first). As I've said in various parts of this blog, that was a very painful experience.

I moved on from web services to just dealing with services that use XML on the wire. That XML has always been backed by XSD. As Dare and Tim recommend, the contracts are very loose.

Until recently, I have been ardently pro tool support with this (i.e., just simple XSD) approach. You can get any tool to work when you dumb things down (e.g., .NET xsd.exe, JAXB, Castor, XMLBeans). I always put the generated classes in some package like "msgmapper". I then write some helper classes that serve as a level of indirection to the generated classes.

I used to think it was pure unadulterated evil to generate XML by hand via JDOM, Xerces, etc. Now, I almost prefer that approach. Maybe I am just getting tired. Maybe I am taking the simplicity thing too far. Its just that all the tools come with baggage. You have to run them in the build, they spew out thousands of classes, developers forget to isolate them and they bleed deep into your code (and are hard to extract) etc. With the brute force approach, you go to the "msgmapper" package and there is your code that generates your XML - no fuss, no muss.

More and more I think that any tool that generates code (at least for high level languages like Java & C#) is to be avoided at all costs.

So are contracts important? Of course they are. The thing is the devil is in the details & there is no tool that is going to just give it to you or even make it easy. IMHO contracts are more of an art then a science.

Stonefly Hatch

I am anxiously looking forward to a fly fishing trip I have coming up on Oregon's famous Deschutes river. Some friends and I are trying to time the stonefly hatch perfectly. It is a very famous hatch. Trout go nuts for these huge orange bugs. The stoneflys (orange and the size of your thumb) crawl out on reeds and alder branches along the river. The wind blows them in. The soneflys flap their wings furiously to get out. The trout hammer them. It is a very brief hatch. It is tough to time it right.

The Deschutes is my favorite place to go. I totally zone out and don't care about anything else while I'm there. It is a "blue ribbon" (i.e., kick butt) fishery. The trout in the Deschutes are called "Redsides". They are legendary for their size and fighting ability. They are wild trout that have been there forever. The Deschutes is a big western river. Strong current all year long. Makes the trout strong.

I used to wet a fly at least every other weekend. Since I had a baby a year ago, I don't get out as much anymore.

What does this have to do with EDA, SOA, or ESBs? Nothing - I'm spent. I have been travelling a lot lately. Man that sucks the life out of you. I used to travel almost every week for 5 years. Now I thankfully usually only do it once or twice a quarter.

Anyway, I'll take some pictures.

Temporary MOM bigot

I'm currently a MOM bigot, but only because I haven't seen another viable way to do extensive service oriented integration today.

I yapped about Message-Centric vs. Service-Centric a month or so ago.

I heard that SOA Software acquired Blue Titan this week. This is an interesting development. Here is a good post on why this happened (via Stefan Tilkov).

As I tried to get going on the Yahoo REST message board and in various posts in this blog, this is where I'd like to see things go. Leverage HTTP, but add:

• guaranteed delivery
• durable destinations
• event driven support (i.e., pub/sub)

It is ok to support SOAP, WS-* for the true believers, but do not mandate it. If anything, tolerate SOAP/WS-*, but prefer POX and REST. I wouldn't buy something that preferred SOAP/WS-* and had an extension or subset of support for POX and REST.

Now, I have no idea if Blue Titan even does this (I haven't used it) - it sounds like it does some of this or will. Anyone worked with it?

Ideally, I'd like to see OSS impls or at least dual licensing of this type of thing. This type of infrastructure is too critical to rely on a small vendor for. I can't afford another vendor-pire.

EDA Lessons Learned - Content Based Routing

See EDA Lessons Learned for the list

I have found Content Based Routing (CBR) to be helpful in EDA when a sink receives events, processes them, and then produces a subsequent event.

I like CBR because it pulls the routing logic out of the code. The code just processes an XML input and builds an XML output. It does not set the next Topic explicitly. The routing is controlled in the configuration of the sink. This keeps the code simple and reduces coupling. You can also change the routing without changing the code.

We had success using XPath to determine the next Topic. You can use anything in the XML output as part of the next destination. Or you can have several XPath statements building up the next destination based on attributes/elements in the XML. Examples include event type and event status.

I found an article from IBM from way back in 2000 advocating this approach: Using the JMS API and XML in content-based routing. Glad to see I'm so ahead of the times ;)

Spitting up the J2EE Hippo - now what about WS-*?

Richard Monson-Haefel continues to take the gloves off on J2EE and WS-* (at least the Java WS-* impls). I might not be reading closely enough, but I wish his firm, Burton Group would do the same on WS-* (I'm a customer). He and Burton Group have been consistent on J2EE, but Richard's boss Anne Thomas Manes continues to beat the WS-* drum. I read a couple of her pieces on a plane this week and she has totally drunk the WS-* Kool-Aid. She isn't a total zealot on it or anything, but she is astonishingly pro WS-* in the face of a lot of criticism and lack of success stories in the field. To be fair, it is possible that I just haven't found the right research. Richard - are you (or have you already and I missed it) going to take the gloves off on WS-* in your Burton Group writings? I love your blog, but it only makes a real difference if it is in the official Burton Group position papers.

Also, if J2EE is on the ropes today, who is to say that WS-* won't be in a couple of years? Given its trajectory compared to where J2EE was in the beginning, I think it is quite possible that WS-* will die a much quicker death then J2EE.

Update 08-May-06

To clarify, I thought that Anne's research was very thorough in terms of coverage - it was quite useful. I just was disturbed that given the history on things like J2EE she so easily assumed that WS-* will survive and thrive. Perhaps its her job to be hopeful like that and to try to get momentum going in one direction. My focus is solving integration problems today and in the not so distant future. Perhaps our requirements are different.

Update 13-May-06

I found some interesting links to Anne's response:
James Governor
Mark Nottingham - (he used to chair the WS-Addressing WG at W3C)
Dave Johnson

EDA Lessons Learned - Canonical Message Format

See EDA Lessons Learned for the list

I posted about Canonical Message Format a little over a year ago. I still agree with most of what I said then.

I have been exposed more to the Acord insurance standard since then. Sounds like Acord is addressing a lot of my complaints in the 2.0 version. Main quote that caught my eye is:

V2 will be delivered with an option to "slice" or prune the schema down into smaller files with the message content of your choosing.

That says a lot on how "standard" the XML really is on the wire in the insurance industry. This is essentially admitting defeat IMHO. Nothing wrong with it - just facing reality. The truth is that every carrier has differences - you make a valiant attempt to map to the standard, but you still have 20% more to go. There is an extension mechanism to account for this.

The trouble I have with industry standards like Acord is that if you follow them without pruning the original schema and making it your own (i.e., owning your contract), you have a very loose contract between services.

I'm all for using the industry standard for integrating with third parties. I think it is ok to use internally too, but it makes me nervous. People tend to toss the "standard" term around and act like the fact that it is being used as the canonical message format means something major - as if you are going to get a huge payback for it. I have never seen anyone get a big payback from any XML industry standard. And it takes a lot of effort to follow the standard. New versions come, the company has extensions, etc. At some point, people throw in the towel and have something that looks like the standard, but the chances of it resulting in any sort of out of the box interop with anything are nil.

So I think its mostly about setting expectations ... it is ok to use an industry standard as your canonical message format. I think it is wise to go into it with the "prune" strategy. Just use the standard as a reference. Don't event attempt to use the standards schema files. Just lift the structures. And dumb it down as much as you can to make it a tight contract. You will stray from the standard eventually - better to just admit defeat going into it IMHO. No shame in that.

Here are some more thoughts:

1. Avoid request/response designation in your root elements.

   EDA is not sync - don't act like it is. Even if you have some workflow that is sync, don't bake it into the messaging model - it will likely change over time. The request/response in the XML will just confuse people.

2. If using XML schema, keep it simple - avoid the fancy stuff
3. Question elaborate tools that make managing your canonical message format easier

   These tools often make it easier for you to make things more complex then they need to be and lock you in. They typically require training - so only a couple people know how to use it (single threaded through them). Your canonical message format should be simple & should only require something like XML Spy, source control, and a web site for easy reference.

EDA Lessons Learned - EDA and Object Oriented Design

See EDA Lessons Learned for the list

Event producers and event sinks are very often services written in an Object Oriented language. All the same OO rules apply. OO Design Patterns are still applicable.

This is a pretty minor point, but I have seen a lot of people throw this stuff out the window with EDA/SOA.

In my opinion, services (i.e., event producers & sinks) should have good separation of:

• service handler (should have minimal code - just delegates to message mapping and business logic)
• message mapping (e.g., O/X)
• business logic
• domain model
• data access layer

An aside I'll add is that the Spring Framework also works just fine for Java based services. I only mention it because I have found that its usage tends to have a positive affect on the OO design of services.

EDA Lessons Learned - Logging

See EDA Lessons Learned for the list

This is a small one, but better to get this right from the beginning - we didn't, but are almost done fixing it. And this isn't really EDA specific - its applicable to any large scale distributed programming effort. Our problem was that various developers logged different things. Most used log4j, some didn't. Some configured it one way, others another. Some logged everything to INFO - some used DEBUG properly ... What you want is uniform logging.

We have approximately 50 different services. Each service is typically an event source and sink. Some are just event sources, others just sinks.

Our system has an admin GUI that lets you see the last 10 events (configurable). The runtime just keeps various buffers for incoming and outgoing events. When a user clicks on a particular service, he sees the incoming/outgoing events.

But logging to log files is a different issue. Obviously, this is where you go looking when there is a problem in the system.

Due to privacy / regulatory requirements, we have to be careful what we log. When there is an error, however, you want to know as much as possible.

Here are some thoughts on how to log. As I said, better to do this in development then to dedicate a large portion of a maintenance release (like we did) to getting it right:

• Write a brief doc / WIKI entry on logging policy. Get agreement on what categories to log to and when (e.g., DEBUG, INFO, ERROR)
• Service entry/exit logging
  1. In the original event source, generate a UUID. Put this in the event header
  2. Log the UUID, event type, status (if applicable), and other appropriate meta data
• If you must log the payload of the event when there is an error, scrub the private information (e.g., SSN, DOB, etc.)
• If you are using error queues, you have no reason to log the event body. Just log the stack trace, event meta data, and event history. The error message should also contain the stack trace, event meta data, and event history. If you have an Error Queue Admin tool like we do - it will protect the sensitive information
• Keep the log config files out of the artifacts (e.g., .jar) that you deploy so that if there is panic in prod, you can turn on DEBUG by editing the config

Like I said, this isn't rocket science, but I figured I would point it out. You'll save yourself some angst down the road by spending a couple hours getting consensus from the beginning.

The Big Boys

Saw IBM's big bet on SOA via Simon Tilkov.

To quote the article:

Clearly SOA is key, if not the core, to IBM's software strategy. We will of course have to wait and see if this "shock and awe" approach to dominating a market will work - not to mention solve the customer problems. I do wonder how 31 separate products (so far!) can really deliver the simplicity and agility that is meant to be at the heart of SOA. More importantly, SOA is about providing a solution, not selling an even more complex collection of products.

I don't really have anything to add to that. I tried to, but I just keep staring at my screen muttering things.

EDA Lessons Learned - Choose Topics over Queues

See EDA Lessons Learned for the list

If you are using JMS as the back bone of your EDA, choose Topics (i.e., pub/sub) for 99% of your Destinations. I guess that is pretty obvious - I guess it is more applicable to the Messaging-Centric ESB world.

Exceptions where I think Queues are appropriate:

• Error Queues
• Existing integrations that have queues (I'd still route it through a Topic before it got there)

There are other times where Queues are appropriate, but just triple check that it is appropriate and if in doubt, use a Topic instead.

Topics have a lot of benefits over Queues:

• Facilitate event driven architecture
• Not Point-to-Point (you have a prayer against the n(n-1) problem (where n= # of systems to integrate)
• Very flexible if using in a event workflow - can fan out/fan in easily
• Ease of debugging (can snoop on a Topic)
• Can do 1:M Request/Reply (one request, get N responses)

Also, use hierarchical Topic names. Hierarchical Topic names are great because they:

1. Help to avoid dependence on message selectors (slower)
2. Organize things
3. Enable listening to a pattern - for example:
   If you have Topics in this pattern (several different "EVENTTYPE" values: COMPANYNAME.DIVISION.PRODUCTLINE.PRODUCT.NOUN.EVENTTYPE
   Say you are interested in all of the event types ... you can listen to them like this: COMPANYNAME.DIVISION.PRODUCTLINE.PRODUCT.NOUN.*

Unless you don't need it, use durable subscriptions with your Topics - they work great and will bail you out of some hairy situations. While durable subscriptions were originally intended for momentary connection problems, they can be used to help bridge the EDA & batch world. You can register a durable subscription and then just connect to it at night for instance. You accumulate events all day on the durable subscription and then just drain it as part of the batch run.

To be fair on my Topic preference, I do take advantage of a SonicMQ feature that makes a Topic like a Queue in terms of the ability to have competing consumers (i.e., different connections (can be on different hosts) competing to drain the queue) (Shared Subscriptions). Other vendors do similar things. It is a pity really that Sonic's "Shared Subscriptions" isn't in the JMS spec.

SonicMQ 7

Yesterday I watched / listened to an early access web cast on SonicMQ 7.

I have been using SonicMQ for about 5 years. Rock solid - just works. And when it doesn't, their support people are very helpful - they dig into your problem and crush it. If the problem is a defect - they fix it. If the support person can't figure it out, they work with engineering. If they still can't figure it out, they get engineering on the phone. And they keep following up with you. Really can't say enough about the success I have had with the product.

Anyway, there are some improvements that will help me that I am looking forward to. We use pub/sub, durable subscriptions, and Sonic's "shared subscription" feature extensively. They apparently have some fixes/enhancements that will prevent messages from getting trapped on brokers when bad things happen. I have had this happen when consumers of a Topic on a node in a cluster get horked.

They also are introducing (don't know exact name) "MultiPublishers" and "MultiSubscribers". Apparently, this feature was driven by wall street customers who have thousands of Topics. It takes 100ms to register a subscription at start up. This adds up when you have thousands of topics in use. So, my understanding is that with "MultiSubscribers" and "MultiPublishers" you can add Topics to a Set and register them all at once.

They also apparently tricked out the "Continuous Availability Architecture" (CAA) further. This is wicked cool and I don't know anyone else who does it. Basically, you can have a client publishing to BrokerA, kill -9 BrokerA, and the client just fails over to "BrokerB" and continues publishing ... even within a transaction. And this takes 10 minutes to set up. You don't need exact hardware, bios settings, etc. (i.e., hardware based failover). You just configure your setup that way. Another thing that is huge when fault tolerance really matters (more and more these days). You can recover within 10 seconds vs. 10 minutes. Wicked cool.

Anyway, I'm not the type to spaz out about a product like this - I am a huge skeptic from years of being let down by integration companies. Sonic just makes things stupid simple like they should be - so you can get on with SOA & EDA.

Update 17-JUL-06 - Now that it is possible to determine my employer by reviewing my blog posts, I must follow my employer's blog policy.

This is not an endorsement by Liberty Mutual, but my personal view. Vendors are NOT free to quote with attribution.

EDA Lessons Learned - Service Contracts & Event Stream Design

See EDA Lessons Learned for the list

The contracts between services are the most important thing there is in either SOA or EDA IMHO. You get this right and your future is bright, get it wrong and it is dim. Pretty much that simple.

I came to SOA/EDA from CORBA, J2EE, and DCOM projects. I thought that IDL was the only way to fly. When WSDL came out, I treated it like any other IDL. I was one of the people who endured significant pain with WSDL when it first came out because I was such a believer in "contract first" design/development.

The first time I started muttering "what is the contract?" when discussing OO design, service design, anything software related really was when I was mimicking one of my mentors (Dave Read when we worked at eXcelon). Just because I knew he was wicked smart and he said that a lot, I made sure I knew what he was talking about and added it to my repertoire.

So I am still a huge fan of contract first development, but it isn't always practical - especially with EDA. For one thing if you wanted to use WSDL today with EDA, you really couldn't as WS-* doesn't support pub/sub yet and WS-Notification & WS-Eventing are being merged into WS-EventNotification. I tend to think they will be too rigid when they are updated, but we'll just have to wait and see. Anyway, just because you don't have an IDL, doesn't mean you shouldn't think about contracts.

Service contracts in EDA aren't usually request/reply like they typically are in SOA. It is easier to get your hands around contracts with request/reply. With event streams, you have N services listening to an event and possibly publishing N events to N destinations. A service may listen to many destinations or a pattern. A service may receive XML documents conforming to different schemas. So you pretty much have to let the whole IDL thing go ... unless you just allow "any" or whatever it is in XSD (which is pointless).

This isn't very concise, but for me, contracts in EDA are:

• Event input payload(s)
• Service input destinations (set/patterns)
• Event output payload(s)
• Service output destinations (set)

Here are some more details on my opinions on service contracts and event stream design and EDA:

• Devote appropriate time to service contract/event stream design
  1. Have several sets of experienced eyes review
  2. Review a number of times - allow for "soak time" as my friend Sarge would say
• The goal is low coupling and high cohesion
• Anticipate event stream flow changes
• Consider persistence when thinking about contracts
• Get the granularity right (narrow vs. coarse) - the truth is somewhere in the middle
• UML sequence diagrams with embellishments (persistence, state change, etc.) and annotations works very well for technical design and review
• Great website/book - Enterprise Integration Patterns

EDA Lessons Learned - Persistence

See EDA Lessons Learned for the list

Our EDA is Stream Event Processing based.

There are *lots and lots* of events. It blows your mind to listen to all of the events at once (just subscribe to # in SonicMQ). It is like watching the Matrix. Most of the events aren't terribly meaningful to a typical subscriber. Many of the events are only used in the source system in "event workflows".

EDA is highly distributed. Our system has many different processes and machines publishing and subscribing to events. Persistence is very easy to get wrong in an environment like this. The first lesson is to put enough data in each event so that subscribers of the event do not need to gather more data from the source. Also, in terms of inserts and updates, you have to be careful of table / row locks etc. If you are not careful, you'll have N processes with N threads on N machines trying to hit the same data source.

For us, a lot of these "ordinary / non-notable to downstream service events" are used in event workflows. State is important - you have to put it somewhere. The key is to think very carefully about where you put it and non screw it up.

Here is a list of some general thoughts on the subject:

• As mentioned above, put the appropriate amount of data in each event so that subscribers do not need to gather more data from the sending system
• If you are using persistent messaging & durable subscriptions as your EDA backbone, trust it! You do not need to persist state in each service that handles an event. If you do, you are going to regret it
• XML shredding is expensive and results in a high defect rate (on the way in and out)
• Separate transient (i.e., part of event workflow), terminal state (i.e., completed event workflow), and reporting data bases (ODS, Data ware house, OLAP)
• Beware of O/R tools
1. Work ok in experienced hands, but caching difficult if multiple services in different JVMs have separate cache, mistakes can be hard to fix
2. Use proper granularity
3. Emitted SQL not always performant
4. Often ends up being more complicated then JDBC
• Consider XML database, high performance reliable file system, or caching solution for transient data