Pub/Sub vs. Atom & AtomPub?

I came across an interesting article by Sean McGrath entitled I'll push and you pull. The mashup approach to application integration.

I posted about Push vs. Pull in the spring. I'm a sucker for Lean thinking so the title of this article got my attention.

Sean says:

Now here is the kicker. The web - with all its concomitant bits'n'bobs from XML to RSS/Atom to AJAX - is an extremely good platform for pull-centric design. On the Web, if you try to pull some piece of information and something goes wrong, well you just pull again and again until you get it or give up. Nothing fancy. Just brutish repetition. Something machines are extremely good at. If you want to look at information from yesterday, you just go to the URL that contains yesterday's information. Nothing fancy. Just a simple naming convention that includes dates in URLs.

I have done a fair amount with pub/sub over the years (complete with clustering etc., fail over strategies, etc.). I like it, but it is not without it's challenges. If only there was real guaranteed delivery etc. If only there wasn't "flow control". If only fail over always worked. If only messages didn't get trapped, etc. All I'm saying is having implemented several large implementations of pub/sub (EDA whatever you like), I know that it isn't easy - that nothing is really "guaranteed".

Clearly Atom can't replace ALL pub/sub use cases, but for every day integration architecture where you want business events / EDA why can't we use Atom feeds? In an extreme case, you might have an event sink requesting the feed every 10 seconds - in most cases every 10 minutes would likely be fine?

Who is doing this today? Any lessons from the trenches?

Update I meant to read the article earlier linked off of this post by James Snell. Just did. Dang. It is a sample of using Apache Abdera.

Update 20-AUG Bill de hÓra has a pretty $$ response - thanks Bill.

Update 21-AUG Dan Creswell has some thoughtful comments. When I blog I often do the x vs. y thing - but clearly the truth is in the middle. I used to be a push bigot and have just learn the hard way how difficult it is to achieve. Clearly, not everything can be pull - of course it is a mix. In case anyone is curios, I'm thinking of inter-domain integration (e.g., getting customer additions/updates/deletions to many interested systems) with this line of thinking rather than some sort of intra-domain integration (e.g., trading system where there is massive high performance pub/sub). You have to choose the right tool for the job. More and more for me, simplicity is winning out. This is just an evolution of my thinking - a year ago I was still a MOM bigot.

Simplicity & Integrating with Less Code

I have been on a distributed caching & JavaSpaces kick lately due to the reasons listened here.

I was talking to my co-worker Erik yesterday and we concurred that one of our biggest goals is to come up with an architecture that is as simple as possible and requires the least amount of code.

Coming from my experience with SOA of various flavors and EDA, I have been on many projects that wrote way too much code. The more code the more defects. This is what is so appealing to me about JavaSpaces and potentially distributed caching (e.g., memcached, Tangosol Coherence) (or some of both). In the core of an application it is possible to use these layers as your service orchestration tier as well as your transient data store. This is very appealing as you skip a significant amount of OR and OX mapping which saves heaps of time and defects.

The JavaSpaces API itself is extremely seductive. Lifted from here:

• write: Places one copy of an entry into a space. If called multiple times with the same entry, then multiple copies of the entry are written into the space.
• read: Takes an entry that is used as a template and returns a copy of an object in the space that matches the template. If no matching objects are in the space, then read may wait a user-specified amount of time until a matching entry arrives in the space.
• take: Works like read, except that the matching entry is removed from the space and returned as the result of the take.
• notify: Takes a template and an object and asks the space to notify the object whenever entries that match the template are added to the space. This notification mechanism, which is built on Jini's distributed event model, is useful when you want to interact with a space using a reactive style of programming.
• snapshot: Provides a method of minimizing the serialization that occurs whenever entries or templates are used; you can use snapshot to optimize certain entry usage patterns in your applications. We will cover this method in detail in a later article.

With the addition of JavaSpaces05, there is collection based (i.e., bulk) read, take, and write.

The integration patterns possible with this API are very powerful. Master/Worker, distributed, highly scalable data structures, etc.

It appears that with certain distributed caching vendors, you can achieve some semblence of these patterns, although I am not convinced of that yet, and the associative nature of JavaSpaces (i.e., you use templates containing objects with what you are looking for in a space to find it) is just amazingly seductive.

Love it or hate it, JavaSpaces comes with Jini. To a newbie like myself, Jini takes some getting used to. The whole mobile code bit tends to make people who have lived through J2EE classloader hell nervous. But there are those who say it works fine in Jini. It is a different paradigm and this makes people very nervous.

Maybe what is needed is a mix of J2EE (specifically servlet, JMS, JMX MBeans), distributed caching, and JavaSpaces. Maybe you can use JavaSpaces for both caching and service orchestration. Maybe you should use all of Jini. Maybe there is something else.

From previous lessons learned around persistence in EDA, I am 100% convinced that we need some sort of flexible transient data store (no work in progress database - I beg you!). I see that today as being some form of a persistent distributed cache that requires very little if any mapping code. I also have seen the power of asynchronous integration and SEDA and am not about to give up on it. JavaSpaces and Master/Worker appear to give me that. What other ways are there? Maybe things that I would have considered anathema a year ago like sending objects through JMS for the eventing layer isn't really that awful? I do know that I do not want XML at the core of a brand new system again - again way too many defects with mapping etc. Too slow, too lame. Sure, at the periphery to integrate with services that require it, but never again in the core of the system.

Lots of thinking results in lots of blather . . .

Bottom line, I want this to be as simple as it can be and I want to write as little code as possible. Oh and it should scale like the dickens, be simple to specialize, maintain, etc.

SBA & EDA Lessons Learned

I am still getting my head around SBA (Space Based Architecture)/distributed caching patterns.

I have a ways to go, but the more I read about it (blogs, docs, specs, etc.), talk about it with my insanely smart co-workers, and do stuff with it (hands on) the more I like it.

Like most humans, my mind works quickest when I can associate concepts. As I have mentioned before, I see a lot of parallels between SBA and EDA - specifically Message-Centric EDA. Last Spring, I did a series on lessons learned from an implementation of what I consider Message-Centric EDA. I'm going to pick out a few notable differences that I have noticed so far that makes SBA extremely appealing to me.

1. Content Based Routing
   Content Based Routing, Itinerary Based Routing, any service orchestration you can think of other than frightening things like BPM/BPEL are all achievable with SBA. And its done by using the state of your objects, not Topics, XML and all the complexity and angst that comes with it.
2. Choose Topics over Queues
   SBA wins hands down. Instead of mapping your system into using Hierarchical Topic Names including event types and states you put objects into the space and the state of the objects drives the service orchestration (take, notify, etc.). It's insanely simple. This eliminates 2 mappings: system state to Hierarchical Topic Names and Object to XML (and back). In a large system, this results in heaps and heaps of complexity eliminated.
3. Persistence
   No XML shredding in SBA, the space fits the transient data solution I mentioned. It can be configured to be persistent/reliable & in some implementations mirrored, clustered, etc. Getting relational databases out of the transient state business is great for simplicity / flexibility / maintainability / scalability. Again, one less mapping to deal with - only persist to long term storage at well defined states (again based on your objects, not a serialized XML version of them).

SBA basically turns messaging patterns upside down. The patterns are similar, just simpler in SBA. At first look, the SBA versions seem better. Perhaps the grass is greener, perhaps not.

The right answer is probably all of the above / both SBA and EDA. The really hard part is figuring out where to use which in building loosely coupled cohesive systems.

Update 29-DEC-2006 I was inspired thinking about this further on the walk to work this morning . . . I mention complexity and mapping above. Things like object to XML (and back) (OX), system state to hierarchical Topic names (and back), object to relational db (OR), Object to XML (Castor, JAXB, etc.), XML to relational db (XML shredding). Why do you care about all this? I'll tell you why you do. Not having this cruft in your system results in:

1. Hundreds of conversations and meetings that you will not have sorting this mapping out
2. Many design decisions / design details you won't care about
3. Thousands of unit tests you won't write
4. Hundreds of defects that you won't file, ponder, fix, and unit test

Update 07-FEB-2007 Appeasing the "then" vs. "than" police.

Update 22-JUL-2007 As this page is linked to from a Wikipedia page on Space Based Architecture, and gets a fair amount of referrals, I think that I have a responsibility to temper my original comments with what I believe today. I continue to have high hopes for Tuple Spaces, JavaSpaces, and SBA. In my opinion, however, there are not enough viable alternatives today. This is very important to me. I am watching with great interest what will happen with Apache River prior to making any further investment in this technology.

2007 Predictions Starting

Here come the 2007 predictions. Who doesn't like these? Perhaps we need another round of blog tag on that ;)

Patrick Logan has one that I'd like to see come true. I'd at least like to see a good hard look at it. I've lived web services, ESB, EDA, and they all have their issues. I'm sure Jini/JavaSpaces has its as well, but perhaps the best of all of them is the right fit (ok REST not web services).

As the industry comes to term with the failure of WS-* we can either start over and reinvent things again or perhaps take a look back at some of the technology that should have been front and center to begin with.

I will post my prediction next week.

SBA trumps EDA?

We are taking a look at some architectures for a next generation of a critical system at work.

I am a fan of EDA - Event Driven Architecture. I posted about it a bit here.

Its nice and all, but it is not without its problems. It certainly has a role to play in any architecture. But the last time I built one, I ran into some problems as detailed in my series linked to above.

Does SBA (Space Based Architecture) simplify things even more? For one thing, it minimizes "legacy" XML which is certainly a good thing. And the location transparency is mind boggling. Makes things like Sonic's shared subscriptions look quaint.

Just when you were happy that web services are finally dead, there is another shift. And if you are still talking about web services, well . . . sorry.

OOPSLA Event Driven Architecture Workshop

Saw this workshop at OOPSLA on EDA here in Portland, OR this fall via Marco on ESP.

I haven't been spending as much time obsessing about EDA as of late. But I still think it has the most promise in terms of enabling wicked cool integration. We just need something like AMQ IMHO.

Anyway, I hope to be there.

Maybe some of my EDA practitioner friends will be too?

EDA Lessons Learned - Config Files

See EDA Lessons Learned for the list

EDA is highly distributed.

This results in LOTS of processes and LOTS of config files.

This is another topic you want to think about at the beginning of the project.

We got this mostly right, by having Agent properties (JVM level) and Services properties. Service properties inherit from Agent properties (and can override them). We had some Java classloader problems, however, with Hibernate (not a big fan of ORM period - time savings are a myth - what is so wrong with SQL!? - sorry for the rant) and ended up with a property file on the system classpath. This seemingly small little problem became a total hassle. Last time I checked, it is still setup this way. We have talked a couple times about fixing it, but it keeps just getting talked about. We have talked about it enough to fix it 2X probably. And people trip over it occasionally.

So anyway, not too proud of this - don't do this. Pick a consistent approach and stick to it. If there is a problem, get it fixed right away or it will cause all sorts of little problems.

EDA Lessons Learned - Transactions

See EDA Lessons Learned for the list

I'm not doing much (ok any) 2PC (two-phased commit) these days. I used to be pretty hip on XA in the Java world. I still run into people that yell at me about XA. Perhaps I wear size 16 burgundies (clown reference for the un-indoctrinated).

XA is great in theory and fun to drop in meetings (not quite as fun as 2PC), but:

• Slow (sync to disc)
• Expensive (resources)
• Complex
• Implementations often buggy
• Not always easy to find XA impls (try VSE mainframe)

It isn't perfect, but pseudo 2PC works pretty well:

• JMS transaction + JDBC
• Compensating transaction if necessary

Here is a good write up on the topic.

EDA Lessons Learned - Shared Code

See EDA Lessons Learned for the list

Sharing code between services requires careful thought and planning to avoid too much coupling between service impls. I have seen both extremes - no sharing and uber sharing (i.e., too much).

So my advice is do share code just don’t overdo it.

Some modules that I found useful (in hindsight) to share by user interfaces and various services are:

• schema (code that validates XML against schema)
• persistence (consistent way of accessing database)
• msgmapper (mapping XML to POJOs that you pass into business logic)
• domainmodel (the most important package/module that most apps don't have)

The obvious issue here is that if you don't share code, you are likely to get the same bug in all your services. To fix it, you have to fix it in each service. But when you do share code between services, then you have dependencies. You can't win. Just quit now ;)

EDA Lessons Learned - Batch vs EDA

See EDA Lessons Learned for the list

I generally shy away from throwing down phrases like impedance mismatch as they tend to get abused, but no joke - batch and EDA is one.

We deal with this with our large systems (big iron), which are batch oriented.

Bridging the batch world with the EDA world requires careful thought and design. We found it effective to have very clear control boundaries between the two.

We have batch jobs that run in the middle of the night that generate many events. The events are not generated at the same time every night - it depends on how long the batch cycle takes. So basically the EDA side gets huge bursts of events in the middle of the night.

We see the most benefit from Staged Event Driven Architecture when batch jobs produce large bursts of events.

We also have services that want to write to the mainframe when it is in a batch run. The mainframe is unable to accept the writes when it is in this state. The "Rollback and Pause" concept is very helpful in cases like these. Basically, the service (filter in our case) just holds the event by rolling it back to the messaging layer and letting it retry at an interval. Having this logic in a filter rather then the service responsible for writing to the mainframe is good because you don't muddle the service impl. with environmental issues.

Operationally, the batch/EDA impedance mismatch is definitely our biggest challenge. It generally works, but we have our days. Async send from the batch system, clear control boundaries, "rollback and pause", and ErrorQAdmin, make it tolerable.

EDA Lessons Learned - XML Aware Analysts

See EDA Lessons Learned for the list

We backed into this by accident, but our business analysts (report into IT) became very "XML Aware" (i.e., understand XML). This is a huge asset IMHO.

In a perfect world, perhaps this wouldn't be necessary, but in the real world, where projects get delayed this is very useful.

In the height of our panic (development) of the first phase of this system, we had services in varying degrees of completeness with scaffolding all over the place. The scaffolding allowed for some semblance of "end to end" testing.

Certain services were considered "code complete" by development. XML Aware testers allowed us to do black box testing against these services. Analyst started writing test cases with expected XML inputs and outputs. This was translated to XPath assertions, which, in some cases were automated (extension to XMLUnit/JUnit). The analysts invariably found defects in the code complete services. If we had waited to do traditional end to end testing, the project would have been even later.

"XML Aware" analysts had the side effect of making the analysis and design better in a major subsequent phase of the project as designers and analysts were competent in EDA style architecture and XML.

If you are lucky enough to have analysts on your team, I recommend getting them down into the weeds a bit and having them learn XML & the concepts of the architecture style you are using. If nothing else, they will ask one "dumb question" that will make you think. If you are lucky, they will start calling b.s. on you at appropriate times ;)

SOA and EDA make sense to everyone in IT from a high level. Spreading the knowledge of how they are implemented in your organization outside of the development team will result in a higher quality impl.

EDA Lessons Learned - Build and SCM

See EDA Lessons Learned for the list

Everyone's favorite topic - build and SCM ;)

SCM is a hot topic within my company right now. For whatever reason, people get very excited/emotional about this topic (myself included). After living with various SCM tools, I prefer simple ones that aren't a barrier to distributed development. I actually used to like ClearCase - a lot. But now I think that it is overly complex and is a barrier to distributed development. ClearCase absolutely must be installed on a LAN for it to work - this forces a replication model for distributed development. This results in too much complexity for my taste and makes it very difficult to truly collaborate at the code level. There are OSS and commercial SCM products that were designed from the ground up to facilitate distributed development. Pick one of them if your developers are all over the world.

These lessons learned are from a large EDA style application, but like many of my lessons learned, it is applicable to a SOA project or any large integration project. I do not (yet) have an approach for company-wide SCM for SOA/EDA. That is certainly more complicated.

I knew a lot of this stuff from previous jobs, but walked into this project as it was in full development. It took *significant* effort to reorganize and fix. Like many things, planning from the beginning would have saved a ton of effort. Here is a list:

1. Plan the SCM repository carefully

   Don't have too many roots - don't have too few roots. Under the root, service, shared, web directories with modules beneath works well for us.

2. Branching model

   Choose carefully - here is a good article on branching models. I prefer "branch by purpose" - it keeps it as "simple as possible, but not simpler" as my boy A. Einstein would say.

3. Keep build simple
   • Developer should be able to run entire build locally easily
   • Developer should be able to build just required modules (i.e., shouldn’t have to run entire thing if just working on 1 service)
   • Developer should be able to make changes to the build (avoid single threading it through the build guy)
4. Deploy beefy build box

   Buy a nice high end workstation with plenty of memory and 2 very fast processors for a build box. You don't want the build to run on the build guy's local machine. It is much easier to automate from a centralized official build box IMHO. Anyone with access should be able SSH to the build box and launch a build. At a previous job, we even had a web page that allowed developers to launch builds (way to go Sarge)

5. Write and maintain simple instructions to get new developer’s environment setup
   Shouldn’t take more then an hour
6. Configuration

   Either keep config files for each environment (e.g., dev, test, prod) under source control and have deploy script use appropriate configs or script the setup of each environment. You don't want to do this by hand. This can be difficult with vendor products that want to invade your entire SDLC. Ensure that your vendors have deployment friendly support

7. Avoid Maven

   I am not up to speed on the latest version of Maven. I understand it is supposed to be better so take this with a grain of salt. At one point we had competing builds - Ant and Maven. Ant won because it is simpler and more people understand it well. We do use the Maven directory structure, however. I'll leave it at that.

8. Don't mandate a particular IDE

   Quickest way to piss off developers is to mandate an IDE. The build should never mandate an IDE.

9. Beware of tools that generate code

   If you must use these tools, at least either check in the results of the generated code (can be very dangerous if the code isn't just POJO style) or script the usage of the tool in the build (and watch your developers start to hate the build). And people wonder why I hate generated code ;).

EDA Lessons Learned - Performance

See EDA Lessons Learned for the list

For starters, specify a requirement or two on what acceptable performance is. Then, start measuring early. Better to know that you are way off the mark early on IMHO. Fixing services that are way too slow late in development or once they are in production often means rewriting the service from scratch in my experience (of course you still call it refactoring in meetings to make everyone feel better ;) ).

I have seen the most performance problems with Persistence.

Another thing to focus on early is how your services will scale. Our system resembles SEDA (Staged Event-Driven Architecture).

• Thread Safe - in our system that is built on top of JMS at the core, we use a JMS ServerSessionPool impl that makes adding threads (sessions) to a service configurable. This is a great way to scale and throttle a service. Sadly, we deal with some obscure stuff that isn't multithreaded - so we have to configure just one thread. For most services, however, you can have 10+ threads waiting to process events
• Competing Consumers & Shared Subscriptions - Anyone familiar with JMS understands competing consumers with Queues. We use SonicMQ, which, also offers this concept with Topics. We use these heavily. It is great because you can spread processing across machines. You can have 1 logical service running on N machines. When it is flooded with events, the processing is evenly distributed across the logical instances of the service.

Another area to focus on from the start is service granularity (the black art of SOA/EDA) - this is a lot harder to get right then a lot of people think. Especially when you are dealing with many services. Its like anything of sufficient complexity - there are 3-5 ways to do it right and 50 ways to do it wrong - pick one of the right ways.

Last thing I'll mention is async processing. This is huge in an EDA system - basically everything is async. But some developer's little minds don't like async and want to make it sync. You must find these people early and get them off the project. Just kidding (kinda). We had a couple problems with performance like this. Our login for instance was a dog at one point. This was because it was sync and the granularity of service calls was awful. So we "refactored" (where "refactor" means tore out old code and stomped on it and called it names, then wrote it again from scratch). We rewrote it in a way that was async. When a user logs in, an event is generated that the system responds to. Immediately after the login event is produced the first page of the UI is displayed. Other events in the backend are produced that the UI is interested in. So it listens for these events. The key is with the rewrite, we don't make that process sync. When the events the UI needs are produced, the UI receives them async.

EDA Lessons Learned - Scaffolding

See EDA Lessons Learned for the list

Scaffolding is useful all over the place in software. EDA and SOA really can benefit from it.

Scaffolding is especially useful in large new development efforts. Services reach code complete at different times. You are totally hamstrung if you wait until all services are complete to start testing. The scaffolding (i.e., mock services) can be used to simulate the services that are still in development.

Once developed, scaffolding can be useful when a service that is not available. If you don't have scaffolding, development and testing can grind to a halt while you wait for the service to come back up.

Scaffolding certainly doesn't come for free - you have to develop it like any other service & it can be every bit as complex as the real service. Its often worth the investment though. You don't need a scaffolding impl for every service you write. Mock services are particularly helpful for services that wrap a volatile resource or a service that is outside of your control (e.g., different org. that has a different development schedule).

A SOA or EDA that provides good location transparency makes swapping in/out scaffolding easier. For our EDA style system we just disable the real service and enable the mock service. The mock service subscribes to the same JMS Destinations, and produces the same event types. This is great because you don't have to modify anything on the clients to swap in the scaffolding. You can achieve the same thing with UDDI or RSS - I do really like the EDA style though - when you are working in an event cloud you get extreme decoupling which rocks.

EDA Lessons Learned - Content Based Routing

See EDA Lessons Learned for the list

I have found Content Based Routing (CBR) to be helpful in EDA when a sink receives events, processes them, and then produces a subsequent event.

I like CBR because it pulls the routing logic out of the code. The code just processes an XML input and builds an XML output. It does not set the next Topic explicitly. The routing is controlled in the configuration of the sink. This keeps the code simple and reduces coupling. You can also change the routing without changing the code.

We had success using XPath to determine the next Topic. You can use anything in the XML output as part of the next destination. Or you can have several XPath statements building up the next destination based on attributes/elements in the XML. Examples include event type and event status.

I found an article from IBM from way back in 2000 advocating this approach: Using the JMS API and XML in content-based routing. Glad to see I'm so ahead of the times ;)

EDA Lessons Learned - Canonical Message Format

See EDA Lessons Learned for the list

I posted about Canonical Message Format a little over a year ago. I still agree with most of what I said then.

I have been exposed more to the Acord insurance standard since then. Sounds like Acord is addressing a lot of my complaints in the 2.0 version. Main quote that caught my eye is:

V2 will be delivered with an option to "slice" or prune the schema down into smaller files with the message content of your choosing.

That says a lot on how "standard" the XML really is on the wire in the insurance industry. This is essentially admitting defeat IMHO. Nothing wrong with it - just facing reality. The truth is that every carrier has differences - you make a valiant attempt to map to the standard, but you still have 20% more to go. There is an extension mechanism to account for this.

The trouble I have with industry standards like Acord is that if you follow them without pruning the original schema and making it your own (i.e., owning your contract), you have a very loose contract between services.

I'm all for using the industry standard for integrating with third parties. I think it is ok to use internally too, but it makes me nervous. People tend to toss the "standard" term around and act like the fact that it is being used as the canonical message format means something major - as if you are going to get a huge payback for it. I have never seen anyone get a big payback from any XML industry standard. And it takes a lot of effort to follow the standard. New versions come, the company has extensions, etc. At some point, people throw in the towel and have something that looks like the standard, but the chances of it resulting in any sort of out of the box interop with anything are nil.

So I think its mostly about setting expectations ... it is ok to use an industry standard as your canonical message format. I think it is wise to go into it with the "prune" strategy. Just use the standard as a reference. Don't event attempt to use the standards schema files. Just lift the structures. And dumb it down as much as you can to make it a tight contract. You will stray from the standard eventually - better to just admit defeat going into it IMHO. No shame in that.

Here are some more thoughts:

1. Avoid request/response designation in your root elements.

   EDA is not sync - don't act like it is. Even if you have some workflow that is sync, don't bake it into the messaging model - it will likely change over time. The request/response in the XML will just confuse people.

2. If using XML schema, keep it simple - avoid the fancy stuff
3. Question elaborate tools that make managing your canonical message format easier

   These tools often make it easier for you to make things more complex then they need to be and lock you in. They typically require training - so only a couple people know how to use it (single threaded through them). Your canonical message format should be simple & should only require something like XML Spy, source control, and a web site for easy reference.

EDA Lessons Learned - EDA and Object Oriented Design

See EDA Lessons Learned for the list

Event producers and event sinks are very often services written in an Object Oriented language. All the same OO rules apply. OO Design Patterns are still applicable.

This is a pretty minor point, but I have seen a lot of people throw this stuff out the window with EDA/SOA.

In my opinion, services (i.e., event producers & sinks) should have good separation of:

• service handler (should have minimal code - just delegates to message mapping and business logic)
• message mapping (e.g., O/X)
• business logic
• domain model
• data access layer

An aside I'll add is that the Spring Framework also works just fine for Java based services. I only mention it because I have found that its usage tends to have a positive affect on the OO design of services.

EDA Lessons Learned - Logging

See EDA Lessons Learned for the list

This is a small one, but better to get this right from the beginning - we didn't, but are almost done fixing it. And this isn't really EDA specific - its applicable to any large scale distributed programming effort. Our problem was that various developers logged different things. Most used log4j, some didn't. Some configured it one way, others another. Some logged everything to INFO - some used DEBUG properly ... What you want is uniform logging.

We have approximately 50 different services. Each service is typically an event source and sink. Some are just event sources, others just sinks.

Our system has an admin GUI that lets you see the last 10 events (configurable). The runtime just keeps various buffers for incoming and outgoing events. When a user clicks on a particular service, he sees the incoming/outgoing events.

But logging to log files is a different issue. Obviously, this is where you go looking when there is a problem in the system.

Due to privacy / regulatory requirements, we have to be careful what we log. When there is an error, however, you want to know as much as possible.

Here are some thoughts on how to log. As I said, better to do this in development then to dedicate a large portion of a maintenance release (like we did) to getting it right:

• Write a brief doc / WIKI entry on logging policy. Get agreement on what categories to log to and when (e.g., DEBUG, INFO, ERROR)
• Service entry/exit logging
  1. In the original event source, generate a UUID. Put this in the event header
  2. Log the UUID, event type, status (if applicable), and other appropriate meta data
• If you must log the payload of the event when there is an error, scrub the private information (e.g., SSN, DOB, etc.)
• If you are using error queues, you have no reason to log the event body. Just log the stack trace, event meta data, and event history. The error message should also contain the stack trace, event meta data, and event history. If you have an Error Queue Admin tool like we do - it will protect the sensitive information
• Keep the log config files out of the artifacts (e.g., .jar) that you deploy so that if there is panic in prod, you can turn on DEBUG by editing the config

Like I said, this isn't rocket science, but I figured I would point it out. You'll save yourself some angst down the road by spending a couple hours getting consensus from the beginning.

EDA Lessons Learned - Choose Topics over Queues

See EDA Lessons Learned for the list

If you are using JMS as the back bone of your EDA, choose Topics (i.e., pub/sub) for 99% of your Destinations. I guess that is pretty obvious - I guess it is more applicable to the Messaging-Centric ESB world.

Exceptions where I think Queues are appropriate:

• Error Queues
• Existing integrations that have queues (I'd still route it through a Topic before it got there)

There are other times where Queues are appropriate, but just triple check that it is appropriate and if in doubt, use a Topic instead.

Topics have a lot of benefits over Queues:

• Facilitate event driven architecture
• Not Point-to-Point (you have a prayer against the n(n-1) problem (where n= # of systems to integrate)
• Very flexible if using in a event workflow - can fan out/fan in easily
• Ease of debugging (can snoop on a Topic)
• Can do 1:M Request/Reply (one request, get N responses)

Also, use hierarchical Topic names. Hierarchical Topic names are great because they:

1. Help to avoid dependence on message selectors (slower)
2. Organize things
3. Enable listening to a pattern - for example:
   If you have Topics in this pattern (several different "EVENTTYPE" values: COMPANYNAME.DIVISION.PRODUCTLINE.PRODUCT.NOUN.EVENTTYPE
   Say you are interested in all of the event types ... you can listen to them like this: COMPANYNAME.DIVISION.PRODUCTLINE.PRODUCT.NOUN.*

Unless you don't need it, use durable subscriptions with your Topics - they work great and will bail you out of some hairy situations. While durable subscriptions were originally intended for momentary connection problems, they can be used to help bridge the EDA & batch world. You can register a durable subscription and then just connect to it at night for instance. You accumulate events all day on the durable subscription and then just drain it as part of the batch run.

To be fair on my Topic preference, I do take advantage of a SonicMQ feature that makes a Topic like a Queue in terms of the ability to have competing consumers (i.e., different connections (can be on different hosts) competing to drain the queue) (Shared Subscriptions). Other vendors do similar things. It is a pity really that Sonic's "Shared Subscriptions" isn't in the JMS spec.

EDA Lessons Learned - Service Contracts & Event Stream Design

See EDA Lessons Learned for the list

The contracts between services are the most important thing there is in either SOA or EDA IMHO. You get this right and your future is bright, get it wrong and it is dim. Pretty much that simple.

I came to SOA/EDA from CORBA, J2EE, and DCOM projects. I thought that IDL was the only way to fly. When WSDL came out, I treated it like any other IDL. I was one of the people who endured significant pain with WSDL when it first came out because I was such a believer in "contract first" design/development.

The first time I started muttering "what is the contract?" when discussing OO design, service design, anything software related really was when I was mimicking one of my mentors (Dave Read when we worked at eXcelon). Just because I knew he was wicked smart and he said that a lot, I made sure I knew what he was talking about and added it to my repertoire.

So I am still a huge fan of contract first development, but it isn't always practical - especially with EDA. For one thing if you wanted to use WSDL today with EDA, you really couldn't as WS-* doesn't support pub/sub yet and WS-Notification & WS-Eventing are being merged into WS-EventNotification. I tend to think they will be too rigid when they are updated, but we'll just have to wait and see. Anyway, just because you don't have an IDL, doesn't mean you shouldn't think about contracts.

Service contracts in EDA aren't usually request/reply like they typically are in SOA. It is easier to get your hands around contracts with request/reply. With event streams, you have N services listening to an event and possibly publishing N events to N destinations. A service may listen to many destinations or a pattern. A service may receive XML documents conforming to different schemas. So you pretty much have to let the whole IDL thing go ... unless you just allow "any" or whatever it is in XSD (which is pointless).

This isn't very concise, but for me, contracts in EDA are:

• Event input payload(s)
• Service input destinations (set/patterns)
• Event output payload(s)
• Service output destinations (set)

Here are some more details on my opinions on service contracts and event stream design and EDA:

• Devote appropriate time to service contract/event stream design
  1. Have several sets of experienced eyes review
  2. Review a number of times - allow for "soak time" as my friend Sarge would say
• The goal is low coupling and high cohesion
• Anticipate event stream flow changes
• Consider persistence when thinking about contracts
• Get the granularity right (narrow vs. coarse) - the truth is somewhere in the middle
• UML sequence diagrams with embellishments (persistence, state change, etc.) and annotations works very well for technical design and review
• Great website/book - Enterprise Integration Patterns