Open Source Jobs Posted

The jobs I alluded to the other day have now been posted. Come be a part of the next wave of Open Source Software!

Here is the HR approved job posting. I wanted to just leave it at "Open Source Stone Killer" to see what we would get, but they wouldn't allow it. ;)

--------------------------------------------------------

Utilizes expert knowledge of information and processing services to support the delivery of timely, quality software solutions. Develops, enhances and tests highly complex systems and/or software across multiple platforms, applications and projects. Acts as principal project architecture designer for 1 or more project teams. Transforms common system requirements into design for shared and reusable services. Coordinates upgrades and rollouts of larger, more complex scope. Leads projects or subprojects of significant technical complexity.

In this strategic role, you will deliver business value by building systems that meet or exceed business requirements and agreed upon timelines using Open Source Software.

Participate in developing the Liberty Mutual Agency Markets IT Open Source Software strategy.

Partner with the Open Source Community on Linux usage and middleware usage and contribution.

Evangelize lessons learned from Open Source Software community to our distributed developer network.

For these roles, we have the opportunity to hire at two levels: "Principal Software Engineer" and "Technologist".

As a “Principal Software Engineer” on our team, you would utilize expert knowledge of information and processing services to support the delivery of timely, quality software solutions. Develop, enhances and tests highly complex systems and/or software across multiple platforms, applications and projects. Act as principal project architecture designer for 1 or more project teams. Transform common system requirements into design for shared and reusable services. Coordinate upgrades and rollouts of larger, more complex scope. Lead projects or subprojects of significant technical complexity.

As a “Technologist” on our team, you would be responsible for developing clear, comprehensive and integrated technical and product architectures which support the IT infrastructure and provide long-term competitive advantage in the three to five year time frame. Review architectural designs to ensure consistency, maintainability and flexibility. Analyze highly complex enterprise wide system, technical and product issues to develop the strategic infrastructure direction. Investigate and forecast IT trends. Act as migration planning consultant and architecture guide. Lead projects or subprojects of strategic impact.

To be considered as a strong candidate for either role, you will need to have:

Extensive knowledge and experience building systems to support complex business functions.

Experience in financial services industry preferred, but not required.

Extensive knowledge of IT integration concepts including two or more of Portal technology, Event Driven Architecture, Service Oriented Architecture, Web Services, REST, and JMS.

Demonstrable experience with Open Source Software including Linux, Java OSS tools and frameworks and three or more of the following: Tomcat, JBoss, MySQL, PostgreSQL, PHP, Python, Ruby.

Negotiation, facilitation and consensus building skills. Strong oral and written communication skills; presentation skills.

Bachelors or Masters Degree in technical or business discipline or equivalent experience; technical degree preferred.

Generally a minimum of 8 years related experience.

--------------------------------------------------------

Here is how you apply

1. Click here
2. Click on Experienced Hires
3. Job Category: Information Technology
4. Job Location City: OR-Portland
5. External Job Title: Open Source Software Principal Software Engineer

OSS = Synchronized Self Interest

Great quote from Sun's Simmon Phipps:

Instead, the future is in co-operation and in organizations preserving what is ultimately of value to them, he said.

This is not volunteerism, It is directed self-interest, synchronized self-interest, and there is nothing wrong with self-interest.

My new friend, Jason Gilmore sent me a link to the story. This quote inparticular really struck a nerve with a lot of people I shared it with.

Here is the link to Simmon's presentation at OSBC.

I liked this quote as well: Three Golden Rules:

• Collaborate over what does not differentiate
• Compete by innovating on the commodity base
• Contribute!

OSS is pretty mainstream now, but there is a huge third wave coming. It is going to be tough in the future to be a commercial software company I think.

It isn't just about cost savings. It isn't freeware. It is about innovation, agility, flexibility, and attracting talent.

Complexity and Stockholm Syndrome

What is it about us human beings that we have this bizarre tendency to exhibit behavior consistent with Stockholm Syndrome when we work with technologies that are more complex then they need to be?

I have been guilty of this several times and will no doubt be guilty of it again. For all I know I'm guilty of it right now.

I saw this a lot with EJB. I was certainly guilty of it. Having achieved some success and decent understanding of EJB, I became an ardent defender of it even though it was terrible technology.

I see this same phenomena today with WS-*. I like to think that I learned to spot this trend with EJB and am wiser now, but for all I know I have Stockholm Syndrome with my messaging bias. I don't really think I do because I keep challenging myself to look past messaging in its current proprietary form.

I now think that I also had Stockholm Syndrome when I was an advocate of Rational ClearCase (sorry Sarge). Everyone new to ClearCase runs kicking and screaming from it because it is so complex. Once you suffer through the arduous learning curve, however, it is common to become a ClearCase zealot. I know, I was one for years! I'm dealing with this currently at work. Note to coworkers - I mean to offense by this and this is just my opinion. I think you are used to me by now.

Maybe I am just getting old and dumb and can only handle simple things.

Insane Week + New Role + Jobs

I had a pretty crazy week. Last Sunday I flew with a crew from work to Indianapolis, IN for some post reorg meetings. The CIO announced his direct reports and the gist of the reorg the week before, but there were a lot of loose ends and various interpretations of the reorg to be worked out.

Talk about long and stressful days - we would get to the office at 7:30 AM and leave at 7:00 PM for dinner where work was discussed further.

The new org. is highly matrixed. This is how we were organized in Portland so I am used to that. Time will tell how it works on a much larger distributed scale.

I did get to go water skiing with some of my work friends one day - that was cool. Although it highlighted the point that I am indeed completely washed up (as if that was still a question) as I couldn't get up on one. I tried 4 times before I through in the towel and went with two.

The new CIO is pretty innovative. He has embraced distributed development and Open Source Software. My new job is leading the charge on OSS within our division. This marks my departure from a "Technologist" into management. This should be very interesting work. I will be working like a dog, however (not really a new thing sadly). I will be hiring a small senior team to get this organization off the ground. I have to put the job reqs together this week and will post links once they are available. But if you have a *passion* for OSS and live in Portland, Indianapolis, IN, or Wausau, WI and want to drive OSS adoption/contribution within a large company, let me know. For this initial round of hiring I am looking for stone killers with demonstrable OSS experience. As usual, communication skills are fairly crucial. If you can't collaborate and communicate effectively, your technical skills are useless to me.

Also, since we are now a distributed organization, there are some more jobs open in Portland, OR. Basically, going forward, most jobs that are open in Indianapolis will be open for staffing in Portland, OR. The opposite will be true for projects based in Portland, OR for Indianapolis and other cities. There are pros and cons to distributed development. It is the way the world is going, however. Our CIO is putting the right infrastructure in place to make this model work. The transition will no doubt take some time, but I am optimistic.

This list will be growing the next few months, but here is a list of the current open jobs (search for Information Technology).

Advanced Message Queue Protocol

My friend Erik pointed me at Advanced Message Queue Protocol a year ago, but I assumed it died.

Well, he pointed me at it again and it looks like it may go somewhere.

This would be fantastic. While I love messaging, bridging messaging systems really limits their reach.

I wonder how they are going to get everyone to play along (IBM, Tibco, Sonic, etc.). Hopefully, RedHat's effort will help: RedHat is currently working on an implementation that will be built into the operating system, making AMQ as free and available as SendMail, and accessible from any technology API such as JMS.

If this happens, it would be a major step forward in my opinion and a major enabler for large scale event driven architecture.

EDA Lessons Learned - Transactions

See EDA Lessons Learned for the list

I'm not doing much (ok any) 2PC (two-phased commit) these days. I used to be pretty hip on XA in the Java world. I still run into people that yell at me about XA. Perhaps I wear size 16 burgundies (clown reference for the un-indoctrinated).

XA is great in theory and fun to drop in meetings (not quite as fun as 2PC), but:

• Slow (sync to disc)
• Expensive (resources)
• Complex
• Implementations often buggy
• Not always easy to find XA impls (try VSE mainframe)

It isn't perfect, but pseudo 2PC works pretty well:

• JMS transaction + JDBC
• Compensating transaction if necessary

Here is a good write up on the topic.

Distributed Development Tools

I am spending a lot of my time working on tool sets lately. I have lots of biased opinions.

My biggest core requirement is distributed development with internal developers and third parties. I want it to be easy. It shouldn't take more then 1 hour for a developer in my office to get at the source code and having it building for a project managed out of another office.

I don't want to bias this (yet) with what I am looking at.

I'd love to hear what others are doing with distributed development and tools.

I am looking for the following tools (mainly):

• Source Control
• Defect/Issue/Enhancement tracking
• Project management
• Build/Deployment tool (ala CruiseConrol)
• Requirements Analysis Tool

Reorg

Huge reorg. went down at my company yesterday.

Things went well for me and the Portland office.

Reorgs cause a ton of stress. Even though it appears that I have an extremely bright future, I am totally fried. The new CIO basically took the place, shook it 11 times, and poured it out.

In all honesty, the new structure looks great on paper, but it is going to be a tough transition.

Anyway, watch this space - we will hopefully be doing a lot of hiring - I'll post opportunities here.

We will continue to work on fairly massive SOA and EDA projects and accelerating OSS adoption, contribution, and other stuff I can't talk about yet.

EDA Lessons Learned - Shared Code

See EDA Lessons Learned for the list

Sharing code between services requires careful thought and planning to avoid too much coupling between service impls. I have seen both extremes - no sharing and uber sharing (i.e., too much).

So my advice is do share code just don’t overdo it.

Some modules that I found useful (in hindsight) to share by user interfaces and various services are:

• schema (code that validates XML against schema)
• persistence (consistent way of accessing database)
• msgmapper (mapping XML to POJOs that you pass into business logic)
• domainmodel (the most important package/module that most apps don't have)

The obvious issue here is that if you don't share code, you are likely to get the same bug in all your services. To fix it, you have to fix it in each service. But when you do share code between services, then you have dependencies. You can't win. Just quit now ;)

EDA Lessons Learned - Batch vs EDA

See EDA Lessons Learned for the list

I generally shy away from throwing down phrases like impedance mismatch as they tend to get abused, but no joke - batch and EDA is one.

We deal with this with our large systems (big iron), which are batch oriented.

Bridging the batch world with the EDA world requires careful thought and design. We found it effective to have very clear control boundaries between the two.

We have batch jobs that run in the middle of the night that generate many events. The events are not generated at the same time every night - it depends on how long the batch cycle takes. So basically the EDA side gets huge bursts of events in the middle of the night.

We see the most benefit from Staged Event Driven Architecture when batch jobs produce large bursts of events.

We also have services that want to write to the mainframe when it is in a batch run. The mainframe is unable to accept the writes when it is in this state. The "Rollback and Pause" concept is very helpful in cases like these. Basically, the service (filter in our case) just holds the event by rolling it back to the messaging layer and letting it retry at an interval. Having this logic in a filter rather then the service responsible for writing to the mainframe is good because you don't muddle the service impl. with environmental issues.

Operationally, the batch/EDA impedance mismatch is definitely our biggest challenge. It generally works, but we have our days. Async send from the batch system, clear control boundaries, "rollback and pause", and ErrorQAdmin, make it tolerable.

Reading Books

I'm reading a couple books.

On the advice of Sacrificial Rabbit, I picked up a book named, Getting Things Done. Now I generally do "get a lot of things done" - all by myself. But my organization systems have crashed the last few months as I am working on more and more different things all at once. I'm looking for something new and people seem to be into this so what the heck! I'll let you know how it goes.

I also picked up Charlie Wilson's War ... "The extraordinary story of how the wildest man in congress and a rogue CIA agent changed the history of our times". I'm with Don Imus ... it is "The most unbelievable book". And its true! I picked it up at O'Hare for $5. Best $5 I have spent since $.25 draft night that one time in college where things got a bit out of control.

EDA Lessons Learned - XML Aware Analysts

See EDA Lessons Learned for the list

We backed into this by accident, but our business analysts (report into IT) became very "XML Aware" (i.e., understand XML). This is a huge asset IMHO.

In a perfect world, perhaps this wouldn't be necessary, but in the real world, where projects get delayed this is very useful.

In the height of our panic (development) of the first phase of this system, we had services in varying degrees of completeness with scaffolding all over the place. The scaffolding allowed for some semblance of "end to end" testing.

Certain services were considered "code complete" by development. XML Aware testers allowed us to do black box testing against these services. Analyst started writing test cases with expected XML inputs and outputs. This was translated to XPath assertions, which, in some cases were automated (extension to XMLUnit/JUnit). The analysts invariably found defects in the code complete services. If we had waited to do traditional end to end testing, the project would have been even later.

"XML Aware" analysts had the side effect of making the analysis and design better in a major subsequent phase of the project as designers and analysts were competent in EDA style architecture and XML.

If you are lucky enough to have analysts on your team, I recommend getting them down into the weeds a bit and having them learn XML & the concepts of the architecture style you are using. If nothing else, they will ask one "dumb question" that will make you think. If you are lucky, they will start calling b.s. on you at appropriate times ;)

SOA and EDA make sense to everyone in IT from a high level. Spreading the knowledge of how they are implemented in your organization outside of the development team will result in a higher quality impl.

Want a job in Portland, OR - Java SOA/EDA?

Want a job in Portland, OR - quite possibly the best city on the planet?

We'll even move you here if you can slang wicked code. Sorry, you must have a green card or be a US citizen.

Insurance systems are a lot more fun then you might think. We pay above market, have old school benefits and OSCON is across the street every year.

We are looking for 2 full time senior Java developers. We are starting a series of new projects where we will be doing serious bi-directional integration with an IBM i5/iSeries.

Click here to apply. Job code is 23568BR.

EDA Lessons Learned - Build and SCM

See EDA Lessons Learned for the list

Everyone's favorite topic - build and SCM ;)

SCM is a hot topic within my company right now. For whatever reason, people get very excited/emotional about this topic (myself included). After living with various SCM tools, I prefer simple ones that aren't a barrier to distributed development. I actually used to like ClearCase - a lot. But now I think that it is overly complex and is a barrier to distributed development. ClearCase absolutely must be installed on a LAN for it to work - this forces a replication model for distributed development. This results in too much complexity for my taste and makes it very difficult to truly collaborate at the code level. There are OSS and commercial SCM products that were designed from the ground up to facilitate distributed development. Pick one of them if your developers are all over the world.

These lessons learned are from a large EDA style application, but like many of my lessons learned, it is applicable to a SOA project or any large integration project. I do not (yet) have an approach for company-wide SCM for SOA/EDA. That is certainly more complicated.

I knew a lot of this stuff from previous jobs, but walked into this project as it was in full development. It took *significant* effort to reorganize and fix. Like many things, planning from the beginning would have saved a ton of effort. Here is a list:

1. Plan the SCM repository carefully

   Don't have too many roots - don't have too few roots. Under the root, service, shared, web directories with modules beneath works well for us.

2. Branching model

   Choose carefully - here is a good article on branching models. I prefer "branch by purpose" - it keeps it as "simple as possible, but not simpler" as my boy A. Einstein would say.

3. Keep build simple
   • Developer should be able to run entire build locally easily
   • Developer should be able to build just required modules (i.e., shouldn’t have to run entire thing if just working on 1 service)
   • Developer should be able to make changes to the build (avoid single threading it through the build guy)
4. Deploy beefy build box

   Buy a nice high end workstation with plenty of memory and 2 very fast processors for a build box. You don't want the build to run on the build guy's local machine. It is much easier to automate from a centralized official build box IMHO. Anyone with access should be able SSH to the build box and launch a build. At a previous job, we even had a web page that allowed developers to launch builds (way to go Sarge)

5. Write and maintain simple instructions to get new developer’s environment setup
   Shouldn’t take more then an hour
6. Configuration

   Either keep config files for each environment (e.g., dev, test, prod) under source control and have deploy script use appropriate configs or script the setup of each environment. You don't want to do this by hand. This can be difficult with vendor products that want to invade your entire SDLC. Ensure that your vendors have deployment friendly support

7. Avoid Maven

   I am not up to speed on the latest version of Maven. I understand it is supposed to be better so take this with a grain of salt. At one point we had competing builds - Ant and Maven. Ant won because it is simpler and more people understand it well. We do use the Maven directory structure, however. I'll leave it at that.

8. Don't mandate a particular IDE

   Quickest way to piss off developers is to mandate an IDE. The build should never mandate an IDE.

9. Beware of tools that generate code

   If you must use these tools, at least either check in the results of the generated code (can be very dangerous if the code isn't just POJO style) or script the usage of the tool in the build (and watch your developers start to hate the build). And people wonder why I hate generated code ;).

ACORD is a Guideline NOT a Standard

Apologies to my readers who are not in the insurance industry and do not care about the ACORD XML "Standard". I have gabbed a lot about it lately. Perhaps this applies to other "industry standard" xml efforts. I know that it also applies to my experience with OAGIS. I also recently posted on canonical message format as well (related). I hope that this post and the comments that may ensue will put this topic to bed for a while.

While at the ACORD conference last week in Las Vegas, I challenged myself to interrogate every vendor, carrier, and ACORD employee on whether ACORD is a standard or a guideline.

To a person the consensus was that it is a guideline.

Perhaps I am just a stickler for words, but this really bothers me. I listened to executive after executive and industry analysts yapping and banging drums about "standards" and how everything is just great in the insurance world on the wire. I listened to the Gartner insurance dude spaz out about ACORD "standards" and then recommend that carriers fork it (excuse me, use ACORD as a guideline) in one sentence. The insurance analyst panel (Gartner, TowerGrouup, Celent - liked their guy Matthew Josefowicz the most)) was pretty comical - one of the more arrogant groups I have seen. Sniping at each other, challenging each other to dare to question the standards -- saying things like, "the standards have never been as close as they are today ...". They certainly all know more then I do about the insurance business, but someone needs to remind them that their PowerPoint presentations don't compile. They should take a look beneath the covers of a couple SOA projects and check to see what is really on the wire.

I talked to several aggregators and AMS reps who said that in their eyes ACORD is at best a guideline. They did say that since carrier xml is often ACORD-like and at least has some similarities, it makes things better then if everyone started from scratch. I can buy that - but that isn't standard.

I listened to a presentation from a reinsurance intermediary (Guy Carpenter) on an event driven impl. They forked the ACORD standard and have no regrets. By the way that was the best presentation of the show (even though it was called "Using ACORD XML Standards to Build Enterprise SOA"). I have a feeling that the presenter, Paul Fox didn't name it that.

I challenged some CIO level folks who were talking the standards bit. They quickly folded and agreed with me.

What bothered me the most about the ACORD conference was that there was no serious discussion about the realities of ACORD XML in practice today - no sessions, no critical thinking - zip. Everyone just seemed to wink and nod and say nice things about standards. Also, zero talk about 2.0.. Perhaps I just wasn’t in the right sessions.

Every sane person I spoke with is following the prune strategy accepted in 2.0.

I still believe that it is appropriate to use the ACORD guideline as an starting point for a insurance entity's canonical form - provided it is aggressively pruned and there is no requirement to keep up with future versions of the ACORD guideline. Most importantly, don't spread the illusion that your canonical form is "standard" -- that there will be some magic interop some day. There will not. This is ok. Owning your canonical form outright is far more important then conforming to a standard that doesn't really exist.

ACORD Conference - lots of vendor-pires & some that get it

Wow, lots of vendor-pires large and small here.

I had dinner last night with some folks from Skywire Software - we use one of their products, Insbridge.

What I like most about them is that they aren't a vendor-pires and aren't trying to be. They are SOA based and stay out of your way. I wish there were more vendors like this - that just offered service based software.

Instead, every vendor seems to want to invade your whole stack. I met with a couple of these vendors yesterday and told them straight - "I would never buy this - it will lock me in and make me miserable". Most of these vendors have crazy GUIs and scary proprietary scripties that are intended to make your life easier. "Easier" sadly also = lock in and esoteric skills that few developers want to acquire.

IMHO, SOA strategies that rely on lots of extensive clickie tool support are doomed to failure. Or at least lots of bites from various vendor-pires. I'm all for tool support - just not invasive tool support. As long as the tool isn't generating files with weird file extensions and isn't required to build a service, you are probably ok.

Insbridge. does have a clicky tool, but the way you call it at runtime is via a service. So the clicky is useful and contained - that works for me.

Update 17-JUL-06 - Now that it is possible to determine my employer by reviewing my blog posts, I must follow my employer's blog policy.

This is not an endorsement by Liberty Mutual, but my personal view. Vendors are NOT free to quote with attribution.

ACORD Conference

I, standards basher, am at the ACORD LOMA Insurance Systems Forum in Las Vegas.

The conference really starts tomorrow. I went to a technology forum run by insurance agents. It was fairly interesting. I have gone to several agencies and watched these people work. As the saying goes, "they don't call them independent agents for nothing". They all do business differently, use different tools (agency management systems (or none at all), aggregators, etc.), yet they want everything integrated and simple. Makes my life fun, what can I say.

At the agency technology forum, the word "standard" was tossed out there quite a bit. I kept muttering to myself, "what version?", "who's extensions"?

I had an interesting conversation with one of our aggregators in the conference hall. An aggregator is used by insurance agencies to do comparative rating. Basically, they just type in an insured's info once and click one button and get back rate quotes from N carriers (think Travelocity with airline tickets). If anyone should be benefiting from the industry standards that we all say have such a high payback with no questions asked, it should definitely be him. Guess what he told me? Every carrier that his system integrates with uses dramatically different XML on the wire. There is no "standard" XML that they send to each carrier.

Now I found that pretty funny.

He did say that it is nice because each carrier's XML has some resemblance, but then conceded, with some prodding, that the whole thing is b.s.

Anyway, I still hold out a slim bit of hope for Acord 2.0, but time will tell.

Don't get me wrong on standards - I think they are fine. I just get miffed when people blindly cite them as a panacea when I know under the hood what is going on isn't standard. I think we might get there someday with Acord XML - it just is going to take a long time IMHO.

EDA Lessons Learned - Performance

See EDA Lessons Learned for the list

For starters, specify a requirement or two on what acceptable performance is. Then, start measuring early. Better to know that you are way off the mark early on IMHO. Fixing services that are way too slow late in development or once they are in production often means rewriting the service from scratch in my experience (of course you still call it refactoring in meetings to make everyone feel better ;) ).

I have seen the most performance problems with Persistence.

Another thing to focus on early is how your services will scale. Our system resembles SEDA (Staged Event-Driven Architecture).

• Thread Safe - in our system that is built on top of JMS at the core, we use a JMS ServerSessionPool impl that makes adding threads (sessions) to a service configurable. This is a great way to scale and throttle a service. Sadly, we deal with some obscure stuff that isn't multithreaded - so we have to configure just one thread. For most services, however, you can have 10+ threads waiting to process events
• Competing Consumers & Shared Subscriptions - Anyone familiar with JMS understands competing consumers with Queues. We use SonicMQ, which, also offers this concept with Topics. We use these heavily. It is great because you can spread processing across machines. You can have 1 logical service running on N machines. When it is flooded with events, the processing is evenly distributed across the logical instances of the service.

Another area to focus on from the start is service granularity (the black art of SOA/EDA) - this is a lot harder to get right then a lot of people think. Especially when you are dealing with many services. Its like anything of sufficient complexity - there are 3-5 ways to do it right and 50 ways to do it wrong - pick one of the right ways.

Last thing I'll mention is async processing. This is huge in an EDA system - basically everything is async. But some developer's little minds don't like async and want to make it sync. You must find these people early and get them off the project. Just kidding (kinda). We had a couple problems with performance like this. Our login for instance was a dog at one point. This was because it was sync and the granularity of service calls was awful. So we "refactored" (where "refactor" means tore out old code and stomped on it and called it names, then wrote it again from scratch). We rewrote it in a way that was async. When a user logs in, an event is generated that the system responds to. Immediately after the login event is produced the first page of the UI is displayed. Other events in the backend are produced that the UI is interested in. So it listens for these events. The key is with the rewrite, we don't make that process sync. When the events the UI needs are produced, the UI receives them async.

EDA Lessons Learned - Scaffolding

See EDA Lessons Learned for the list

Scaffolding is useful all over the place in software. EDA and SOA really can benefit from it.

Scaffolding is especially useful in large new development efforts. Services reach code complete at different times. You are totally hamstrung if you wait until all services are complete to start testing. The scaffolding (i.e., mock services) can be used to simulate the services that are still in development.

Once developed, scaffolding can be useful when a service that is not available. If you don't have scaffolding, development and testing can grind to a halt while you wait for the service to come back up.

Scaffolding certainly doesn't come for free - you have to develop it like any other service & it can be every bit as complex as the real service. Its often worth the investment though. You don't need a scaffolding impl for every service you write. Mock services are particularly helpful for services that wrap a volatile resource or a service that is outside of your control (e.g., different org. that has a different development schedule).

A SOA or EDA that provides good location transparency makes swapping in/out scaffolding easier. For our EDA style system we just disable the real service and enable the mock service. The mock service subscribes to the same JMS Destinations, and produces the same event types. This is great because you don't have to modify anything on the clients to swap in the scaffolding. You can achieve the same thing with UDDI or RSS - I do really like the EDA style though - when you are working in an event cloud you get extreme decoupling which rocks.

SOAP vs. REST Pointless?

Steve Jones thinks that SOAP vs. REST is pointless. I assume from his post that he thinks that POX is pointless too.

I can understand that opinion. I think everyone in IT integration is frustrated that every wave that comes by that promises to give us massive reuse fails.

The good news is that IMHO services are here to stay. At the very least, they will get a fair shake. Especially with success stories like this from Amazon.

Regardless if Steve thinks the arguments are pointless or not, the WS-* standards aren't there yet. And SOAP really doesn't buy you all that much out of the box. Sure maybe if all the higher level standards amount to something it might, but I'm not holding my breath.

I think that dissent is important.

Steve argues as Anne Thomas Manes does that the tools will save us. He even cites XDoclet as an example. Funny that I use that as an example on why tools are not the answer.

Like I said, the good news is that services are here to stay. How about we just go back to the drawing board on some of the things that are broken instead of slogging on with the 60+ specs? I don't see that happening. There are too many vendor-pires too heavily invested at this point.

The only answer is enough simple OSS rebel frameworks and hopefully enlightened vendors that treat POX, REST, and whatever innovative ideas come next as first class citizens. The truth is there is massive premature standardization occurring. We might get there some day, but you can't just beat people into submission. IT Integration isn't the commodity that Steve describes. At least not where I work.

A big complaint I have against drinking the SOAP/WS-* Kool-Aid is I haven't seen any real success stories on following the WS-* approach with heavy integration. There are many around SOA, but strangely the specifics of what is on the wire are often left out. Admittedly I may be missing these - please let me know if you know of any.

In general I think that the concepts of SOA and EDA are what are really important today. If you can break things down at the proper level of granularity and use whatever you want on the wire (try a couple!), you will be fine in the end. People including me in the past do a lot of hand waving about standards, service contracts, tool support and what not which are important - there just not as important as you think - at least today.

Brighter Sun

Good to hear via James Governor that things are looking up for Sun.

I have largely made my living off of Sun's innovation (Java) for the past 10 years.

It has bugged me that they haven't open sourced Java, but sounds like they have decided to do so now eventually.

And sounds like Jonathan Schwartz is bringing a much needed new direction to the company.

I think that we all had enough of the negative Sun - I'm looking forward to a brighter Sun (impressive come back - you just have to love OSS).

Service Contracts = Hall of Mirrors

Nice post on service versioning by Dare Obasanjo. He refers to posts by Tim Ewald extensively.

I generally agree with their conclusions. Sadly, it demonstrates that "design by contract" with web services really is a hall of mirrors.

I beat the "contract first" drum as loud as anyone. The truth is, however, at runtime, the true contract between my services is very loose. This is due to lots of things - complexity of WSDL (I try to avoid at all costs), over engineering of XSD, industry standard schemas, etc.

Tool support / code gen from XSD used to be important to me. I came to services via CORBA and was enamored with the whole IDL thing. IDL in CORBA was much simpler - just basic C structs etc. It generally worked, but was certainly not without its problems. So when I first started writing web services in 2002, I dedicated myself to making WSDL an IDL (i.e., contract first). As I've said in various parts of this blog, that was a very painful experience.

I moved on from web services to just dealing with services that use XML on the wire. That XML has always been backed by XSD. As Dare and Tim recommend, the contracts are very loose.

Until recently, I have been ardently pro tool support with this (i.e., just simple XSD) approach. You can get any tool to work when you dumb things down (e.g., .NET xsd.exe, JAXB, Castor, XMLBeans). I always put the generated classes in some package like "msgmapper". I then write some helper classes that serve as a level of indirection to the generated classes.

I used to think it was pure unadulterated evil to generate XML by hand via JDOM, Xerces, etc. Now, I almost prefer that approach. Maybe I am just getting tired. Maybe I am taking the simplicity thing too far. Its just that all the tools come with baggage. You have to run them in the build, they spew out thousands of classes, developers forget to isolate them and they bleed deep into your code (and are hard to extract) etc. With the brute force approach, you go to the "msgmapper" package and there is your code that generates your XML - no fuss, no muss.

More and more I think that any tool that generates code (at least for high level languages like Java & C#) is to be avoided at all costs.

So are contracts important? Of course they are. The thing is the devil is in the details & there is no tool that is going to just give it to you or even make it easy. IMHO contracts are more of an art then a science.

Stonefly Hatch

I am anxiously looking forward to a fly fishing trip I have coming up on Oregon's famous Deschutes river. Some friends and I are trying to time the stonefly hatch perfectly. It is a very famous hatch. Trout go nuts for these huge orange bugs. The stoneflys (orange and the size of your thumb) crawl out on reeds and alder branches along the river. The wind blows them in. The soneflys flap their wings furiously to get out. The trout hammer them. It is a very brief hatch. It is tough to time it right.

The Deschutes is my favorite place to go. I totally zone out and don't care about anything else while I'm there. It is a "blue ribbon" (i.e., kick butt) fishery. The trout in the Deschutes are called "Redsides". They are legendary for their size and fighting ability. They are wild trout that have been there forever. The Deschutes is a big western river. Strong current all year long. Makes the trout strong.

I used to wet a fly at least every other weekend. Since I had a baby a year ago, I don't get out as much anymore.

What does this have to do with EDA, SOA, or ESBs? Nothing - I'm spent. I have been travelling a lot lately. Man that sucks the life out of you. I used to travel almost every week for 5 years. Now I thankfully usually only do it once or twice a quarter.

Anyway, I'll take some pictures.

Temporary MOM bigot

I'm currently a MOM bigot, but only because I haven't seen another viable way to do extensive service oriented integration today.

I yapped about Message-Centric vs. Service-Centric a month or so ago.

I heard that SOA Software acquired Blue Titan this week. This is an interesting development. Here is a good post on why this happened (via Stefan Tilkov).

As I tried to get going on the Yahoo REST message board and in various posts in this blog, this is where I'd like to see things go. Leverage HTTP, but add:

• guaranteed delivery
• durable destinations
• event driven support (i.e., pub/sub)

It is ok to support SOAP, WS-* for the true believers, but do not mandate it. If anything, tolerate SOAP/WS-*, but prefer POX and REST. I wouldn't buy something that preferred SOAP/WS-* and had an extension or subset of support for POX and REST.

Now, I have no idea if Blue Titan even does this (I haven't used it) - it sounds like it does some of this or will. Anyone worked with it?

Ideally, I'd like to see OSS impls or at least dual licensing of this type of thing. This type of infrastructure is too critical to rely on a small vendor for. I can't afford another vendor-pire.

EDA Lessons Learned - Content Based Routing

See EDA Lessons Learned for the list

I have found Content Based Routing (CBR) to be helpful in EDA when a sink receives events, processes them, and then produces a subsequent event.

I like CBR because it pulls the routing logic out of the code. The code just processes an XML input and builds an XML output. It does not set the next Topic explicitly. The routing is controlled in the configuration of the sink. This keeps the code simple and reduces coupling. You can also change the routing without changing the code.

We had success using XPath to determine the next Topic. You can use anything in the XML output as part of the next destination. Or you can have several XPath statements building up the next destination based on attributes/elements in the XML. Examples include event type and event status.

I found an article from IBM from way back in 2000 advocating this approach: Using the JMS API and XML in content-based routing. Glad to see I'm so ahead of the times ;)

Spitting up the J2EE Hippo - now what about WS-*?

Richard Monson-Haefel continues to take the gloves off on J2EE and WS-* (at least the Java WS-* impls). I might not be reading closely enough, but I wish his firm, Burton Group would do the same on WS-* (I'm a customer). He and Burton Group have been consistent on J2EE, but Richard's boss Anne Thomas Manes continues to beat the WS-* drum. I read a couple of her pieces on a plane this week and she has totally drunk the WS-* Kool-Aid. She isn't a total zealot on it or anything, but she is astonishingly pro WS-* in the face of a lot of criticism and lack of success stories in the field. To be fair, it is possible that I just haven't found the right research. Richard - are you (or have you already and I missed it) going to take the gloves off on WS-* in your Burton Group writings? I love your blog, but it only makes a real difference if it is in the official Burton Group position papers.

Also, if J2EE is on the ropes today, who is to say that WS-* won't be in a couple of years? Given its trajectory compared to where J2EE was in the beginning, I think it is quite possible that WS-* will die a much quicker death then J2EE.

Update 08-May-06

To clarify, I thought that Anne's research was very thorough in terms of coverage - it was quite useful. I just was disturbed that given the history on things like J2EE she so easily assumed that WS-* will survive and thrive. Perhaps its her job to be hopeful like that and to try to get momentum going in one direction. My focus is solving integration problems today and in the not so distant future. Perhaps our requirements are different.

Update 13-May-06

I found some interesting links to Anne's response:
James Governor
Mark Nottingham - (he used to chair the WS-Addressing WG at W3C)
Dave Johnson

EDA Lessons Learned - Canonical Message Format

See EDA Lessons Learned for the list

I posted about Canonical Message Format a little over a year ago. I still agree with most of what I said then.

I have been exposed more to the Acord insurance standard since then. Sounds like Acord is addressing a lot of my complaints in the 2.0 version. Main quote that caught my eye is:

V2 will be delivered with an option to "slice" or prune the schema down into smaller files with the message content of your choosing.

That says a lot on how "standard" the XML really is on the wire in the insurance industry. This is essentially admitting defeat IMHO. Nothing wrong with it - just facing reality. The truth is that every carrier has differences - you make a valiant attempt to map to the standard, but you still have 20% more to go. There is an extension mechanism to account for this.

The trouble I have with industry standards like Acord is that if you follow them without pruning the original schema and making it your own (i.e., owning your contract), you have a very loose contract between services.

I'm all for using the industry standard for integrating with third parties. I think it is ok to use internally too, but it makes me nervous. People tend to toss the "standard" term around and act like the fact that it is being used as the canonical message format means something major - as if you are going to get a huge payback for it. I have never seen anyone get a big payback from any XML industry standard. And it takes a lot of effort to follow the standard. New versions come, the company has extensions, etc. At some point, people throw in the towel and have something that looks like the standard, but the chances of it resulting in any sort of out of the box interop with anything are nil.

So I think its mostly about setting expectations ... it is ok to use an industry standard as your canonical message format. I think it is wise to go into it with the "prune" strategy. Just use the standard as a reference. Don't event attempt to use the standards schema files. Just lift the structures. And dumb it down as much as you can to make it a tight contract. You will stray from the standard eventually - better to just admit defeat going into it IMHO. No shame in that.

Here are some more thoughts:

1. Avoid request/response designation in your root elements.

   EDA is not sync - don't act like it is. Even if you have some workflow that is sync, don't bake it into the messaging model - it will likely change over time. The request/response in the XML will just confuse people.

2. If using XML schema, keep it simple - avoid the fancy stuff
3. Question elaborate tools that make managing your canonical message format easier

   These tools often make it easier for you to make things more complex then they need to be and lock you in. They typically require training - so only a couple people know how to use it (single threaded through them). Your canonical message format should be simple & should only require something like XML Spy, source control, and a web site for easy reference.

EDA Lessons Learned - EDA and Object Oriented Design

See EDA Lessons Learned for the list

Event producers and event sinks are very often services written in an Object Oriented language. All the same OO rules apply. OO Design Patterns are still applicable.

This is a pretty minor point, but I have seen a lot of people throw this stuff out the window with EDA/SOA.

In my opinion, services (i.e., event producers & sinks) should have good separation of:

• service handler (should have minimal code - just delegates to message mapping and business logic)
• message mapping (e.g., O/X)
• business logic
• domain model
• data access layer

An aside I'll add is that the Spring Framework also works just fine for Java based services. I only mention it because I have found that its usage tends to have a positive affect on the OO design of services.

EDA Lessons Learned - Logging

See EDA Lessons Learned for the list

This is a small one, but better to get this right from the beginning - we didn't, but are almost done fixing it. And this isn't really EDA specific - its applicable to any large scale distributed programming effort. Our problem was that various developers logged different things. Most used log4j, some didn't. Some configured it one way, others another. Some logged everything to INFO - some used DEBUG properly ... What you want is uniform logging.

We have approximately 50 different services. Each service is typically an event source and sink. Some are just event sources, others just sinks.

Our system has an admin GUI that lets you see the last 10 events (configurable). The runtime just keeps various buffers for incoming and outgoing events. When a user clicks on a particular service, he sees the incoming/outgoing events.

But logging to log files is a different issue. Obviously, this is where you go looking when there is a problem in the system.

Due to privacy / regulatory requirements, we have to be careful what we log. When there is an error, however, you want to know as much as possible.

Here are some thoughts on how to log. As I said, better to do this in development then to dedicate a large portion of a maintenance release (like we did) to getting it right:

• Write a brief doc / WIKI entry on logging policy. Get agreement on what categories to log to and when (e.g., DEBUG, INFO, ERROR)
• Service entry/exit logging
  1. In the original event source, generate a UUID. Put this in the event header
  2. Log the UUID, event type, status (if applicable), and other appropriate meta data
• If you must log the payload of the event when there is an error, scrub the private information (e.g., SSN, DOB, etc.)
• If you are using error queues, you have no reason to log the event body. Just log the stack trace, event meta data, and event history. The error message should also contain the stack trace, event meta data, and event history. If you have an Error Queue Admin tool like we do - it will protect the sensitive information
• Keep the log config files out of the artifacts (e.g., .jar) that you deploy so that if there is panic in prod, you can turn on DEBUG by editing the config

Like I said, this isn't rocket science, but I figured I would point it out. You'll save yourself some angst down the road by spending a couple hours getting consensus from the beginning.

The Big Boys

Saw IBM's big bet on SOA via Simon Tilkov.

To quote the article:

Clearly SOA is key, if not the core, to IBM's software strategy. We will of course have to wait and see if this "shock and awe" approach to dominating a market will work - not to mention solve the customer problems. I do wonder how 31 separate products (so far!) can really deliver the simplicity and agility that is meant to be at the heart of SOA. More importantly, SOA is about providing a solution, not selling an even more complex collection of products.

I don't really have anything to add to that. I tried to, but I just keep staring at my screen muttering things.

EDA Lessons Learned - Choose Topics over Queues

See EDA Lessons Learned for the list

If you are using JMS as the back bone of your EDA, choose Topics (i.e., pub/sub) for 99% of your Destinations. I guess that is pretty obvious - I guess it is more applicable to the Messaging-Centric ESB world.

Exceptions where I think Queues are appropriate:

• Error Queues
• Existing integrations that have queues (I'd still route it through a Topic before it got there)

There are other times where Queues are appropriate, but just triple check that it is appropriate and if in doubt, use a Topic instead.

Topics have a lot of benefits over Queues:

• Facilitate event driven architecture
• Not Point-to-Point (you have a prayer against the n(n-1) problem (where n= # of systems to integrate)
• Very flexible if using in a event workflow - can fan out/fan in easily
• Ease of debugging (can snoop on a Topic)
• Can do 1:M Request/Reply (one request, get N responses)

Also, use hierarchical Topic names. Hierarchical Topic names are great because they:

1. Help to avoid dependence on message selectors (slower)
2. Organize things
3. Enable listening to a pattern - for example:
   If you have Topics in this pattern (several different "EVENTTYPE" values: COMPANYNAME.DIVISION.PRODUCTLINE.PRODUCT.NOUN.EVENTTYPE
   Say you are interested in all of the event types ... you can listen to them like this: COMPANYNAME.DIVISION.PRODUCTLINE.PRODUCT.NOUN.*

Unless you don't need it, use durable subscriptions with your Topics - they work great and will bail you out of some hairy situations. While durable subscriptions were originally intended for momentary connection problems, they can be used to help bridge the EDA & batch world. You can register a durable subscription and then just connect to it at night for instance. You accumulate events all day on the durable subscription and then just drain it as part of the batch run.

To be fair on my Topic preference, I do take advantage of a SonicMQ feature that makes a Topic like a Queue in terms of the ability to have competing consumers (i.e., different connections (can be on different hosts) competing to drain the queue) (Shared Subscriptions). Other vendors do similar things. It is a pity really that Sonic's "Shared Subscriptions" isn't in the JMS spec.

SonicMQ 7

Yesterday I watched / listened to an early access web cast on SonicMQ 7.

I have been using SonicMQ for about 5 years. Rock solid - just works. And when it doesn't, their support people are very helpful - they dig into your problem and crush it. If the problem is a defect - they fix it. If the support person can't figure it out, they work with engineering. If they still can't figure it out, they get engineering on the phone. And they keep following up with you. Really can't say enough about the success I have had with the product.

Anyway, there are some improvements that will help me that I am looking forward to. We use pub/sub, durable subscriptions, and Sonic's "shared subscription" feature extensively. They apparently have some fixes/enhancements that will prevent messages from getting trapped on brokers when bad things happen. I have had this happen when consumers of a Topic on a node in a cluster get horked.

They also are introducing (don't know exact name) "MultiPublishers" and "MultiSubscribers". Apparently, this feature was driven by wall street customers who have thousands of Topics. It takes 100ms to register a subscription at start up. This adds up when you have thousands of topics in use. So, my understanding is that with "MultiSubscribers" and "MultiPublishers" you can add Topics to a Set and register them all at once.

They also apparently tricked out the "Continuous Availability Architecture" (CAA) further. This is wicked cool and I don't know anyone else who does it. Basically, you can have a client publishing to BrokerA, kill -9 BrokerA, and the client just fails over to "BrokerB" and continues publishing ... even within a transaction. And this takes 10 minutes to set up. You don't need exact hardware, bios settings, etc. (i.e., hardware based failover). You just configure your setup that way. Another thing that is huge when fault tolerance really matters (more and more these days). You can recover within 10 seconds vs. 10 minutes. Wicked cool.

Anyway, I'm not the type to spaz out about a product like this - I am a huge skeptic from years of being let down by integration companies. Sonic just makes things stupid simple like they should be - so you can get on with SOA & EDA.

Update 17-JUL-06 - Now that it is possible to determine my employer by reviewing my blog posts, I must follow my employer's blog policy.

This is not an endorsement by Liberty Mutual, but my personal view. Vendors are NOT free to quote with attribution.

EDA Lessons Learned - Service Contracts & Event Stream Design

See EDA Lessons Learned for the list

The contracts between services are the most important thing there is in either SOA or EDA IMHO. You get this right and your future is bright, get it wrong and it is dim. Pretty much that simple.

I came to SOA/EDA from CORBA, J2EE, and DCOM projects. I thought that IDL was the only way to fly. When WSDL came out, I treated it like any other IDL. I was one of the people who endured significant pain with WSDL when it first came out because I was such a believer in "contract first" design/development.

The first time I started muttering "what is the contract?" when discussing OO design, service design, anything software related really was when I was mimicking one of my mentors (Dave Read when we worked at eXcelon). Just because I knew he was wicked smart and he said that a lot, I made sure I knew what he was talking about and added it to my repertoire.

So I am still a huge fan of contract first development, but it isn't always practical - especially with EDA. For one thing if you wanted to use WSDL today with EDA, you really couldn't as WS-* doesn't support pub/sub yet and WS-Notification & WS-Eventing are being merged into WS-EventNotification. I tend to think they will be too rigid when they are updated, but we'll just have to wait and see. Anyway, just because you don't have an IDL, doesn't mean you shouldn't think about contracts.

Service contracts in EDA aren't usually request/reply like they typically are in SOA. It is easier to get your hands around contracts with request/reply. With event streams, you have N services listening to an event and possibly publishing N events to N destinations. A service may listen to many destinations or a pattern. A service may receive XML documents conforming to different schemas. So you pretty much have to let the whole IDL thing go ... unless you just allow "any" or whatever it is in XSD (which is pointless).

This isn't very concise, but for me, contracts in EDA are:

• Event input payload(s)
• Service input destinations (set/patterns)
• Event output payload(s)
• Service output destinations (set)

Here are some more details on my opinions on service contracts and event stream design and EDA:

• Devote appropriate time to service contract/event stream design
  1. Have several sets of experienced eyes review
  2. Review a number of times - allow for "soak time" as my friend Sarge would say
• The goal is low coupling and high cohesion
• Anticipate event stream flow changes
• Consider persistence when thinking about contracts
• Get the granularity right (narrow vs. coarse) - the truth is somewhere in the middle
• UML sequence diagrams with embellishments (persistence, state change, etc.) and annotations works very well for technical design and review
• Great website/book - Enterprise Integration Patterns

EDA Lessons Learned - Persistence

See EDA Lessons Learned for the list

Our EDA is Stream Event Processing based.

There are *lots and lots* of events. It blows your mind to listen to all of the events at once (just subscribe to # in SonicMQ). It is like watching the Matrix. Most of the events aren't terribly meaningful to a typical subscriber. Many of the events are only used in the source system in "event workflows".

EDA is highly distributed. Our system has many different processes and machines publishing and subscribing to events. Persistence is very easy to get wrong in an environment like this. The first lesson is to put enough data in each event so that subscribers of the event do not need to gather more data from the source. Also, in terms of inserts and updates, you have to be careful of table / row locks etc. If you are not careful, you'll have N processes with N threads on N machines trying to hit the same data source.

For us, a lot of these "ordinary / non-notable to downstream service events" are used in event workflows. State is important - you have to put it somewhere. The key is to think very carefully about where you put it and non screw it up.

Here is a list of some general thoughts on the subject:

• As mentioned above, put the appropriate amount of data in each event so that subscribers do not need to gather more data from the sending system
• If you are using persistent messaging & durable subscriptions as your EDA backbone, trust it! You do not need to persist state in each service that handles an event. If you do, you are going to regret it
• XML shredding is expensive and results in a high defect rate (on the way in and out)
• Separate transient (i.e., part of event workflow), terminal state (i.e., completed event workflow), and reporting data bases (ODS, Data ware house, OLAP)
• Beware of O/R tools
1. Work ok in experienced hands, but caching difficult if multiple services in different JVMs have separate cache, mistakes can be hard to fix
2. Use proper granularity
3. Emitted SQL not always performant
4. Often ends up being more complicated then JDBC
• Consider XML database, high performance reliable file system, or caching solution for transient data

EDA Lessons Learned - Error Queue Administration

See EDA Lessons Learned for the list

When a service is processing an event and a non-recoverable error is encountered, the poisoned event/message is routed to a named error queue (i.e., not the dead message queue).

It is painful and error prone to handle error queues manually (e.g., command line programs & email requests).

A web based error queue administration tool is very helpful. Due to regulatory requirements, we have authorization requirements on who can see the error queues (i.e., read only) and who can act on the error queue (i.e., edit access). All actions must be logged etc.

Some useful features:

• View all error queues & error count
• View queue details (list all errors)
• View error detail (lists event meta data (e.g., event history, stack trace (root cause of the error), original headers, properties, event body etc.)
• Redrive event
• Edit & Redrive event
• Delete event

EDA Lessons Learned - Event History

See EDA Lessons Learned for the list

Events Driven Architecture is about as loosely coupled as it gets.

One event can result in the generation of other events that can result in still more events ad infinitum. This is how our system works - there are many events. For example, one user click is 1 event, but it very often results in 10+ other events. The "event type" typically stays the same, but different attributes on the event change (e.g., the status). Also, the event is re-published to different destinations.

When there are problems in the system (e.g., an event is routed to an error queue), it is very useful to know where the event came from. Event history is also very useful in bringing new developers up to speed on how the system works. In a system that has many different types of events and many different services processing events, it is very easy to get confused. Also, documentation doesn't always keep up with the system. Event history is a form of system self documentation ;)

We record basic history on events. Each time an event is generated, history is appended to it. We track information like service name, service type, destination name, destination type, and host name. We are using JMS so we record the history in a message property.

OSS CEP

Looks pretty good ... OSS CEP: Esper

EDA Lessons Learned

I put together a presentation at work on some lessons my team learned on EDA/ESB related topics a few months ago. I have made a couple friends recently via my blog. They have expressed an interest in this information. While I can't ship off the original presentation because:

1. it would bore them to tears
2. it contains proprietary information

the guts of it can be shared as they are standard architecture issues that anyone dealing with EDA/ESB will likely encounter. I'm curios what other people have experienced. Hopefully I'll learn something from sharing :) This list is by no means exhaustive, I might add to it as I think of other topics.

So I'll be blogging on these topics the next few weeks. I'll update this post with links as I complete the topics (no particular order).

Here are some of the topics I'll cover:

1. Persistence
2. Service Contracts & Event Stream Design
3. Choose Topics over Queues
4. Canonical Message Format
5. Scaffolding
6. Transactions
7. Performance
8. Build/SCM
9. XML Aware Analysts
10. Batch vs EDA
11. Content Based Routing
12. EDA and Object Oriented Design
13. Shared Code
14. Logging
15. Config Files
16. Error Queue Administration
17. Event History
18. Aggregation

WS-DeepBreathing

Saw this by Dave Podner on Richard Monson-Haefel's blog.

Pretty funny. I'd add a #6, however ...

6. Cut your losses and move on

Message-Centric vs. Service-Centric

Stumbled across this on Dave Orchard's blog. It talks about leaky abstractions in SOAP. Also discusses how SOAP hides the underlying protocol. This is the main reason that REST or POX/HTTP appeals to me more then SOAP - they say you have to embrace the network and not hide it. SOAP just brings too much unnecessary complexity to the table for my taste. It truly is the doorknob to hell.

When I was working on "Aspira" - the EDA (Event Driven Architecture) bus that never was, we dealt with this by intentionally forbidding any:any transport. You couldn't for instance have an HTTP endpoint talk directly to a Socket endpoint. Every transport was designed to have an adapter. That adapter would deal with the transport in its native form and get the event to the persistent / trusted / consistent messaging layer as fast as possible. In short, it was "Message-Centric" rather then "Service-Centric". Decent write-up on the differences by CapeClear advocating the opposite of what I think.

The nice thing about the "Message-Centric" approach is most of your services end up being vanilla - they use one of the standard service types, which, deal directly with the messaging layer for its inputs and outputs. They get the benefits of persistent messaging, clustering, consistent security, fail over, durable subscriptions, standard error handling (e.g., rollback and pause, route to error queue). Yes, they do hide the network, but that is ok, because network problems have been accounted for. And the services don't care about the payload - so long as it is XML. The result is your service impls are stupid simple. They just do their little bit.

Dave Orchard's post prescribes some proposals for fixing the problems he sees. Sounded good enough for me. I just think the REST or POX/HTTP approach is the better way to go. WSDL/SOAP intended to ease pain, but sadly, I think they make things more complicated then they need to be. I think it is far simpler to just embrace various protocols directly and stay in control. Yeah, at the outset, it might feel a bit more complicated, but in the long run, you save yourself pints of blood.

Anyone care to talk me out of the "Message-Centric" approach?

Be Afraid

This post by Andrew Townley is really well written. Scared the crap out of me.

Anne Thomas Manes says JAX-WS will fix this and everyone will use the attribute approach.

Like Yoda and Andrew say, "Once you start down the dark path, forever will it dominate your destiny".

Update - plea for help
Can someone from the future please send me an email and let me know how WS-* turns out? I am sick of worrying about it. Thanks in advance for your help!

Sonic Software - Actional in Seattle

I went up to Seattle today to listen to Sonic Software and Actional (Progress/Sonic recently acquired Actional) talk about SOA/Event Driven Architecture best practices and maturity models and such.

I had to get up at the butt-crack of dawn to get there in time. It is a 3 hour drive from Portland to Seattle - the meeting started at 8:30. Not fun. So early that I forgot my belt. Good thing is I look particularly handsome without a belt unlike most people.

I also participated in a forum talking about lessons learned and SOA challenges. It was interesting to hear other points of view, etc.

My company uses SonicMQ as our JMS provider. One of the better ones out there. Coolest thing they do is "shared subscriptions". Basically lets you scale a subscription horizontally. You just use their clever little shared subscription syntax when you register the subscription. Then, multiple connections on N hosts are considered one logical entity and they shared the burden of processing the messages for that entity. Basically, you get the benefit of competing consumers on the Topic. It is just like it is a Queue, but you can have other shared subscriptions and normal subscriptions listening to the same topic at the same time.

I didn't know much about Actional until recently. I had lunch with Sonic's new CTO, Dan Foody and some other Sonic guys. Dan was the CTO at Actional. They have some cool products. Using their products, you can monitor a SOA/EDA deployment end to end. It traces transactions through all your services. Apparently they do this by some byte code magic where they listen at all the appropriate times. They append headers to HTTP/XML requests, JMS, CORBA, RMI, etc. that allows them to trace the logical transaction where it goes on the SOA/EDA. Wicked cool.

We do this by hand currently - as event bounce around, we scribble which hosts/Destinations they were routed on, which services processed them, etc to a "event history" message property. This information is really useful when events end up on error queues etc. Also nice for debugging and bringing new developers up to speed on the system. Self documenting and what not. We just do this for JMS though ... we don't have the same visibility for HTTP based services. Anyway, the Actional stuff seems pretty useful. I want to give it a spin.

Update 17-JUL-06 - Now that it is possible to determine my employer by reviewing my blog posts, I must follow my employer's blog policy.

This is not an endorsement by Liberty Mutual, but my personal view. Vendors are NOT free to quote with attribution.

Sad but true

I came across this over the weekend.

Very sad, but so often true.

Just how much REST - 'Web Style' are we talking about here?

##########
Update I posted the gist of this thread to the SOA and REST Yahoo discussion groups. Here are the threads:
Service Oriented Architecture
REST
##########

After about a week of quiet, the panic is back on REST ("Web Style" according to Tim Bray which I think is better) vs WS-*.

Tim Bray takes the gloves off:

The End of SOA

Important, I Think

He is preaching to the converted when it comes to "Web Style" vs WS-* for me. Perhaps the acronym "SOA" will eventually meet its demise, but, we'll just have to come up with something else. Sadly, "Web Style" doesn't cut it for every type of service. We've had some flavor of "SOA" since ARPAnet. We like services. And I'm sure they'll be something after "Web Style".

But I have to wonder, just how much Web Style do we need?

Which types of integration is it appropriate for? I'm specifically talking about internal integration within a company. Web Style all the way for integrating between companies IMHO.

I have my own shiny object -- its messaging (e.g., JMS). While I don't think it should be used for everything, it works very well for providing the "plumbing" for complex integration. Yes, JMS is just a Java thing, but good JMS providers have C, C++, COM, .NET, & HTTP APIs so it is platform independent enough to get the job done. I'd love to see a truly platform independent reliable messaging, but as the REST guys say re:HTTP, JMS is a technology that works and is here today.

I like Web Style as opposed to WS-* for the class of service that should be over the web. I just don't see either being that great for extensive internal integration compared to something like JMS that provides things like clustering, fail over, guaranteed delivery, error queues, and durable subscriptions.

As part of the future state architecture planning I am participating in, we are trying to identify the high level types of integration that are common for us. I think they are common for most large companies. Here they are:

Update 18-APR-06: Added some prose around each integration type.

1. Sync Request/Reply

   ServiceA.Thread1 sends request and waits for response from ServiceB.ThreadX.

2. Async Request/Reply

   ServiceX.Thread1 makes request. ServiceX.ThreadX receives response.

3. Async 1:1 (fire/forget)

   ServiceA.Thread1 publishes message and does not wait. ServiceB.Thread1 receives message. ServiceA.Thread1 may or may not be aware of ServiceB.Thread1 (depends on protocol used).

4. Async 1:M (pub/sub)

   ServiceA.Thread1 publishes message. ServiceB:ServiceN receive message. ServiceA.Thread 1 unaware of ServiceB:ServiceN.

5. Async 1:M Request/Reply (pub/sub Request/Reply)

   ServiceA.Thread1 publishes message. ServiceB:ServiceN receive message and send response. ServiceA.Thread1 receives N responses. ServiceA.Thread1 unaware of ServiceB:ServiceN - just know that N responses were returned.

6. Async M:1 (event stream subscription)

   ServiceB:ServiceN publishes message. ServiceA.Thread1:ThreadN receive messages. ServiceB:ServiceN unaware of ServiceA.Thread1:ThreadN

7. Batch Feed

   ProcessA sends file to ProcessB

I think Web Style is perfectly fine for a certain class of Request/Reply services. For example looking up existing customer information. It definitely wouldn't be my first choice, however, for something transactional like creating an order. Not saying I'd never do it, just wouldn't be my first choice. My problem with using Web Style at all for heavy integration is the coupling and complexity that starts to add up when you have a lot of services all doing Request/Reply. Each client has to handle errors and retry logic. Certainly not rocket science, but it is more then I want to deal with.

I think that Batch Feeds could also be done via Web Style. They don't tend to be coupled sequentially the way I described Request/Reply above.

Once you get to anything async, however, Web Style quickly falls down for me. Lets start with "Async Request/Reply". An example is requesting a report from a third party that takes processing time (say 1 day). You make the request, and want to get the reply async. With Web Style, people often end up making the request for the report, get an id back and then polling at an interval for the results for the id. You add logic so that you don't poll until you think the report is ready, etc. This is complexity. Again, not rocket science, but lines of code with defects in it to be sure. To be fair, you could have the report service call you back which isn't as bad - again, however, there is non-trivial complexity here when you deal with the error cases. If you are writing one service, it is no big deal - its when there are more then a few where you get a cumulative amount of complexity that starts to get your attention.

For "Async 1:1", you can do that with Web Style like Request/Reply. You do have to contend with error cases, however. And "out of the box" you don't get guaranteed delivery. You can't send the message persistently so that you know that the send being ack'd means something. You can certainly do this by hand - and it isn't that hard, but again, this stuff tends to add up and every time you roll your own guaranteed delivery, there are going to be defects. Not the type of stuff I want to deal with.

Alright, so now we are getting to the harder stuff - "Async 1:M", "Async 1:M Request/Reply" and "Async M:1". This is where I really struggle with Web Style. I've tried to challenge myself to find a way to do this with Web Style, but haven't gotten really far. Admittedly, however, I tired fairly quickly - please let me know if you know of something.

How would you do "Async 1:M"? This is classic pub/sub. A client sends 1 message and N subscribers listen to the message. The client has no idea who is listening. I don't know of any way to do this w/o writing a ton of code.

How about "Async 1:M Request/Reply"? This is when you send one message and get N responses. This is pub/sub request reply.

Last one on my list is "Async M:1". This is 1 service listening to N destinations / Topics for events.

Don't get me wrong. I'm all for Web Style. The sooner SOAP and WS-* are deprecated, the better, IMHO. I just don't see Web Style as a silver bullet for integration any more then anything else is. Not saying that Tim is saying that - just think that there are lots of cases in integration where Web Style OR WS-* isn't the best choice.

SCA - Pretty bold

I heard about SCA when it came out. I was on a bit of an anti standard tear then and I was tuned out a bit in general. I've started looking at it more closely because I am looking at IBM ESB 7.0 a little bit and I am addicted to the Internet and I read a lot.

David Chappell has a good post on it.

Anyway, SCA is way better then JBI. For starters, it is platform independent and IBM and BEA are both behind it. Man it must suck to be Sun these days. I agree with David that SCA is not w/o politics. Basically, IBM and BEA are coming to wrestle the JCP process out of their hands and start to take control of the future. Makes sense in a lot of ways. Anyway, I don't really care about that crap.

SCA is like M$ WCF/Indigo but for Java. Good stuff, now we are making some progress. There is already an Apache impl underway (Apache Tuscany).

So I looked at the SCA Whitepaper. Didn't see anything that made me writhe on the floor like when I looked at JBI the first time.

Also, happily WSDL/SOAP doesn't appear to be mandated on the wire. Yippee! I'd also agree with David, that this doesn't look real good for J2EE / app centric model. Thank goodness, there is a way out. There is a REST binding as well, but I doubt it will meet the requirements of purists. Really more of a POX binding. Good enough for me. I wouldn't be surprised if this is how WSDL/SOAP eventually gets deprecated. Everyone will just not use WSDL/SOAP when there is the REST/POX binding and they'll slow boil it to death.

The spec is really young, but looks promising. They had the word "pub/sub" in there so I'm happy. Not included in this version of the spec though. Perhaps I'll have to try to contribute.

Probably a couple years before it starts becoming mainstream, but color me hopeful. Wow, maybe I'll have to stop being so full of panic now. I doubt that will happen.

Update (after Easter naivete wore off): Marc Fleury at Jboss no likey SCA. If JBI didn't suck, IMHO, he'd have a better argument. But it does - Java specific SOA doesn't really get you far - at least in my world. Jeez, how did I miss this whole thing last Winter? I was tuned out, but not that tuned out.

Dangit, I probably am going to end up recommending the same thing .. really sad.

Other applicable links:

Macehiter Ward-Dutton

James Strachan

James Governor

Future state

I am on a small distributed team planning the future state integration architecture for a $6B business. It isn't as glamorous as it might sound (replace glamorous with painful depending on your outlook). It is just a small part of my job. Most of time is spent dealing with present day integration and maintenance of past integrations. Plus, we all know how integration architecture standards go -- they are rarely followed when project deadlines loom, etc. But, I have learned a lot about integrating systems during my career and I have strong opinions so I'm going to give it my best effort. I don't think it will go 100% the way I think it should (nor should it), but I hope that I can at least save some poor sap some agony by keeping things as simple as possible, but not simpler.

Should be interesting to see where this goes.

I'm trying to remain objective and not just be knee jerk anti WS-* and anti anything EJB (i.e., ESBs built upon app servers). It is hard to a) keep my biases in check (established from in the trenches angst) b) compete with the industry echo chamber that says WS-* is good and you need an app server for every Java process.

I wonder if I'll end up recommending the same thing I said the last time I participated in planning a SOA integration strategy in 2002. How depressing would that be? There better be better options 4 years later.

The previous company I did this for couldn't be more different then the current. For starters it was a $700M business and is a high tech company. The integration challenges were just Java vs. .NET & Unix vs. Windows + a bunch of COTS packages (e.g., SAP, Siebel, Onyx). The current company has every platform ever invented - just like most major financial services companies. We have Unix, Linux, AS/400 (iSeries), MVS (z/OS), VSE (z/VSE - been around for 40 years), just to name the ones you might have heard of.

I don't know where this will go. Perhaps we'll just punt. It would be easy enough to lay something out that a) looked respectable, b) would sit on the file server unaccessed

I do know one thing - right now is a very bad time to place bets on future state. Sadly, there is a lot of uncertainty right now IMHO in integration land. I would have thought things would have come further in 4 years, but here we are. We have WS-* & SOAP, Service Centric ESBs, Message Centric ESBs, App Server Centric Don't-Write-Us-Off ESBs, EAI Vendor Centric Repackaged as Not-Sucking-Anymore ESBs, WS-* vs. REST, JBI (quickest spec to die in history? I set the over under at 1 year 6 months ago and still am comfortable as the house), SCA (Service Component Architecture), and vendors drafting standard after standard and having to sing Kumbaya before the whole house of cards falls down (dramatic?).

SOA is here to stay, that is for sure, but SOA is just an integration style. We've had that style since before I was born (don't tell my boss), but it is finally mainstream. You'd think there would be some consensus in 2006 around web service impl of SOA, but, the battle lines are just being drawn it seems. There is more uncertainty now, then ever IMHO.

So anyway, I'll likely be posting about this future state bit-o-panic a bit the next few months.

Oh yeah, so what was my recommendation last time? I suggested that we avoid SOAP based web services and find the best JMS provider (Microsoft still doesn't really do messaging in 2006) we could that would a) scale b) cluster c) fail over d) notify/monitor e) secure clients, destinations simply, support Java, C, C++, COM, .NET, & HTTP clients, maintain a simple Canonical Message Format (de-crufted OAGIS at the time), and build a lightweight ESB layer on top of it and then write services against the ESB layer's api.

WS-DeathStar

I want a T-shirt dangit.

WS-DeathStar

Crappy Release - events to the rescue

We had a monthly release of an auto insurance system I spend a lot of my time working on this past weekend.

It didn't go very well. The week leading up to the official code freeze uncovered some critical defects. Then we found some post code freeze. The team pulled it together and got the system in good enough shape to release.

The release was deployed to the production environment without incident on Saturday. I multitasked showing my aunt and uncle who were in town from Sun Valley, ID Multnomah Falls and occassionally glanced at the CrackBerry. We got through Monday with only one problem and it was "just" the monitoring software. So we'll fly blind a few days not knowing what Mercury Biz Activity Monitor says our performance is in Atlanta vs. California etc. Big deal right? My boss says we missed a Choice Point outage so actually it is somewhat of a big deal, but anyway ...

But today, the wheels fell off. 2 major problems were discovered.

This is going to be a total pain the rest of the week to resolve, but, the good thing is, the users of the system have no idea this is going on. The only reason they don't is because the system uses an event driven architecture. Yeah ok, they would have had the same result if the features in question were async, but we get bailed out by this a lot because most of what we do is async - event driven.

Typically, in this system, recoverable errors are rolled back to the ESB/messaging layer where they are paused and then resent. Non-recoverable errors are simply routed to an appropriate error queue. We have an application that manages the error queues. Pretty standard stuff - browse all queues, browse specific queue, browse message, delete, re-send, edit-resend. This type of error handling gets it done quite nicely for 99.99% of our errors. But today the critical defect was causing threads to hang and events to hang up in 2 of the services. Hung threads don't rollback and don't route the poisoned message to an error queue.

So .. what to do besides panic?

Luckily, as I stated above, events and more specifically guaranteed delivery & durable subscriptions came to the rescue anyway. From examining the thread dumps of one of the services, it looked like the service just woke up on the wrong side of the bed. So we restarted it. Whatever pissed it off before had passed ... it happily drained its durable subscription - as its threads had been hung, it had not ackd any of its events - they all were automatically resent to the service. Now what if this wasn't event driven? What if it used web services (i.e., JAX-RPC or JAX-WS) ... you would be hosed ... yes that is exactly right. Or, your poor developers would have to account for this stuff in each of your service impls. Good luck getting that right.

<sarcasm text="Remember, web services make interop / integration easier. You don't need all the complexity of any integration products ... just roll it all your self. Interop is easy. You just need web services. The tool support is great."/>

Sadly, we did not get so lucky on our second problem. This one is going to leave a mark for some people for most of the week I imagine. Looks like we have to drain a durable subscription and purge out some poisoned messages and then redrive the good ones. And likely some other misery ... but without an event driven architecture and more specifically the goodness that guaranteed delivery and durable subscriptions provide, we'd be totally f'd.

SOAP is the backbone - REST/POX is the eject button

Read a good post by Clemens Vasters REST/POX with WCF: Version 2, Part 1: Foreword via Stefan Tilkov.

Here are my thoughts on it.

Nice write up. I really actually miss the old interop problems pre WS-*. I used to curse DCOM/Java interop (it really was horrible), but it is no worse then what we have today with WS-* IMHO. I understand how we got here … I think it is understandable. It is just really unfortunate. Perhaps if the XML Schema writers had focused on the 80/20 rule, & the WSDL folks toned it down a bit (oh how I miss the simplicity of CORBA IDL), we wouldn’t be in the complexity hell that we are in today, but maybe interop is just this hard regardless. I for one have to say that I’m pretty disappointed where things are today, and don’t see a bright future for the course we are on. I just don’t buy the part you wrote about “…SOAP for building the backbone of it” in terms of interop. Yes in theory it should be, but I think the complexity of it all, the industry politics, and the endless specification churn point a pretty bleak future for SOAP and WS-*. I mean it is April 2006 – shouldn’t things be further along then they are now?

I think that sadly, WS-* is fatally flawed. Perhaps I am just jaded from trying to actually use it for interop or have a mental block when I stare at SOAP messages with oodles of headers. I know that HTTP Headers and SOAP Headers are at the end of the day essentially the same thing, but to me, it is just fundamentally wrong to have the headers in the XML document. To me, the XML document is the application developer’s domain – sticking that stuff there just invites panic.

Also, perhaps I am not informed, but are there any success stories out there on interop and WS-ReliableMessaging, AtomicTransactions, etc.? I’ve found it hard enough to make complex documents interoperate between .NET and Java using vanilla WSDL.

The stuff you guys are doing to make Indigo POX friendly is laudable. At least developers will have an eject button once they give up on building their “backbone”.