Atom & Pub/Sub Revisited

Tim Bray picked up on my Pub/Sub vs. Atom & AtomPub? post.

During the last year, I have not been doing that much messaging, but the previous 7 years I did a substantial amount of it - both point-2-point and lots of pub/sub. I absolutely don't think that Atom / AtomPub by itself will replace messaging. I don't know enough about XMPP to know if it will replace traditional messaging some day. I am a big fan of AMQP, but don't know if it is viable yet or will be viable soon. Tim is dead on that PSB is an impressive corpus of work

My point, was only that Atom & AtomPub appears to be a very good format / protocol to do business events with a polling model. Having done a lot of pub/sub, I have done a lot of clustering. The buffer that Tim describes is not so simple when you get lots of clients and lots of consumers. That is where flow control starts to appear (if you really want to guarantee message delivery) and where the dark corners of fail over and clustering rear their heads. It is nothing against messaging per say - just how it is. My point really was that even "guaranteed" messaging is often not truly guaranteed. And that for the right usage scenario the poll model is simpler and more appropriate.

Furthermore - push models can be appealing to developers because it feels cool (has felt very cool to me) - even simple. My point is just that this isn't typically really the case.

It really just comes down to your requirements. For example message rate, how quickly your event sinks need to process, etc. For where I am thinking right now, high level business events don't typically need to be delivered more than every 10 minutes - perhaps ever minute. I'd much rather deploy some simple HA web infrastructure & Atom/AtomPub than AMQP, TIBCO, SonicMQ, ActiveMQ, etc. to meet those requirements.

In my experience with pub/sub style integration architectures when you are pushing information amongst major domains (e.g., party/customer information), there is always the nagging question if you missed an event - or how do I deal with drift. This is because even though it is supposedly guaranteed messaging, there is always a risk. I've seen people try to capture events and persist them so that they can be replayed in the event of this type of a problem. Or more commonly, check the master system every month to ensure that the systems are synchronized. An Atom feed seems like a good way to avoid this failure all together - in that rather than pushing a copy of the event to each interested sink, the sink just reads the feed. Sure the client has to be a little bit more intelligent, but it just seems more deterministic to me. But I have more to learn.

Atom Service Directory

I re-read some of the Atom specifications on a plane today.

I haven't been as excited about a data format since creating a mostly proprietary, but somewhat based on ACORD one at a Canadian insurance company in 1999-2000.

I still have a lot to learn about ATOM, but it seems to address all of the pain points I have seen in integration data formats so I'm optimistic.

I obviously can't be the first one to think of it, but wouldn't Atom / AtomPub make a nice simple Service Directory (think UDDI, but with legs)? Just a feed with entries for each service instance? Categrories and other meta data? You could dynamically register services via AtomPub - perhaps services would have to re-register every so often. Is there something like this already?

I also looked back at some of the blogosphere spaz-outs regarding Atom yesterday. I have more to learn, but perhaps it is best that Atom get's a head of steam before embrace and extend sets in anyway.

Here's to hoping that Atom stays simple - seems to be a beautiful format.

XML Schema Languages

Ok, so Atom for the document format, what about the XML Schema Language?

According to Tim Bray (and others), it is no contest - RELAX NG wins.

The last two canonical models I was involved in used XSD. I have a hate hate relationship, with XSD, but I'm not yet convinced that RELAX NG is the path forward for me right now. I need to get my learn on. The biggest issue, really, is that there is a lot of XSD out there - a lot of schemas that I may want to fork into my canonical format are already in XSD.

In my experience, the basic 25% of XSD is ok - it is the other 75% that has a lot of very dark corners. That and the never ending evil of code generation from schema. Here are some other benefits of RNG over XSD according to Wikipedia.

I also have to figure out where Schematron fits (or doesn't fit).

What tools are people using with RELAX NG and Schematron that they are happy with? I see Topologi. Last time I was involved in this we used XML Spy. It is a pitty that it does not seem to support RNG?

What about runtime support - what works well for Java, .NET, Ruby, and z/OS (the last one should be fun).

Any experience in the trenches?

Canonical Model Format & Atom

It has been awhile since I thought a lot about canonical models. I have been involved in a few over the years.

I'm thinking about them tonight.

In creating one today, is there a reason why you wouldn't use Atom instead of say SOAP or more likely your own proprietary solution? I re-read Pete Lacey's classic The S stands for Simple to remind myself why I left SOAP behind.

SOAP isn't even on my radar these days, but perhaps someone can convince me otherwise on why it belongs in the heart of your integration architecture? Of course you have to interoperate with it, but you can certainly keep it at the edges (i.e., integrate with legacy web services, packaged apps that only expose a SOAP interface that you want to use).

We are talking about your lingua franca here - there are few things in my opinion, more important to an integration architecture than a canonical model - you want to get this right.

SOAP started simple enough - it just became an industry and now it is fairly hard to separate the wheat from the chaff. Here's to hoping that Atom stays simple.

I kinda doubt that many people are using SOAP as a part of their canonical model. I would have to bet that they are using something they made up. From what I can tell, Atom seems to give you what you need out of the box - most people will need to add some more standard elements to it, but it has all of the basics so why reinvent the wheel?

Jive Funding

Wow, Jive took some VC funding.

A couple of these guys are my friends - they work their butts off. Great guys, great company.

Congratulations Jive!

Amnesia & Terminal Gravity

These brewers are not easy to find. But they are incredible.

Terminal Gravity (Enterprise, OR).

Amnesia Brewing (Portland, OR)

After a long day yesterday, I had one of each at Henry's Tavern last night :)

Weak Sauce: RESTful Flex Angst

I think this report from Erik Onnen on Flex and REST is pretty weak sauce.

I'm surprised Flex doesn't have a real HTTP client built into it. You can't do REST with just GET and POST. Being dependent on the browser is pretty lame - is this driven by some sort of other requirement or some security sand box what-nots?

I guess that is what we get for hurting the web.

How are others dealing with this?

Pub/Sub vs. Atom & AtomPub?

I came across an interesting article by Sean McGrath entitled I'll push and you pull. The mashup approach to application integration.

I posted about Push vs. Pull in the spring. I'm a sucker for Lean thinking so the title of this article got my attention.

Sean says:

Now here is the kicker. The web - with all its concomitant bits'n'bobs from XML to RSS/Atom to AJAX - is an extremely good platform for pull-centric design. On the Web, if you try to pull some piece of information and something goes wrong, well you just pull again and again until you get it or give up. Nothing fancy. Just brutish repetition. Something machines are extremely good at. If you want to look at information from yesterday, you just go to the URL that contains yesterday's information. Nothing fancy. Just a simple naming convention that includes dates in URLs.

I have done a fair amount with pub/sub over the years (complete with clustering etc., fail over strategies, etc.). I like it, but it is not without it's challenges. If only there was real guaranteed delivery etc. If only there wasn't "flow control". If only fail over always worked. If only messages didn't get trapped, etc. All I'm saying is having implemented several large implementations of pub/sub (EDA whatever you like), I know that it isn't easy - that nothing is really "guaranteed".

Clearly Atom can't replace ALL pub/sub use cases, but for every day integration architecture where you want business events / EDA why can't we use Atom feeds? In an extreme case, you might have an event sink requesting the feed every 10 seconds - in most cases every 10 minutes would likely be fine?

Who is doing this today? Any lessons from the trenches?

Update I meant to read the article earlier linked off of this post by James Snell. Just did. Dang. It is a sample of using Apache Abdera.

Update 20-AUG Bill de hÓra has a pretty $$ response - thanks Bill.

Update 21-AUG Dan Creswell has some thoughtful comments. When I blog I often do the x vs. y thing - but clearly the truth is in the middle. I used to be a push bigot and have just learn the hard way how difficult it is to achieve. Clearly, not everything can be pull - of course it is a mix. In case anyone is curios, I'm thinking of inter-domain integration (e.g., getting customer additions/updates/deletions to many interested systems) with this line of thinking rather than some sort of intra-domain integration (e.g., trading system where there is massive high performance pub/sub). You have to choose the right tool for the job. More and more for me, simplicity is winning out. This is just an evolution of my thinking - a year ago I was still a MOM bigot.

Bell's Oberon

Speaking of beer, my family had a fair amount of Bell's beer in Glen Arbor.

Oberon is delightful.

Bell's is far and away my favorite non-west-coast-microbrew.

Deschutes Brewery Down the Street

My favorite beer of all time is Mirror Pond.

It is certainly the beer I have consumed the most since I got out of college. Sadly, the beer I probably have consumed the most of is The Beast.

I am pretty happy that there will be a Deschutes Brewery down the street from me this coming Spring.

Note: the picture above is their new label - finally they are getting rid of the crappy current one. It isn't that great either. Oh well you don't drink the label very often. I still think the first 2 generations were best: (source)

Second Generation

Original

New blog - Erik Onnen

Hey cool, one of the smartest people I know, Erik Onnen, now has a blog.

Yipee, more of the world can start to learn from once of my favorite collaborators.

Glen Arbor

I just got back from a Herrick family vacation in Glen Arbor, MI.

It was a lot of fun. I really needed a break - I have been very busy.

I used to go there every summer as a kid with my family. It was nice to get back there - I hadn't been there in about 20 years.

I can be a bit of a bigot about Oregon and how it is superior to just about anywhere. But it is tough to beat the Great Lakes in the summer. My mother will love to hear that (no mom this does not mean I am moving back to Michigan any time soon - sorry :) ).

Here is a picture of Pyramid Point that I took.

Don't Move to Portland

It Rains All the Time - you will hate it.

I'm rUl busy, blog is not getting a lot of action. Hopefully it will pick up soon.