Tuesday, April 25, 2006

EDA Lessons Learned - Service Contracts & Event Stream Design

See EDA Lessons Learned for the list

The contracts between services are the most important thing there is in either SOA or EDA IMHO. You get this right and your future is bright, get it wrong and it is dim. Pretty much that simple.

I came to SOA/EDA from CORBA, J2EE, and DCOM projects. I thought that IDL was the only way to fly. When WSDL came out, I treated it like any other IDL. I was one of the people who endured significant pain with WSDL when it first came out because I was such a believer in "contract first" design/development.

The first time I started muttering "what is the contract?" when discussing OO design, service design, anything software related really was when I was mimicking one of my mentors (Dave Read when we worked at eXcelon). Just because I knew he was wicked smart and he said that a lot, I made sure I knew what he was talking about and added it to my repertoire.

So I am still a huge fan of contract first development, but it isn't always practical - especially with EDA. For one thing if you wanted to use WSDL today with EDA, you really couldn't as WS-* doesn't support pub/sub yet and WS-Notification & WS-Eventing are being merged into WS-EventNotification. I tend to think they will be too rigid when they are updated, but we'll just have to wait and see. Anyway, just because you don't have an IDL, doesn't mean you shouldn't think about contracts.

Service contracts in EDA aren't usually request/reply like they typically are in SOA. It is easier to get your hands around contracts with request/reply. With event streams, you have N services listening to an event and possibly publishing N events to N destinations. A service may listen to many destinations or a pattern. A service may receive XML documents conforming to different schemas. So you pretty much have to let the whole IDL thing go ... unless you just allow "any" or whatever it is in XSD (which is pointless).

This isn't very concise, but for me, contracts in EDA are:

  • Event input payload(s)
  • Service input destinations (set/patterns)
  • Event output payload(s)
  • Service output destinations (set)

Here are some more details on my opinions on service contracts and event stream design and EDA:

  • Devote appropriate time to service contract/event stream design
    1. Have several sets of experienced eyes review
    2. Review a number of times - allow for "soak time" as my friend Sarge would say
  • The goal is low coupling and high cohesion
  • Anticipate event stream flow changes
  • Consider persistence when thinking about contracts
  • Get the granularity right (narrow vs. coarse) - the truth is somewhere in the middle
  • UML sequence diagrams with embellishments (persistence, state change, etc.) and annotations works very well for technical design and review
  • Great website/book - Enterprise Integration Patterns


jack.vanhoof.ns@gmail.com said...

Perhaps these guidelines may help:

Identify business events

Define canonical message formats for the identified events (a existing canonical data model can be a great help)

Define services that publish the identified events

Define services that subscribe to the identified events

Define transformation- and enrichment rules to transform the messages from the local semantics of the publishing services to the canonical semantics

Define transformation- and enrichment rules to transform the messages from the canonical semantics to the local semantics of the subscriber

Map the services to the applications that process the events (publish as wel as consume)

And furthermore:

A service is always one-way, either publish or subscribe

A service has only one function (publish or subscribe to one event-type)

A service is nothing more than an interface to or from application logic

An aplication may have more than one service mapped to it (publish as well as subscribe)

A service is only mapped to one application

An event-type can be published by more then one service (and so more than one application)

Comments and additions are more then welcome...

fuzzy said...

Jack, good stuff.

I disagree with "a service is always one-way, either publish or subscribe". For me at least, most services do both (typically receive 1 event and publish a new event). Also, my services do often just receive 1 event type, but it is quite common for them to listen to many different event types (e.g., "Stream Based Processing" and CEP).

Thanks again for commenting.

jack.vanhoof.ns@gmail.com said...

Hi Mike,

Here are my arguments on the principles.

When one service receives an event and publishes another one, it sounds to me like some mix of request/reply and pub/sub. In my opinion the design is more elegant and flexible when a service only performs one single function. The application behind the service may generate a new event and expose this event via another service. Otherwise you are applying two deferent roles to one service.

The same reasoning applies to CEP. The design is more elegant and flexible when in the front-end a service is listening to only one event-type. Multiple event-types are processed in an event-processor (application) where the multiple services are mapped to.

I don't know how exactly commercial CEP-engines work. It is all very new. A Sonic representative told me that their new CEP-software is all propriatery stuff at the moment. So I suppose this is not based on "services" but rather on "streams" that are captured with high-performance proprietary software logic. That's quite a different story.

fuzzy said...

Hi Jack,

Yes we do a lot of mixing of pub/sub and request/reply. These are pub/sub workflows or event workflows. It may seem odd at first, but it is very effective - fantastic for location transparency. But we have other event streams that just cascade – many events may be generated and no client is listening. And yes, services do typically have one event input and one output. That event output just may come from a set of destinations (or a pattern) and may be published to a set of destinations. We do, however, have services that necessarily need to listen to many different event types. This is appropriate IMHO and one of the major benefits of using an EDA architecture. By listening to an event stream with multiple event types you can do very clever things.

This post covers the types of integration we deal with cover:

1. Sync Request/Reply
ServiceA.Thread1 sends request and waits for response from ServiceB.ThreadX.
2. Async Request/Reply
ServiceX.Thread1 makes request. ServiceX.ThreadX receives response.
3. Async 1:1 (fire/forget)
ServiceA.Thread1 publishes message and does not wait. ServiceB.Thread1 receives message. ServiceA.Thread1 may or may not be aware of ServiceB.Thread1 (depends on protocol used).
4. Async 1:M (pub/sub)
ServiceA.Thread1 publishes message. ServiceB:ServiceN receive message. ServiceA.Thread 1 unaware of ServiceB:ServiceN.
5. Async 1:M Request/Reply (pub/sub Request/Reply)
ServiceA.Thread1 publishes message. ServiceB:ServiceN receive message and send response. ServiceA.Thread1 receives N responses. ServiceA.Thread1 unaware of ServiceB:ServiceN - just know that N responses were returned.
6. Async M:1 (event stream subscription)
ServiceB:ServiceN publishes message. ServiceA.Thread1:ThreadN receive messages. ServiceB:ServiceN unaware of ServiceA.Thread1:ThreadN
7. Batch Feed
ProcessA sends file to ProcessB

jack.vanhoof.ns@gmail.com said...

Thanks Mike,

I got it. Your post made me clear that we use different definitions of a service. Where I mention a service I mean what you call a service.thread. Your definition of a service is what I call application logic. In terms of technical implementation I refer to a web service, which in fact is the interface to the application logic which in a layered design may consist of a composition of other web services (possibly a BPEL process).