Friday, May 19, 2006

EDA Lessons Learned - Performance

See EDA Lessons Learned for the list

For starters, specify a requirement or two on what acceptable performance is. Then, start measuring early. Better to know that you are way off the mark early on IMHO. Fixing services that are way too slow late in development or once they are in production often means rewriting the service from scratch in my experience (of course you still call it refactoring in meetings to make everyone feel better ;) ).

I have seen the most performance problems with Persistence.

Another thing to focus on early is how your services will scale. Our system resembles SEDA (Staged Event-Driven Architecture).

  • Thread Safe - in our system that is built on top of JMS at the core, we use a JMS ServerSessionPool impl that makes adding threads (sessions) to a service configurable. This is a great way to scale and throttle a service. Sadly, we deal with some obscure stuff that isn't multithreaded - so we have to configure just one thread. For most services, however, you can have 10+ threads waiting to process events
  • Competing Consumers & Shared Subscriptions - Anyone familiar with JMS understands competing consumers with Queues. We use SonicMQ, which, also offers this concept with Topics. We use these heavily. It is great because you can spread processing across machines. You can have 1 logical service running on N machines. When it is flooded with events, the processing is evenly distributed across the logical instances of the service.
Another area to focus on from the start is service granularity (the black art of SOA/EDA) - this is a lot harder to get right then a lot of people think. Especially when you are dealing with many services. Its like anything of sufficient complexity - there are 3-5 ways to do it right and 50 ways to do it wrong - pick one of the right ways.

Last thing I'll mention is async processing. This is huge in an EDA system - basically everything is async. But some developer's little minds don't like async and want to make it sync. You must find these people early and get them off the project. Just kidding (kinda). We had a couple problems with performance like this. Our login for instance was a dog at one point. This was because it was sync and the granularity of service calls was awful. So we "refactored" (where "refactor" means tore out old code and stomped on it and called it names, then wrote it again from scratch). We rewrote it in a way that was async. When a user logs in, an event is generated that the system responds to. Immediately after the login event is produced the first page of the UI is displayed. Other events in the backend are produced that the UI is interested in. So it listens for these events. The key is with the rewrite, we don't make that process sync. When the events the UI needs are produced, the UI receives them async.

2 comments:

Sarge said...

With respect to persistence, EJBs are the Root of All Evil. I've not used Hibernate personally (and Amy just mentioned iBATIS) but I'd use something like, even if I had to hand-roll it again.

With respect to UIs, JSPs are the...nah, just kidding. Seriously, though, I'd use AJAX, as in http://code.google.com/webtoolkit/ . Asynchronous web UIs make synchronous web UIs look like Microsoft apps.

fuzzy said...

Yes I hate EJB. I would have been done with it years ago if you had backed me up 3 years ago ;) But NOOO! I had to give CMP a try.

iBATIS looks pretty good. It is dumb and forces you to write real SQL - the SQL you write gets fired against a db.

I tend to think O/R tools are all evil. I just use the Spring JDBC template - all it does is clean up after you so you don't have to write try/catch/finally everywhere. I don't like O/R tools because they are complex and average developers often think they make SQL (so hard!) easy. But in reality they slowly back you into a corner and own you.

Several of our apps use Hibernate. Fine in experienced hands, but it, like any O/R tool will take your thumbs if you aren't careful. And you want your thumbs Sarge - you really do.