For starters, specify a requirement or two on what acceptable performance is. Then, start measuring early. Better to know that you are way off the mark early on IMHO. Fixing services that are way too slow late in development or once they are in production often means rewriting the service from scratch in my experience (of course you still call it refactoring in meetings to make everyone feel better ;) ).
I have seen the most performance problems with Persistence.
Another thing to focus on early is how your services will scale. Our system resembles SEDA (Staged Event-Driven Architecture).
- Thread Safe - in our system that is built on top of JMS at the core, we use a JMS ServerSessionPool impl that makes adding threads (sessions) to a service configurable. This is a great way to scale and throttle a service. Sadly, we deal with some obscure stuff that isn't multithreaded - so we have to configure just one thread. For most services, however, you can have 10+ threads waiting to process events
- Competing Consumers & Shared Subscriptions - Anyone familiar with JMS understands competing consumers with Queues. We use SonicMQ, which, also offers this concept with Topics. We use these heavily. It is great because you can spread processing across machines. You can have 1 logical service running on N machines. When it is flooded with events, the processing is evenly distributed across the logical instances of the service.
Last thing I'll mention is async processing. This is huge in an EDA system - basically everything is async. But some developer's little minds don't like async and want to make it sync. You must find these people early and get them off the project. Just kidding (kinda). We had a couple problems with performance like this. Our login for instance was a dog at one point. This was because it was sync and the granularity of service calls was awful. So we "refactored" (where "refactor" means tore out old code and stomped on it and called it names, then wrote it again from scratch). We rewrote it in a way that was async. When a user logs in, an event is generated that the system responds to. Immediately after the login event is produced the first page of the UI is displayed. Other events in the backend are produced that the UI is interested in. So it listens for these events. The key is with the rewrite, we don't make that process sync. When the events the UI needs are produced, the UI receives them async.