OSS = Synchronized Self Interest

Great quote from Sun's Simmon Phipps:

Instead, the future is in co-operation and in organizations preserving what is ultimately of value to them, he said.

This is not volunteerism, It is directed self-interest, synchronized self-interest, and there is nothing wrong with self-interest.

My new friend, Jason Gilmore sent me a link to the story. This quote inparticular really struck a nerve with a lot of people I shared it with.

Here is the link to Simmon's presentation at OSBC.

I liked this quote as well: Three Golden Rules:

• Collaborate over what does not differentiate
• Compete by innovating on the commodity base
• Contribute!

OSS is pretty mainstream now, but there is a huge third wave coming. It is going to be tough in the future to be a commercial software company I think.

It isn't just about cost savings. It isn't freeware. It is about innovation, agility, flexibility, and attracting talent.

Complexity and Stockholm Syndrome

What is it about us human beings that we have this bizarre tendency to exhibit behavior consistent with Stockholm Syndrome when we work with technologies that are more complex then they need to be?

I have been guilty of this several times and will no doubt be guilty of it again. For all I know I'm guilty of it right now.

I saw this a lot with EJB. I was certainly guilty of it. Having achieved some success and decent understanding of EJB, I became an ardent defender of it even though it was terrible technology.

I see this same phenomena today with WS-*. I like to think that I learned to spot this trend with EJB and am wiser now, but for all I know I have Stockholm Syndrome with my messaging bias. I don't really think I do because I keep challenging myself to look past messaging in its current proprietary form.

I now think that I also had Stockholm Syndrome when I was an advocate of Rational ClearCase (sorry Sarge). Everyone new to ClearCase runs kicking and screaming from it because it is so complex. Once you suffer through the arduous learning curve, however, it is common to become a ClearCase zealot. I know, I was one for years! I'm dealing with this currently at work. Note to coworkers - I mean to offense by this and this is just my opinion. I think you are used to me by now.

Maybe I am just getting old and dumb and can only handle simple things.

Insane Week + New Role + Jobs

I had a pretty crazy week. Last Sunday I flew with a crew from work to Indianapolis, IN for some post reorg meetings. The CIO announced his direct reports and the gist of the reorg the week before, but there were a lot of loose ends and various interpretations of the reorg to be worked out.

Talk about long and stressful days - we would get to the office at 7:30 AM and leave at 7:00 PM for dinner where work was discussed further.

The new org. is highly matrixed. This is how we were organized in Portland so I am used to that. Time will tell how it works on a much larger distributed scale.

I did get to go water skiing with some of my work friends one day - that was cool. Although it highlighted the point that I am indeed completely washed up (as if that was still a question) as I couldn't get up on one. I tried 4 times before I through in the towel and went with two.

The new CIO is pretty innovative. He has embraced distributed development and Open Source Software. My new job is leading the charge on OSS within our division. This marks my departure from a "Technologist" into management. This should be very interesting work. I will be working like a dog, however (not really a new thing sadly). I will be hiring a small senior team to get this organization off the ground. I have to put the job reqs together this week and will post links once they are available. But if you have a *passion* for OSS and live in Portland, Indianapolis, IN, or Wausau, WI and want to drive OSS adoption/contribution within a large company, let me know. For this initial round of hiring I am looking for stone killers with demonstrable OSS experience. As usual, communication skills are fairly crucial. If you can't collaborate and communicate effectively, your technical skills are useless to me.

Also, since we are now a distributed organization, there are some more jobs open in Portland, OR. Basically, going forward, most jobs that are open in Indianapolis will be open for staffing in Portland, OR. The opposite will be true for projects based in Portland, OR for Indianapolis and other cities. There are pros and cons to distributed development. It is the way the world is going, however. Our CIO is putting the right infrastructure in place to make this model work. The transition will no doubt take some time, but I am optimistic.

This list will be growing the next few months, but here is a list of the current open jobs (search for Information Technology).

Advanced Message Queue Protocol

My friend Erik pointed me at Advanced Message Queue Protocol a year ago, but I assumed it died.

Well, he pointed me at it again and it looks like it may go somewhere.

This would be fantastic. While I love messaging, bridging messaging systems really limits their reach.

I wonder how they are going to get everyone to play along (IBM, Tibco, Sonic, etc.). Hopefully, RedHat's effort will help: RedHat is currently working on an implementation that will be built into the operating system, making AMQ as free and available as SendMail, and accessible from any technology API such as JMS.

If this happens, it would be a major step forward in my opinion and a major enabler for large scale event driven architecture.

EDA Lessons Learned - Transactions

See EDA Lessons Learned for the list

I'm not doing much (ok any) 2PC (two-phased commit) these days. I used to be pretty hip on XA in the Java world. I still run into people that yell at me about XA. Perhaps I wear size 16 burgundies (clown reference for the un-indoctrinated).

XA is great in theory and fun to drop in meetings (not quite as fun as 2PC), but:

• Slow (sync to disc)
• Expensive (resources)
• Complex
• Implementations often buggy
• Not always easy to find XA impls (try VSE mainframe)

It isn't perfect, but pseudo 2PC works pretty well:

• JMS transaction + JDBC
• Compensating transaction if necessary

Here is a good write up on the topic.

Distributed Development Tools

I am spending a lot of my time working on tool sets lately. I have lots of biased opinions.

My biggest core requirement is distributed development with internal developers and third parties. I want it to be easy. It shouldn't take more then 1 hour for a developer in my office to get at the source code and having it building for a project managed out of another office.

I don't want to bias this (yet) with what I am looking at.

I'd love to hear what others are doing with distributed development and tools.

I am looking for the following tools (mainly):

• Source Control
• Defect/Issue/Enhancement tracking
• Project management
• Build/Deployment tool (ala CruiseConrol)
• Requirements Analysis Tool

Reorg

Huge reorg. went down at my company yesterday.

Things went well for me and the Portland office.

Reorgs cause a ton of stress. Even though it appears that I have an extremely bright future, I am totally fried. The new CIO basically took the place, shook it 11 times, and poured it out.

In all honesty, the new structure looks great on paper, but it is going to be a tough transition.

Anyway, watch this space - we will hopefully be doing a lot of hiring - I'll post opportunities here.

We will continue to work on fairly massive SOA and EDA projects and accelerating OSS adoption, contribution, and other stuff I can't talk about yet.

EDA Lessons Learned - Shared Code

See EDA Lessons Learned for the list

Sharing code between services requires careful thought and planning to avoid too much coupling between service impls. I have seen both extremes - no sharing and uber sharing (i.e., too much).

So my advice is do share code just don’t overdo it.

Some modules that I found useful (in hindsight) to share by user interfaces and various services are:

• schema (code that validates XML against schema)
• persistence (consistent way of accessing database)
• msgmapper (mapping XML to POJOs that you pass into business logic)
• domainmodel (the most important package/module that most apps don't have)

The obvious issue here is that if you don't share code, you are likely to get the same bug in all your services. To fix it, you have to fix it in each service. But when you do share code between services, then you have dependencies. You can't win. Just quit now ;)

EDA Lessons Learned - Batch vs EDA

See EDA Lessons Learned for the list

I generally shy away from throwing down phrases like impedance mismatch as they tend to get abused, but no joke - batch and EDA is one.

We deal with this with our large systems (big iron), which are batch oriented.

Bridging the batch world with the EDA world requires careful thought and design. We found it effective to have very clear control boundaries between the two.

We have batch jobs that run in the middle of the night that generate many events. The events are not generated at the same time every night - it depends on how long the batch cycle takes. So basically the EDA side gets huge bursts of events in the middle of the night.

We see the most benefit from Staged Event Driven Architecture when batch jobs produce large bursts of events.

We also have services that want to write to the mainframe when it is in a batch run. The mainframe is unable to accept the writes when it is in this state. The "Rollback and Pause" concept is very helpful in cases like these. Basically, the service (filter in our case) just holds the event by rolling it back to the messaging layer and letting it retry at an interval. Having this logic in a filter rather then the service responsible for writing to the mainframe is good because you don't muddle the service impl. with environmental issues.

Operationally, the batch/EDA impedance mismatch is definitely our biggest challenge. It generally works, but we have our days. Async send from the batch system, clear control boundaries, "rollback and pause", and ErrorQAdmin, make it tolerable.

Reading Books

I'm reading a couple books.

On the advice of Sacrificial Rabbit, I picked up a book named, Getting Things Done. Now I generally do "get a lot of things done" - all by myself. But my organization systems have crashed the last few months as I am working on more and more different things all at once. I'm looking for something new and people seem to be into this so what the heck! I'll let you know how it goes.

I also picked up Charlie Wilson's War ... "The extraordinary story of how the wildest man in congress and a rogue CIA agent changed the history of our times". I'm with Don Imus ... it is "The most unbelievable book". And its true! I picked it up at O'Hare for $5. Best $5 I have spent since $.25 draft night that one time in college where things got a bit out of control.

EDA Lessons Learned - XML Aware Analysts

See EDA Lessons Learned for the list

We backed into this by accident, but our business analysts (report into IT) became very "XML Aware" (i.e., understand XML). This is a huge asset IMHO.

In a perfect world, perhaps this wouldn't be necessary, but in the real world, where projects get delayed this is very useful.

In the height of our panic (development) of the first phase of this system, we had services in varying degrees of completeness with scaffolding all over the place. The scaffolding allowed for some semblance of "end to end" testing.

Certain services were considered "code complete" by development. XML Aware testers allowed us to do black box testing against these services. Analyst started writing test cases with expected XML inputs and outputs. This was translated to XPath assertions, which, in some cases were automated (extension to XMLUnit/JUnit). The analysts invariably found defects in the code complete services. If we had waited to do traditional end to end testing, the project would have been even later.

"XML Aware" analysts had the side effect of making the analysis and design better in a major subsequent phase of the project as designers and analysts were competent in EDA style architecture and XML.

If you are lucky enough to have analysts on your team, I recommend getting them down into the weeds a bit and having them learn XML & the concepts of the architecture style you are using. If nothing else, they will ask one "dumb question" that will make you think. If you are lucky, they will start calling b.s. on you at appropriate times ;)

SOA and EDA make sense to everyone in IT from a high level. Spreading the knowledge of how they are implemented in your organization outside of the development team will result in a higher quality impl.

Want a job in Portland, OR - Java SOA/EDA?

Want a job in Portland, OR - quite possibly the best city on the planet?

We'll even move you here if you can slang wicked code. Sorry, you must have a green card or be a US citizen.

Insurance systems are a lot more fun then you might think. We pay above market, have old school benefits and OSCON is across the street every year.

We are looking for 2 full time senior Java developers. We are starting a series of new projects where we will be doing serious bi-directional integration with an IBM i5/iSeries.

Click here to apply. Job code is 23568BR.

EDA Lessons Learned - Build and SCM

See EDA Lessons Learned for the list

Everyone's favorite topic - build and SCM ;)

SCM is a hot topic within my company right now. For whatever reason, people get very excited/emotional about this topic (myself included). After living with various SCM tools, I prefer simple ones that aren't a barrier to distributed development. I actually used to like ClearCase - a lot. But now I think that it is overly complex and is a barrier to distributed development. ClearCase absolutely must be installed on a LAN for it to work - this forces a replication model for distributed development. This results in too much complexity for my taste and makes it very difficult to truly collaborate at the code level. There are OSS and commercial SCM products that were designed from the ground up to facilitate distributed development. Pick one of them if your developers are all over the world.

These lessons learned are from a large EDA style application, but like many of my lessons learned, it is applicable to a SOA project or any large integration project. I do not (yet) have an approach for company-wide SCM for SOA/EDA. That is certainly more complicated.

I knew a lot of this stuff from previous jobs, but walked into this project as it was in full development. It took *significant* effort to reorganize and fix. Like many things, planning from the beginning would have saved a ton of effort. Here is a list:

1. Plan the SCM repository carefully

   Don't have too many roots - don't have too few roots. Under the root, service, shared, web directories with modules beneath works well for us.

2. Branching model

   Choose carefully - here is a good article on branching models. I prefer "branch by purpose" - it keeps it as "simple as possible, but not simpler" as my boy A. Einstein would say.

3. Keep build simple
   • Developer should be able to run entire build locally easily
   • Developer should be able to build just required modules (i.e., shouldn’t have to run entire thing if just working on 1 service)
   • Developer should be able to make changes to the build (avoid single threading it through the build guy)
4. Deploy beefy build box

   Buy a nice high end workstation with plenty of memory and 2 very fast processors for a build box. You don't want the build to run on the build guy's local machine. It is much easier to automate from a centralized official build box IMHO. Anyone with access should be able SSH to the build box and launch a build. At a previous job, we even had a web page that allowed developers to launch builds (way to go Sarge)

5. Write and maintain simple instructions to get new developer’s environment setup
   Shouldn’t take more then an hour
6. Configuration

   Either keep config files for each environment (e.g., dev, test, prod) under source control and have deploy script use appropriate configs or script the setup of each environment. You don't want to do this by hand. This can be difficult with vendor products that want to invade your entire SDLC. Ensure that your vendors have deployment friendly support

7. Avoid Maven

   I am not up to speed on the latest version of Maven. I understand it is supposed to be better so take this with a grain of salt. At one point we had competing builds - Ant and Maven. Ant won because it is simpler and more people understand it well. We do use the Maven directory structure, however. I'll leave it at that.

8. Don't mandate a particular IDE

   Quickest way to piss off developers is to mandate an IDE. The build should never mandate an IDE.

9. Beware of tools that generate code

   If you must use these tools, at least either check in the results of the generated code (can be very dangerous if the code isn't just POJO style) or script the usage of the tool in the build (and watch your developers start to hate the build). And people wonder why I hate generated code ;).