SOA Governance and Vendor Consolidation

About a year ago, it seemed that getting the old brain around SOA governance was a lot easier. There were plenty of vendors in the space, not so much hype and vendors could truly distinguish themselves through creative use of buzzwords. After so many mergers and even one nested acquisition(!), the proliferation of buzzwords around SOA governance has ceased and it's now become very difficult to evaluate products. I don't know how the big analyst shops manage to stay up to date. I just hope I can make it through an initial evaluation before there is one buzzword to rule them all, effectively making it meaningless ... at which point I hope to have something in place before those who manage-by-magazine read the buzzword and want the latest, shiny bauble.

I wonder if the job title "Marketing Engineer" actually exists? I bet each vendors' ministry of truth has a few on the payroll.

Widening Gap Between Business and IT

Alastair comments on an earlier post where I wrote about software skills and development skills. This prompts me to take a closer look at what I wrote based on what I see happening in many companies. I certainly read a lot about the necessity to align IT with the business but, I really don't see that happening. I actually think there is an increasing amount of mis-alignment. One major symptom which I believe proves this is outsourcing.

Setting aside all of the buzzwords about efficiency, core competencies, etc., outsourcing is about saving money and reducing payroll size. Everything else is pretty much bullshit. The root cause of outsourcing being so attractive up front really comes down to 2 issues:

a) Low quality requirements;
b) A mutual lack of respect and understanding between business and tech.

In a typical business/IT arrangement, the first thing that happens on any project is the kickoff meeting followed by 1 person from IT meeting with about 1/10 of the people in the kickoff meeting for several weeks-months, gathering requirements. Based on past experience with waterfall-like development and the length of time required, the business thinks, "If I don't get all of my feature requests in that requirements doc, I'll never get it done." At the end of the process is a comprehensive requirements document, forming the basis of a contract between business and IT. Then, the spec gets handed off to development (an intra-company gap) and the business sees nothing until the first demo/beta.

In a typical outsourcing arrangement, the first thing that happens on any project is an army of experts like analysts, designers and architects from the outsourcing company show up, typically outnumbering the client's staff by 3:1. Why does this happen? Simply, this is the outsourcer's insurance policy; clear, detailed requirements form the basis of the contract for delivery. Based on their experience, the outsourcer knows that businesspeople generally can't write requirements and that discovering requirements is a highly labor-intensive business. There's lots of data stating that the largest costs in any development project are gathering requirements and fixing defective software due to defective requirements. Therefore, it's in the outsourcer's best interest to make sure the requirements doc is as comprehensive as possible. Then, the spec gets handed off to development (a wider, inter-company gap) and the business sees nothing until the first demo/beta.

Look at the similarities! The goal is comprehensiveness, not quality or priority. When the goal is quantity, cheap labor will win, every time. Isn't it easier to see why outsourcing looks so much more attractive, up front? The business sees that due to the cost of labor differences, it will take less time/money to develop a comprehensive contract, or, given equal time the outsourcer can develop a far more comprehensive contract.

How can we solve this problem in favor of business needs and in-house IT? That brings me to my second issue, the mutual lack of respect and understanding between business and tech. The formal solution would be to train businesspeople how to discover and write better requirements and also train IT staff how to better elicit requirements. The less formal (and zero-cost) solution would be to co-locate (no gap) the developers with the business people. This would allow the developers to deeply understand the intent behind the requirement and it would allow the business to focus on the most important requirements at that moment. Sounds a lot like the techniques favored by agile software development methodologies.

It would seem that a relatively simple change in the physical business architecture would reduce both software development costs and legal costs (for when the outsourcer doesn't deliver what was intended for them to deliver).

How To Scale RoR

I attended one of Sun's executive briefings yesterday and I saw the ideal way to scale any one of the database-driven web development stacks:

http://www.sun.com/servers/x64/x4500/specs.xml

Now that solid-state disks are becoming more reasonably priced, replacing a bank of spindles with solid-state drives acting as an advanced disk cache, that should solve any database latency problems. As long as shared-nothing is practiced in the application layer, this could solve the scaling issue.

2 Views of Agile Enterprise Architecture

Most of the EA implementations I have seen (and worked on) involve a tremendous amount of navel-gazing (aka Current State Analysis) followed by a lot of BDUF and BMUF. The resulting documents are usually out of date by the time the initial draft has been blessed by all the reviewers and any of the technical merit in the document reads more like it was generated by the marketing departments of the selected vendors than by the EA team.

When I took the time to read the comment and change request logs for the document, it was pretty easy to see what parts of the document were actually useful to people (lots of comments, complaints) vs. those that simply could have been omitted, entirely. In one case, a 1200 page mini-series could have been condensed into a podcast. Perhaps at it's simplest, this might be a way of practicing Agile Enterprise Architecture, at least in a manner comprehensible by the framework-followers. The most useful sections of the document were:

1. Diagrams explaining the business processes and which systems supported them
   
2. Diagrams showing the logical systems, their inter-relationships and data flow
3. Catalog of physical and software assets
4. Catalog of capabilities and services (for both staff and software)
5. A long-term technology procurement plan and evaluation process
6. Worked examples, not templates

These were the 6 artifacts that the majority of the people in the organization actually found useful, regardless of occupation. What about the future state, you ask? That's the most interesting part. Aside from the execs who liked the large, color plotter posters on the walls and in their offices, only the architects, planners and vendors cared. The rest of the organization only cared about the next phase on the road map. When I walked around the IT organization, the diagram that I saw pinned to most walls was the migration and transition plan from the current year to next year. Think about that for a moment. Furthermore, the meat of that diagram indicated development of new systems that had already been approved plus new paths for systems consolidation identified during the EA exercise. My take-away from this was that the majority of the really useful information could have been published months before the end-state, big-picture was complete. In that case, we missed the boat on practicing Agile Enterprise Architecture.

Now that I'm a bit older and perhaps wiser, I look back at the work I have done and wonder if it was truly valuable? I have never been a big believer of wasting time doing current-state analysis. In my opinion, all of that information and documentation should already exist throughout the organization. What should really happen in the first phase on an EA exercise is to corral all of the existing operations manuals. I know, I'm dreaming but, I digress. Anyhow to answer my own question, I think the value of the exercise was dubious. All we really did was record the tribal knowledge and set forth some common sense guidelines. That's more like Enterprise Archaeology, and there's nothing agile about that! I suppose there was some short-term value but, certainly little that is lasting.

So, as an enterprise architect, what can I do that is valuable? As I stated before, I don't believe that bringing agility to enterprise architecture is very valuable. Necessary, but not valuable. Perhaps a subtle re-statement is all that is needed? When I actually keep my mouth shut and listen to what the people on the business side of the company are actually saying, I hear a lot of words and phrases like process changes, new regulations, nimble, waivers, exceptions, changing market conditions, adaptable product lines, etc. What they really want, in terms of assets I can create, is an Agile Enterprise Architecture. The value of an agile enterprise in tomorrow's business conditions, is obvious, don't you agree?

How To Evaluate ESBs

I was asked how we evaluated ESBs when we did our tests, last year. We used Brenda Michelson's paper from Seybold as our springboard. It provided both use cases for our POC tests and feature categories for our survey. The download is free (after registration).

Ideas 5 Years Ahead of Their Time

I received an email message the other day asking me if I wasn't being too arrogant by stating my ideas are 5 years ahead of their time. Reviewing the work I have done over the years, 5 years is a good average. Please remember I state this timeline based on the behavior of large organizations. These are places where products and services don't get released, they escape. Here is a typical timeline for an idea:

Year 1

• look ahead to see the list of potential risks and problems the business will run into
• find interesting research or immature but innovative product that will solve some business problems
• squeeze in time during day job to throw together a quick prototype
• analyze prototype findings to death
• if successful, put together a demo and a business case to get funding for a POC
• sell

Year 2

• look ahead and see risks and business problems get bigger
• listen to really smart exec (singular!) who see the same problems and beg them for POC funds
• perform POC on a low-priority track so work starts and stops for months
• perhaps finish POC and analyze findings to death
• revise business case with more accurate financials and develop a plan for a pilot project
• sell

Year 3

• look ahead and see risks and business problems starting to effect business operations
• listen to really smart and smart execs (plural but, usually not several) who see the same problems beginning to effect operations and beg them for funds/staff for a pilot
• perform pilot project on a low-priority track so work starts and stops for months
• pilot gets cancelled because all resources must now be focused on fixing some of the more serious problems the pilot could fix
• listen to execs whine and panic about the risks and problems
• form committee to find solution
• committee recommends a solution amazingly similar to the pilot
• sell

Year 4

• risks and problems have become a crisis
• listen to customers complaining about the risks and business problems
  
• listen to scared execs screaming "we don't have time to do a pilot, we need to fix this now!"
• launch large scale project to fix problems and remove risks
• find consultants
• start fixing problem
• write specs
• fail and don't renew consultant's contracts
• fall on sword

Year 5

• listen to CEO talk about the strategic importance of the pilot/mega-project
• take lessons learned, re-scope and re-prioritize pilot/mega-project
• re-name and re-launch pilot/mega-project
  
• maybe re-evaluate solutions
• put lower priority projects on hold (see Year 3)
• form core team of employees and have a clear vision
• re-start pilot/mega-project
• find consultants
• fix problems and remove risks
• roll-out the solution into production

The cycle time varies but in my experience, I have never seen this happen any faster than 2 years and I have heard of similar projects stretching out to almost 7 years. And those numbers are for the projects that actually make it past Year 4; about 70% of ideas don't. In companies that use a waterfall SDLC, this will also be the timeline for almost every idea, regardless of size.

I'm not claiming any kind of prescience or precognition, only the knowledge that in order to set up the game that identifies and fixes a problem in a year (a reasonable amount of time), the set-up time is close to 4 years! Therefore, the original idea for the implemented solution needs to be about 5 years ahead of its time.

Carr, Citi and IT spending

Nicholas Carr is certainly a great writer but, he's a terrible mathematician. In his latest post, he mentions that Citi is going to cut IT spending by $4.6 billion over the next 3 years or as he states, find "...new opportunities to do more with less."

Nicholas, companies only get to do less with less. Given two activities and money for one, there are only two choices: do both activities half-baked or, fund only one. I actually think spending cuts like this are a good thing because it's basically the last ditch effort to force the business to make up it's mind about what features it can't live without and which ones are gravy. Wouldn't that be nice? We all know that IT spending is, to a large part, based on the business forecasts for growth and demand, and we all know how accurate those are!

On the other hand, we in IT really don't help the business make those hard decisions. We want to be nice and say yes to everything. Worse, we often say yes to the wrong thing. Just this morning I was in a meeting where we were evaluating options based on feature groups, and which features were most important. When I actually shut my mouth and really heard what the customer was saying, both their number one fear and number one need were not addressed by any of the requirements, directly. I brought this up which resulted in stunned silence. We re-arranged the spreadsheet and after a couple of minutes worth of back-of-the-envelope calculations, two of the six options just dropped off the map and a third is in serious jeopardy. Option number one isn't looking too credible, either.

Deliver what the IT customer needs, not what they want; a subtle difference with huge implications.

StoryTestIQ - STIQ

My friend, Chris Sterling over at SolutionsIQ, reminded me of the value of executable requirements when we last spoke about dealing with outsourced development. James McGovern writes far more eloquently about the successes and benefits of outsourcing than I can so, I'll avoid that discussion here. Outsourcing is a reality I have to live with so, I need to develop some mitigation strategies. Having well-defined user acceptance tests in the form of executable requirements is one strategy I'm trying to get embedded at work. Some time ago, he posted a message in the Scrum Development mailing list about STIQ and it's intended use:

I will preface my email with the fact that I am one of the developers for StoryTestIQ (aka STIQ) and that we use it on many projects currently at SolutionsIQ. We created STIQ out of a need to help our Product Owner describe what they were asking for in their feature requests and ultimately user stories. STIQ is a combination of Fit, FitNesse, and Selenium with some special sauce that allows you to do both web UI and beneath the UI acceptance testing.

Here is a scenario of how we use it:

Upon completion of the Sprint Planning Meeting we come out with the following artifacts related to the development of our acceptance tests:

* User Stories the team has committed to

* Acceptance Criteria (aka "Confirmation" from Ron Jeffries description of a user story)

* Some high level communications about the user interactions which have been formally or informally captured

Our teams immediately go to work on created automated Acceptance Tests using this information and further collaboration with the Product Owner and other Subject Matter Experts (SME). After we have built up a good amount of Acceptance Tests for our applications we will collect a bunch of utility scripts which get us to specific parts of the application or do repetitive things like login as multiple types of users. Usually we can put together some scaffolding for our Acceptance Tests rather quickly using these utility scripts and much of the collaboration happens after this.

After setting up our Acceptance Tests we will "tag" them with the name of the Sprint so that we can create a test suite which represents all Acceptance Tests for the Sprint. All of the individual tests which are collected into this test suite should be failing to begin with because we have added in the expectations of the User Story's Acceptance Criteria into the tests. Once a user story has tests ready (meaning failing tests created with acceptance of the tests from our Product Owner) then we begin coding using TDD and potentially creating more QA regression tests which go beyond the Acceptance Tests. These extra tests may go into a different tool (such JMeter, QTP, DBUnit, xUnit, etc...) or could be extra tests added to STIQ.

Once the team has developed all of the artifacts needed to meet their Definition of Done and the Acceptance Tests are all passing for a User Story then the User Story is "accepted" with validation from the Product Owner during the Sprint hopefully. We get feedback on a continual basis through a Continuous Integration tool that we use with STIQ to show if all Acceptance Tests are passing that have been included into the build (meaning the development has been worked through to make that test "pass").

We are not perfect in the creation of all Acceptance Tests at the beginning of the Sprint with the Product Owner, SME, business analysts, etc. But we do have a very good start on capturing the intent of the Sprint deliverables. There are many times that development of code to satisfy an Acceptance Test may identify a potential issue which the Product Owner negotiates with the team to resolve. There are also times that the Product Owner, upon seeing the actual working code, may decide to negotiate on specific details of an Acceptance Test. I believe that the tight feedback loops with the Product Owner increases our ability to deliver closer to what the customer wants without as much rework.

I hope this helps and would be interested in what and how you decide to move forward with Acceptance Testing mostly in the automated capacity. In full disclosure it is not an easy practice to work into your daily routines. There are many bumps and bruises along the way but once it starts to settle out things get a whole lot better. I know for myself that I do not like creating software in any other way.

I have a notion that my biggest obstacle will be laziness. After all, it's just so much easier to write down the requirements in a Word doc than it is to really think about what is needed and how to prove success from the outset.

Another Reason Why Enterprises Don't Like Dynamic Languages

Again, I'm trying to get a handful of our developers to look at different approaches and languages to develop applications. We have many, internal applications that the customer never sees but, still get used by thousands of employees all the time throughout the day. Recently, we identified a new series of apps we need to build and instead of using a heavyweight framework plus Java, I suggested something a lot more lightweight and more suited to rapid development since these applications don't need to be engineered to the same tolerances as our customer-facing applications do. Again, I have to bring up scalability.

When I need to run applications in crowded, power-constrained data centers, every CPU cycle counts. Supporting thousands of simultaneous users takes a lot of boxes, even more so when I have to consider performance numbers like these. Ruby zealots should be particularly embarrassed; the language has been out almost as long as Python and that's the best it can do? And as for Smalltalk, at least make a middle-aged language go as fast as that other middle-aged language: Lisp. I'd even be more confident recommending Smalltalk if it were as zippy as C# Mono. Sure, we enterprisey types are the target of a lot of cheap shots (and even get a rare chance to take one) but, when a new data center costs $25 million to build and we have to consider the cost of power in KWhs where pennies make a big difference, feature development time becomes irrelevant vs. the year-over-year operating cost of those apps on a large scale.

When I try to take a stand and be a voice for change in the face of hundreds (literally!) of C++ and Java developers who have a need for speed, benchmark results like the ones at the end of the URL, above, don't help my case at all. It pretty much gets me and your favorite language laughed right out of the room. Add to that the the typical software developers' stubborn and irrational resistance against trying something new, and the battle is pretty much lost from the start. It's actually very frustrating and discouraging.

Addendum: Many thanks to "keith" for showing me the way to ST/X which, after reading the docs, looks to be exactly what I'm looking for with regards to performance.

How To Accurately Estimate Projects

Back in the 1940's, the Rand Corporation described a methodology to accurately estimate just about anything for any project. The technique is called Delphi and it was enhanced in the 1970's and was re-named to Wideband Delphi after Barry Boehm added some refinements. There is an excellent article at embedded.com that describes the technique. It's worthwhile reading before going on to my next paragraph.

I really get a kick out of the hard core PMBOK crowd when they tell me that agile techniques are fine for small projects but don't scale to large ones. Frankly, I just have to look at some of the MS-Project monstrosities I wonder how scalable those are. Wideband Delphi has been used to accurately estimate some pretty large, expensive and life-critical projects. After reading the article, for anyone with even with a bit of familiarity with most agile techniques, one sentence in the article should really stand out:

"Do some architectural design, bring a group of experts together, have them estimate individually, and then use a defined process to make the estimates converge to a common meeting point."

That's really not much different than "elaborate and understand the user stories, have the team individually estimate story sizes and iterate through negotiations to reach an agreed upon size for the story." That's pretty much what happens in an XP or Scrum iteration, isn't it? That's certainly the approach I have used. Let's take a closer look at the Wideband Delphi and Scrum to see where Scrum has corresponding steps.

Delphi: Assemble a team of 3-7 experts and a moderator.
Scrum: Assemble a team of no more than7-9 including a scrum master.

Delphi: Coordinator presents each expert with a specification and an estimation form.
Scrum: Scrum master presents the stories.

Delphi: Coordinator calls a group meeting in which the experts discuss estimation issues with the coordinator and each other.
Scrum: Scrum master calls a planning meeting with the team and product owner to clarify the stories.

Delphi: Experts fill out forms anonymously.
Scrum: Team members individual estimate each story's size.

Delphi: Coordinator prepares and distributes a summary of the estimates.
Scrum: Scrum master puts stories and corresponding sizes on the wall.

Delphi: Coordinator calls a group meeting, specifically focusing on having the experts discuss points where their estimates vary widely
Scrum: Scrum master allows everyone to explain their story sizes, one story at a time.

Delphi: Experts fill out forms, again anonymously, and steps 4 to 6 are iterated for as many rounds as appropriate.
Scrum: Scrum master facilitates a negotiation between team members to come to a resolution of a story size, one story at a time until all stories are sized.

I just find it interesting that a proven technique gets revived about 50 years later to be codified into agile methodologies. It's also interesting to see that expertise and experience are valued as the key ingredients for accurate estimates above all other methods. Too bad PMBOK doesn't include a "History of Project Management" that goes back 100 years; I think fewer projects would fail if project managers actually repeated the useful parts of history, like recalling that people are always more accurate than (MS) Project.

Ivory Tower or Crow's Nest?

Yes, it is true that architects, especially us enterprisey types, often get accused of being ivory tower academics. I will admit, at least in my case, that is sometimes true although that is certainly not my intent. However, I while I try hard to not work in the tower, I just spend time thinking in it. There's nothing wrong with a little bit of professional dissociation from a problem. In fact, it often aids objectivity. One thing a tower does provide me with is a very different perspective and a much longer horizon. That perspective allows me to think and plan ahead or, even see a way out of the maze. When lost in the middle of Mirkwood, the best thing to do is climb a tall tree to get some reference bearings.

When we were kids, how often would we remark that people looked like ants when we were looking down on them from a tall building or even an aircraft? Of course, it would be nice if large groups of people would be as well organized and cooperative as ants instead of being a meandering herd but, that's another post. To continue, some might even call this top down perspective the "big picture" or "the reason why" for the situation at hand. From a bit of a distance, it is easier to see all the interactions, the real scope, shortcuts, dead ends, pitfalls and approaching problems. Those are all benefits of retreating into the tower. Naturally the main drawback to retreating into the tower is that (most) towers are static. The second is that they are difficult to build or tear down. Both of those drawbacks are contrary to the principle of agility.

Is there such a thing as an agile ivory tower and is there a need for one? I think so, as long as a few key requirements are met, including:

• the benefit must be visible
• its occupants must be in step with the rest of the team
• staying in it too long is uncomfortable
• there is a direct communication and feedback loop

That's why I like the idea of a crow's nest or lookout. It provides the benefits of a tower but it's also strictly utilitarian. Now, if we could only convince businesses of the value of having a lookout place high enough to see beyond the next quarter ....

How To Evaluate Open Source Software

The Business Readiness Rating is a proposed as a "framework for evaluating open source software."

Now that's what I've been talking about!! I would encourage my fellow architects and open source developers to get involved so that what is important to you becomes a guideline for evaluating your own software. I'll be sending an email message to getinvolved@openbrr.org as soon as I finish writing this entry.