A Federated Approach to SOA

Follow service-oriented architecture (SOA) to its logical conclusions and you recognize that modern corporations will soon be operated on the equivalent of aviation’s “fly by wire.” The traditional governance and management means of running a business — the levers, pulleys, cables and brute muscle — will through SOA become more automated, rules- and event-driven, self-service, preprogrammed, policy-orchestrated … agile.

Instead of directing a business on how to function from the boardroom megaphone, with explanations and edicts, countless meetings, and then reviews and crass incentives, there may soon be policy-driven decisions on how to execute made on a more federated basis — collaborative and productive by making IT not just the means of operating the computer applications, but making IT the means through which to operate the very business itself.

Sound far-fetched? Consider that whomever controls the full-fledged SOA to a large extent controls the company. So how should that control actually work? Will companies take a lesson from world history on how to run the business and allow for federated and balanced power? Or will mismatched control over business elements, exacerbated by IT that cannot reflect the will or wills of the controlling factors, drag productivity and the company down? To fail at SOA is to fail at modern business?

Our assemblage of analysts and guests have some unconventional and startling conclusions, as well as thoughtful insights. Join experts Steve Garone, Joe McKendrick and Jim Kobielus — along with guest Miko Matsumura, the vice president of SOA products at webMethods — for our discussion, hosted and moderated by me, Interarbor analyst Dana Gardner.

We start the discussion on what contributes to SOA failures, and the need for operational cultural transformation to accompany any meaningful advancement in SOA. We then examine federated approaches to balancing governance and control for business as well as — yikes! — governments. We conclude that SOA “failure” is probably necessary and a good, as it shows momentum — as long as the organization deals with failure maturely and constructively. No quitters!

Listen in: There are lessons from world history on how to run your business, and on how to use IT to define, balance and invoke power. Roll over Clausewitz.

Listen to the discussion (47:48 minutes).

Here are some excerpts:

On Failure in SOA

Miko Matsumura: From our perspective, the thing that is really vital about this topic is that in 2007 we’re as likely to see catastrophic failures as we are limited success. There are a huge number of moving parts within SOA, and I’m going to use that almost as a handout point to this very well-considered group of folks. We need to categorize for the listener which moving parts are more dangerous than other moving parts, because those are the things that eventually cause the thing to kind of wiggle the wrong way and send it to a tailspin.

Steve Garone: Corporate backing … is more focused on the people-oriented things and the collaborative issues associated with deciding what to build and how to build it [than technology].

Matsumura: The most dangerous moving parts — are people. From our perspective, the system is sort of cybernetic, half-human, half-machine. The human pieces of SOA are the parts that we’ve seen in failure mode. It’s not necessarily just the human beings themselves [but] … the interfaces between the human world and the machine world, whether those interfaces are the specifications used to design applications, or the mechanisms used to manifest constraints … [Do] people, when they do fight each other, fight each other in a way that’s productive, as opposed to destructive?

Jim Kobielus: When projects are pitched … [as if] we’re going to totally clean up our development practices and our integration practices … you’re just setting up the SOA project for failure.

A Battle for Control of SOA

Matsumura: The people who control the SOA are the people who essentially control the policies. The policies include metadata, repository and registry — the kind of policies that are machine-enforceable, but also involve human factors. In a way, the model is more of an equal partnership now. On the other hand, system integrators (SIs) like to control policy as a way to permanently set up a base-camp inside an account, pour people through the door, and take over. It’s something that we know they’re salivating about.

A CIO had this mistaken impression that the service interface abstraction allowed him to outsource completely the operational concerns and the implementation concerns, and eventually to treat this service interface as something like a child’s car seat, where really mom is driving.

It’s important to treat the interface abstraction layer like a saddle on a horse, which means that the only people who can successfully get from Point A to Point B are the people who have the skill of riding and controlling the horse, which is the service implementation. It’s really an abstract or complicated metaphor. It’s not hard to lose control.

[Customers said:] “We don’t want a single vendor to come in with a product and a set of services, because we don’t want them to control everything. We want an independent to mix things up.” There is a very significant danger of the inmates running the asylum or the integrators taking over the whole account from the inside.

Joe McKendrick: There is a lot of jockeying going on, and Steve has pointed this out in a previous podcast as well. There’s a tension between various groups in the enterprise. I guess there’s a lack of a clear definition as to who is going to be doing what, and who is going to be controlling what.

SOA, of course, is inherently cross-enterprise — or in theory it’s cross enterprise. An SOA that’s confined to a single silo or single piece of the enterprise, by definition isn’t necessarily an SOA. It’s another proprietary system.

The Risk of Monopolizing Change

Gardner: Is SOA a democratization type of an effect, or is it really giving command-and-control through policies that you could think of as a governor or an accelerator — a brake-pedal/dashboard type of an affair — where suddenly those in the organization that may not have had power before gain it? Is the failure when the control doesn’t go to the right people?

Matsumura: This question is basically The SOA Question, because the people who control the policy metadata are the people who are running the show. The thing that we’re trying to establish here is that the SOA success model is essentially a model where there are federated controls and delegated controls. The reason why this term “federation of control” is so significant is because we’re trying to achieve a balance between the central function, the IT function, and the distributed function, or the business function.

If you want to balance these things, you need a mechanism that enables some amount of control by the people who are on the periphery, in the business units, trying to create agility. Then, [there comes] some amounts of control by the people in the center, who are trying to create more orthodox standardization and security and orthogonal cross-cutting concerns. Having the wrong people controlling the wrong things is exactly the pattern that causes things to go a little nuts.

Kobielus: The extent to which your SOA initiative and your SOA governance are totally centralized and totally rigid — but your business environment and the challenges and threats and so forth are constantly changing — then your SOA failure will ultimately become a business failure, a failure to adapt.

Garone: The world is about finding that midpoint, where control and governance is centralized enough to keep things safe and secure, and to be able to take advantage of business opportunities — where consolidation makes sense — while at the same time staying agile.

A Federal Metaphor

Matsumura: If you look at it from a metaphorical perspective, for example, the federal government of the United States is a very interesting model. You have essentially a bunch of business units called states, that each have their own legislation, their own competency centers called state legislatures, and even their own executives called governors.

Those look a lot like business units to me. If you look at the notion of federation and the federal government model, what you see is this whole principle of jurisdiction. Ultimately, competency centers become the legislative bodies within these organizations. All of the efforts that I’ve seen to codify methodologies around SOA tend to focus on these competency centers or centers of excellence, primarily because there needs to be an inclusive organization for adjudication and jurisdiction, as opposed to having a model, where it’s just a single iron-clad dictator that controls all policy.

You can actually look at the failure modes of failed states. If you look, for example, at how you establish and foment democracy, there are some models, some really good, real-world cases about how not to establish democracy. Not to get too overly abstract, but there are a lot of practices and principles around establishing policy federation. The interest in doing so is the interest in establishing a controlled paradigm that actually serves the common good in a way that enables agility, but also enables this centralized capability of control.

Kobielus: But governance is an abstract concept, and you don’t necessarily want to dictate one governance model that’s applicable or should be applicable to all organizations and industries. Everybody has their own pressures, market pressures and so forth. In terms of SOA governance, there are radically centralized models in a given organization.

Gardner: You might want to do the equivalent of a Myers-Briggs test and figure our what kind of company it actually is. Then, figure out in what way to approach governance, so that we don’t try to overstep what’s possible on a linear basis. I suppose it’s also evolutionary. Some companies might need to start out as strict dictatorships, and then perhaps the government withers away and it becomes a democracy. We’ve seen the example of Eastern Europe over the last 20 years.

Matsumura: The U.S. Constitution, which has some key design patterns in it. If you actually look at the separation of power declared in the preamble, it says that the purpose is, “… in order to form a more perfect union.” So, there’s this notion of the intent of the formation of this governing entity, which is the goal of a more perfect union, which essentially means that there’s a distribution of power and that the consent of the governed essentially be the overriding principle.

The idea that comes out of that, though, is the clause “provide for the common defense.” That’s really talking about the security domain, whether it’s physical security or technical policies associated with the current data. The idea is that it actually should be a federated concern. In other words, security is everybody’s business. You can’t just delegate it to one unit and say, “It’s your business.”

Gardner: Technology so permeates how a company operates, particularly if you’re Internet-facing and if you’re using and exploiting the Internet for more and more of your supply chain, your distribution, your transportation, for the way in which you attract sales and customers, and so on and so forth. So technology now is at an intersection with the corporation as an organization, and perhaps that’s what’s forcing this need for a different look at how to organize in general, and, therefore, on how to govern.

Dealing With Failure

Matsumura: We talk about the post-modern corporation. Where are these companies going to get their IT? Where are they going to get their technology? We’re seeing more and more instances of companies going outside, not wanting to get involved with the bits and bytes of managing a technology infrastructure. We call it “software as a service,” “managed hosting,” and various types of acronyms and terminology.

The metaphor of nations and the competition between nations has typically been along the lines of warfare in our history. Look at the metaphor of business at war, which is essentially competition for the survival of the integrity of your company against all others. It’s not on the battlefield, but it’s for customer value, for creating services that people treasure. In the history of warfare between governments and nations, what we found is that the organizations that leverage technology to their advantage are the ones that come out ahead.

Abdicating the responsibilities of the management of technology to a commoditized provider creates an extreme vulnerability because your competitive differentiation should not be held or embodied by some generic provider. … Control over how your organization behaves and controls your assets and resources strikes me as something that you would never want to commoditize.

Gardner: Clearly we’ve defined here that a successful SOA is a lot about politics, power and moving beyond traditional norms of organization. How you do that probably is going to involve failures. If the Unites States is a good model, it had to fail a couple of times. It failed with the Articles of Confederation. It failed in dealing with slavery up until the Civil War, and perhaps for a hundred years afterward in terms of how it was dealt with in practice, if not in law.

Perhaps we should look to failures as a necessary set of learning activities, in that SOA is not going to just happen and spring up like a fungus or a mushroom after a spring rain, but it’s going to have to be something that’s hard-earned.

Matsumura: The way I want to respond to is that having maturity in the way that you deal with failure is essential. If you look at the way that our policy system functions within the United States, what you have is you have a set of policy assertions about what it is people can and can’t do. But then, you actually have a policy enforcement mechanism that’s heterogeneous and distributed. You have the FBI, the CIA, the state and local law enforcement, the Army and the National Guard.

You have all these different policy enforcement points everywhere, manifesting these policies. I think that having a learning engine that monitors, adapts and revises policies, and having a competency center, an adjudication point that’s deliberately there for the purpose of making those adaptations — that is an essential function.

Click here for more podcasts.