Service Harvesting

SOA is expected to give the corporates, that have a love-hate relationship with their monolithic legacy applications, a new lease of life. Clearly management of these legacy applications mainly in mainframes have become the single largest outlay of IT funds, on the other hand, amoebic growth (in terms of functionality and business processes and data) of these applications over time, has made it nearly impossible to replace these applications. To complicate the matter organizations are also losing people, who are the so called "Knowledge Centers" for these applications. Organizations are also facing numerous challenges in terms of business agility which can be traced back to the inflexible nature these applications and their embedded business logic.

The best possible way out for these organizations lies in creating services out of these applications. These services can then be reused to develop new business processes or modify existing business processes to cater to changing business needs.

But, how to go about this? Is there any standard methodology or approach for this? I have discussed about the top-down and bottom-up approaches in one of my earlier posts. Obviously both of them have their limitations and may make the whole initiative bite dust. No effort to identify and create services would be successful without understanding the existing applications and their routines.

In his article Finding Services in the Mainframe, Mike Oara ,CTO, Relativity Technologies, discusses "meet-in-the-middle" approach for service identifications in mainframe applications. I am sure this can be applied to any kind of legacy applications (mainframe, non-mainframe). He also talks about identifying potential services and harvesting them.

According to him...

"In this approach the service modeling team and the mainframe application experts work together to identify potential services that are both useful and feasible, given the existing legacy constraints"

He defines potential services as ...

"Application artifacts which alone or combined have all the characteristics of services"

He also goes on to add a few ways to dig-up functionalities from these applications and delves into the debatable topics like "service composition" and "service granularity". Finally he talks about importance of interactions and negotiations obetween "mainframe" and distributed applications" communities.

IMHO, there are still quite a few gray areas around Service Design? There are two schools of thought advocating different approacheds for Service Design

- Business Transaction Approach

- Logical Data View Approach

In Logical data view approach, a service would be defined with more CRUD type operations. I am not convinced that this is the right approach for designing services?

My preference would be to design services based on business transaction rather than logical data view. But, within the constraints of legacy applications, this option may prove to be road-block.

Where as, when developing new applications it would be appropriate to go for the business transactions approach and keep the data for the services close to the services. Talking about "Service and Data" I am reminded of this blog post "SOA Question: should we carve service independence into the database?" by Nick Malik . He mentioned that...

"If the services are designed well, there should be no cause for a single transaction that adds data under two different services."

This is only possible if we provide for data redundancy across services and synchronize them.

Mashups and Unified Desktops

A few days back Todd Biske has written this nice blog post on Composite Applications, Mashups, Widgets and Gadgets. I think now the industry has somewhat in agreement that composite apps and mashups mean the same, but mashups are more used in the context of web based presentation of composite applications. Lot is being spoken about about Enterprise Mashups at present. I think the simplicity (its all about javascript,DHML and XML) of the technology would slowly eat away the Portal Technology market.

There seems to be a growing demand in organizations for creating "Unified Desktop" for the employees. The requirement is to create a single user interface to access basic features of all kinds of applications in the enterprise. Sometime back I was dealing with such an requirement from a leading online trading firm. The requirement is to build an "Unified Agent Desktop" for the customer care representatives, which would have case management workflows along with the access to multiple applications they use, to manage the cases.

Dashboard Gadgets and Widgets should hopefully make this simpler in the future. Application Vendors should provide these Application Widgets/Gadgets to their customers along with the applications or organizations can develop them based on their simplicity.

Mule jBPM Connector

Mule 1.4 comes with an BPM Connector. This connector can integrate with BPM engines that provide a Java API. If there is a requirement to integrate with a BPEL engine standard web services with the soap transport (axis/xfire) canbe used.

jBPM is the first BPM engine that comes out of the box with Mule 1.4.

It looks pretty cool. You can easily write your long running integration processes in JPDL.

I hope to see integration support for JBossRules (Drools) engine in the future.

Are you looking at implementing SOA as part of your project?

Recently while going through a set of questions asked by my fellow colleagues to a customer regarding an RFP, I saw this question. This was an RFP for a packaged software implementation and the question was asked in the context of integration requirements with the other applications in the enterprise.

I brought this, to point out how product vendors and system integrators confuse customers with questions like this. Understanding of the concept is of paramount importance for its adoption. Dave Linthicum has clearly emphasized this in his blog, where he talks about the need for the vendors (I would add the System Integrators) to got o SOA School.

http://weblog.infoworld.com/realworldsoa/archives/2007/03/soa_vendors_nee.html

There are some other industry leader who have rightly pointed out how SOA has become a "goal" rathor than a "mean" to achieve business agility. There are certain organizations who talk about ROI of SOA initiative etc... IMHO, rather than starting SOA initiatives, which has become a new name of implementing web services and implementing a service registry (or may be implementing an ESB), organizations should try and inculcate the style in the organization. Microsoft Enterprise Architect Nick Malik has written a very good post (http://blogs.msdn.com/nickmalik/archive/2007/01/16/your-soa-is-jabows-just-a-bunch-of-web-services-and-i-can-prove-it.aspx) ,in which he points out, where the maximum benifits of SOA lie in an enterprise.

A few lines from his post..

"IT projects provide tools for business processes. They automate parts of a business process or collect information or reduce errors. The point is… which processes? In the past, traditional IT only succeeded with the processes that changed rarely"

"The problem is that there is a long list of business processes that occur frequently but that are more difficult to automate because they change frequently"

"That is the SOA sweet spot"

In most of the cases we do not try to understand the customer requires and what are his pain points before suggesting a solution. Therefore, "SOA" has become "magic wand" for the sales guys and consultants.

Business Activity Monitoring (BAM) .. Key components

With the amount of industry buzz around SOA and technologies like MDM, BPM and BAM, it would be difficult to find out an IT Leader who does not want the benefits of BAM in his organization. There are quite a few products in the market in this segment which will catch your eye. I would discuss the key components of BAM tool/framework in this post.

The key to any enterprise is to understand the Business Events (external and internal) and treat them accordingly. There are certain business events that would help you know the business heath. There are certain business events which together can tell you the performance, trends etc. This is nothing but business intelligence, in fact real-time business intelligence.

Ideally there would be three components of a BAM tool

1. Event Absorption Layer also known as the Data Collector Layer
2. Event Processing Layer
3. Delivery Layer

Event absorption layer is the one that collects business events from the enterprise. If you look at application landscape of any organization, you would find number of applications (legacy, custom developed, packaged apps etc). Business processes and business logic lie embedded in these applications in many cases. Data collection can be either using push/pull model. Application pushing data or BAM framework pulling the data from these sources. As it is difficult to collect/absorb data from these diverse sources, integration platforms and technologies play a vital role here. Due care should be taken so that this event collection/absorption is as non-invasive as possible so as not to affect the performance of these business applications.

Event Processing Layer has the job of analyzing and correlating these events based on some rules and assumptions. Key Performance Indicators (KPIs) are nothing but filtered and correlated event data based on some rules. This layer consists of a rules engine,an analytics engine and a predictive engine (fingerprinting engine) . Predictive engine uses the analyzed and correlated data from the business events to predict.

Finally, the Delivery layer to deliver the results to the end users. There should be a notification/alert engine to notify the users if required using different channels. There should be portal(preferably web based) for the users to have a look at the filtered, correlated and analyzed data.

Who acquires what and why?

With so many acquisitions and mergers happening off late, it has become very difficult to keep pace with them . The new one in the bloc is "Software AG to acquire webMethods". There is a clear overlap in the product offerings of these companies. For example, Software AG has a service registry/repository product CentraSite that clearly overlaps with wM registry and repository products X Registry and X Broker from their earlier Infravio acquisition. There is a similar overlap in the BPM space too. webMethods has BPM tools as part of its integration platform ( PRT, Modeller, Workflow etc) and Software AG has a similar product Crossvision. Which one of these parallel products (or both) would survive the merger and would be positioned to the customer, only the future will tell. But, this would definitely create confusion and apprehension among the existing customers of both these vendors.

Now the news... webMethods sold itself for less than half ($550 million compared to $1.3 billion for Active) compared to what it paid back in 1999 for Active Software, the EAI company that was supposed to be its future. Reasons... webMethods was sinking back into the red with CEO David Mitchell blaming it on poor sales execution and gaps in its SOA offerings.

"Miko Matsumura" the face of "SOA" for webMethods (remember he is originally from Infravio) gave a very calculated answer to the future of the Registry Segment. In his opinion, JAXR (http://java.sun.com/webservices/jaxr/index.jsp) compliance of these products would help customers to save their investments irrespective of the future road map.

Software AG relies on a heavily-promoted strategic partnership with Fujitsu in the BPM segment for Crossvision that now faces competition from webMethods. But according to Software AG CEO Karl-Heinz "There is a significant sales pipeline with the Fujitsu product, and Fujitsu BPM will continue to be the company’s strategy in the short- and mid-term". But in the next breath he had also added, "[Our own] IP always has a preference wherever they fit into same [product] segment."


So, it may not be "end of the road" for existing webMethods products.....

XPDL,BEPL,JPDL,BPMNS,BPDM et al.. Standards and More Standards

BPM (Business Process Management) is getting more and more visibility in the industry in spite of the fact there are still some of difficult questions on it's justification remain unanswered or at least have divided opinion in the industry. One of them is "Whether to separate business logic from the components/services or not"? This is definitely against basic OO concept of "Encapsulation" i.e. Data and Business Logic of an object should remain inside the object and shielded from other objects. Let me leave it here and get to the real theme of this post..

There are innumerable standards, often from non-profit consortiums or from a group of vendors that are always interested in getting their ideas standardized and widely accepted in the industry, sometimes driven by what is supported or planned to be supported by their products. You can clearly find out the divide existing between these vendors on some standards with competing/overlapping standards popping up now and then to confuse the users and keep the proponents of these standards pulling each others leg.

There are quite a few advantages of using standards, the primary ones being portability and interoperability. Standards will help your code/artifact/entity to be ported from one tool/application/platform to the other and also inter operate with each other without much difficulty.

Let us look at the standards in the BPM space. Any BPM tool/framework provides for

A. Process Definition/Modeling
There are two distinct areas as part of this
- Creating Process Diagrams (ex. How to represent activities,joins,forks etc ). There can be standard diagramming elements which tools can support.
- Storing/serializing of these process diagrams in a common format for other tools to understand.

B. Process Execution
- Some standard interchange format for Process Semantics, which can be understood by the execution engines and this format should be portable across all process engines.

Now comes the question, where does these standards like BPMN, XPDL and BPEL et al stand today.

BPMN is a modeling notation — more than just a diagram, since each element has defined process semantics, abstracted from implementation details — but BPMN has no official XML schema, i.e. no interchange format. The BPMN proponents suggest than you can use any interchange format like XMI for portability.

XPDL captures all the elements of BPMN for interchange, But from a diagram portability perspective, not process semantic portability.

BPEL captures the process semantics and not the diagrams, but assumes the processes as a sequence of calls to web services. Everything in a BPEL is a web service operation, not "an activity", i.e. a unit of work

JPDL also captures the process semantics but it is very specific to jBPM (now with Redhat) . It assumes that every process consists of "States" and "Actions", somewhat like state machine.

On portability I would completely agree with "Bruce Silver" on what he has to say in one his blog posts.

"The argument over whether BPEL or XPDL is more "portable" is based on different interpretations of what portable means. If you mean the same process semantics can be executed on two different engines, then BPEL is more portable. If you mean that the same diagram can be created in two different tools, then XPDL — especially if you allow the target tool to ignore the graphical details that don't carry over"

Now a days every product vendor claims to support either BPMN, XPDL and BPEL or all of them. But, what does this support mean???There are many of them who have their own proprietary Interchange formats and have capability export/import from/to standard formats.There are some tools which can store these interchanges any standard/non-standard format, but would add one layer of translation before the process engine creates an instance of the process and executes it.

Is this what we are looking for in BPMS tools ?? IMHO, we should look for moving the process semantics and the diagrams repository from one tool/engine to the other as it is.. Also, we should look at whether the process engines can be hosted in any container (ex. JEE Server,ESB etc).
To put it straight we are looking for process tools and engines that are BPEL or XPDL based and not provide support for them in way of export and import or internal translation.

The question is.. while evaluating tools/engines...which of these standards what we should look for? With the current state these standards, I would definitely say there is no concrete answer. The choice is yours.. But, the decision should not be based on what the product vendors support, but what is the pros and cons of these standards and how you foresee the evolution of these standards.

SOA Approach .. Top Down or Bottom Up..

In his blog, Grady Booch (of UML and Rational Fame) talks about 'how Bottom Up approach for SOA can be a disaster'.

"Going back to the A part of SOA, the issue then is one of abstraction, separation of concerns, and all the usual fundamentals of architecture. I've seen some folks suggest creating an SOA from the bottom up: look at a silo, identify the potential services, and publish them, then weave a system together from them. This is in essence technology first. In my experience, this is a recipe for disaster and/or serious over-engineering. You've got to start with the scenarios/business needs, play those out against the existing/new systems, zero in on the points of tangency, and there plan a flag for harvesting a meaningful service. These styles, and their resulting costs/benefits, are rarely discussed. "

Following things should be kept in mind while deciding on the approach for SOA

1. It is business first and technology afterwards.
2. Do not think SOA as the panacea of all Business and IT alignment issues.
3. Keep it simple... Do not over-engineer.
4. Understand your business, existing IT assets/systems before going for something like SOA

Taxonomy and Ontology..

Recently I attended a session on "Architectural Knowledge Management " in a TOGAF conference, where I learnt about the KM concepts like topic maps, concept maps, ontology et al. Of course I was aware of mind mapping as a technique...But, I was quite impressed by the amount of research that is going on in this space...

And here it is again.....while doing some research on Service Registries/Repositories I bumped into Ontology once more...

Do you want to store the relationship between your services and their meta data, store some of the service properties like certain service qualities, say, minimum and maximum service delivery times; may require certain payment obligations, say, advance credit card payment, rather than only classifying them only into categories and sub categories??

Of course, that would be of great help for the consumers in finding services based on what they exactly want. They would also know the service dependencies (how the services are related).

If the service registry is expected to be the management backbone of an Enterprise SOA, then the they should support these features.. Does any of the Service Registries in the market or the open source space have support for this..May be...?? I could only find one of them clearly talking about it..

http://ebxmlrr.sourceforge.net/tmp/Registry_Capability_Matrix.html#ebrr_ontology

SOA and EDA, Do they complement?

There have been plenty of debate on "SOA vs EDA". Vendors give their flavour of the difference which suits their product:)...Everyone with the intention of cashing on these industry jargons somehow...

Of course both of them are architectural styles and can live with/without the other... But what needs to be understood is the underlying concept of these styles.

In SOA, the core building blocks are services.. some provide services for others to consume..
Where as EDA the core building blocks are events/business events (internal/external) and how the business treats it...

So, if you have services that receive and treat business events... what do you have.. a SOA or an EDA or both??

Stefan Tilkov,InfoQ SOA consultant on SOA - Interview excerpts

Stefan Tilkov,InfoQ SOA consultant has a very informative interview on SOA

He defines SOA as....

"I can give you my personal opinion and the aspects that I think are important for a SOA. I think a very important thing is that the driving factor has got to be services, as it's the main construction element. If you're talking about applications, about components, about modules and subsystems you're missing something. You should be talking about services and you should structure your whole landscape by using services as the primary abstraction mechanism. That's a difference against lumping functionality together in applications in a different manner."

And his advice for the organizations looking for SOA....

"I think if you ask a vendor the answer is pretty clear because what you need is their product. For a vendor I think is pretty hard to avoid trying to sell you a product because that's their business and it's totally understandable they will try to sell you some ESB product, some middleware stuff, some magical integration infrastructure that will do anything for you. I think that if you really want to do SOA this is something that you should rather avoid than seek out. What you should do is to make your endpoints aware of the fact that they are being exposed as services."

Defines ESB as...

"If you want to know what an ESB is I think you'll find different answers there as with many of those acronyms. I think a reasonably well-accepted definition is that an ESB will help you connect systems using different technologies to a central service oriented infrastructure so that a lot of the technical tasks are taken over by the ESB product for you so you can rely on it to provide you transformation, routing, adaption of different protocols and all of this WS- * mess, this is all going to be handled by the ESB"

About JBI ....

"The JBI spec is pretty new, it's at version 1.0 about to be released as we speak and obviously with a spec that is so new there's a lot of open and underspecified stuff, but I think it has value because if it takes off it's going to allow you to combine JBI components that you purchased from different vendors into a solution that matches your needs; you can take your BPEL engine from one vendor and you can take your WS-I compliant binding component from another vendor and you can plug them together and hopefully things should work out."

He goes on to add his views about the the reasons for wide acceptance of SOA in the industry

"What I think is exciting is the fact that it creates, or at least that it is perceived to create the opportunity to align business and technology. The whole architectural discussion about aligning your businesses along services has reached a management level where technology usually doesn't play any role. When you talk about services at that level, you're not talking about any of those styles, you're not talking about any technological decision, but you're thinking about your business in terms of services; you have to find the right services, you have to find them at the right granularity, you have to think about ways to price them, to create incentives for people to create or consume them. There are lots of business aspects involved here. It's the first time that the management level actually making those decisions, that the people who actually do the business, really believe in this.

What next for EAI/ESB product vendors???

With JBI being widely aacepted by the industry, there is little these vendors, who had been surviving till date providing proprietary stacks, can do other than jumping onto it.

With the loose-coupling and plugability provided by the JBI framework, there are two areas where the vendors can invest in..

- Building a good JBI Container (Meta Container)
- JBI components (Service Engines, Binding Components)

Think of you buying a JBI container from Company A , buy a set of JBI components(ex. Rule Engine, Orchestration Engine) from Company B and buy some other JBI components (ex. FTP, AS2, JMS connectors, are called binding components.. ) from Company C and create the ESB for yourself and put the stepping stones for an SOA Enterprise.

Also the JBI container would provide with management and monitoring interfaces, and can be monitored by any JMX based Monitoring Tool/Interface. Customer would have the flexibility of having the best of breed JBI components and Containers available.

So, the next question is, if we can choose and plug JBI components, do we then prefer buying them when there are quite a few good ones available in the open source.

With open-source projects like servicemix catching the eye of many organizations, there is an interesting duel between these vendors and the open source community in the offing.

Now that the product vendors have very little role to play, organizations should think of creating their ESBs themselves or involve some system integrators who would create product agnostic solutions. Rather than thinking of how best to use the product architecture and work-arounds for feature defficiencies, there would be more emphasis on the solution architecture.

Ofcourse, there can be arguments that JBI is Java specific and there are many critics of Java, who would go any distance to show how java sucks, but, I feel this is a very good stepping stone in the direction of standardization of integration/SOA space. I can visualize Microsoft with its own set of frameworks and competing specifaications/technolgies playing its own game.

AFAIK microsoft is now obsessed with DSL...........................................................

SOA...Where to start ???

We know, SOA is all about services. With lot of hype around SOA, intiatives/projects started by organizations and leading vendors throwing their products as the perfect solution for SOA, the question remains... Where to start ...

Here is my way of looking at it.....................

We all know, SOA is not about projects, not about products and not about creating webservices... it is about having an architecture style in the enterprise that revolves around business services. Implementing business processes that are nothing but a set of business services coupled loosely to realize a business use-case.

And,my first task is to build/implement business services and then the processes....Hold on...How can I identify these business services??First, identify the key business processes and model them.Most steps and activities would be nothing but business services. Now that you have the list of business services with you along their contracts (schemas- business documents that are exchanged by these services).You can then look at the existing implementations (there can be more than one) in the enterprise/applications that can realize these business services. You may have to wrap some existing implentations or create new implementations of these services. Here we can take the advantage of an ESB for all its features like message transformation, routing, messaging,service mapping etc.

Now that the services are ready for consumption, create a repository/registry to store the metadata about these services (ex. Contracts, Policies etc). You can automate the whole service lifecycle management and the policy enforcement. Some vendors in this area can be of help and now a days some of the ESBs come bundled with these features.
Then create loosely coupled business processes by orchestrating these business services. In my view it is essential but not madatory to have a orchestration engine in place to orchestrate these services.You can add additional capabilities like BAM to provide business visibility and associated agility.

Let us not forget that the basic building block of SOA are services and the better we design these services, the more complete/relevant the SOA is and their reusability will give business the desired agility.It requires an enterprise wide effort rather than projects to realize SOA. As organizations work in a set business units that manage their applications/systems and the way IT projects are funded by these business units, it would be a herculian task to start an enterprise wide SOA intiative. It is easy to have all of them buy to these principles, but the real task lies in making them all follow the same path. But with the SOA road-map, governance model in place and projects understanding and following the basic principles of SOA, may be we can climb half the Everest.

ESB (Product Evaluation), What to look for?

Recently, I visited one of the prospects in the retail industry to discuss about an intiative they were taking up to evaluate ESBs. With ample confusion in the industry about ESB, I assumed that they are looking for an off-the-shelf product/platforms claiming to be ESBs(I do not outrightly reject those claims) that can give them the desired benefits.

With an application landscape that consists of number of stand-alone home grown and packaged applications and predominantly point to point batch interfaces(file based) often developed as and when required by business, clearly there was plenty to look forward in terms of creating an Enterprise SOA?(I always restrict myslef from talking about Enterprise SOA with customers as this is not something which is a mere technolgy thing and can be acheived overnight)

But, having ESB in place, ofcourse helps an enterprise to move towards SOA. But, lets not forget that it is only a small part of the whole story. You can easily end up having a non SOA architecture inspite of having the ESB(s) in place that I would discuss in a later post.


So, coming to the point, what one should you look for in these products during evaluation..

- The architectural fit with existing platforms and technology
- Completeness of the stack and ESB features (ex. routing,transformation,service mapping etc)
- Additional features for SOA (ex.Business Process Management, Business Activity Monitoring and Portal etc)

Note: Mere claim by the vendors would not suffice, you have to look for the support details for their completeness..
- Performance (Some kind of benchmarking..)
- Fault tolerance, Management and Monitoring etc