System, Method and Computer Program Product for Performing an

Evaluation Service

Technical Field

The present invention provides a method, system and computer program product for performing an evaluation of the maturity of an IT production of a company, and particularly maturity related to the structure, process, system management and services provided of the IT production.

Background of the Invention There are currently three main known methods of evaluating the maturity of the IT production of a company (herein then called the ^λ Cie IT' ) , all of which involve interviews and/or questionnaires. These prior art evaluation methods include:

- manual analyses (with or without manual diagrams) which are mainly employed in consulting firms;

- partial analysis tools which are used for contingency planning, e.g. the "Methodologie d'Analyse des Risques Informatique et d' Optimisation par Niveau" - the well-known ^λ MARI0N' method - and other methods linked to security in the banking sector and which all include analysis criteria that can be graphically represented to enable alarm thresholds and targets to be established; and

- the Capability Maturity Model Integration (CMMI) model, which is applicable to all industrial sectors and mainly

focused on the IT application development environment.

Each method presents own drawbacks. The manual analysis methods do not allow automatic calculation of results. In addition, the duration of the interviews conducted with manual analysis methods are not controlled and discussions regarding the evaluation of the ^λ Cie IT' can be quite protracted. Accordingly, such evaluations can be very time-consuming and expensive .

Furthermore, a particularly serious limitation of such manual methods is that they are not applicable to all industrial sectors .

Similarly, whilst the partial analysis methods (the MARION-type) provide automatic tools for calculating performance metrics, they have functional limitation as being applied only to contingency planning and security sector.

The CMMI model has traditionally focused on the development stages of an IT application life-cycle. Accordingly, a CMMI evaluation can be quite time-consuming. In particular, it typically takes about 30 days to complete a CMMI evaluation. However, in today's IT environment, the time to market is becoming increasingly short. Accordingly, such deadlines often do not accommodate the prolonged duration of a traditional CMMI evaluation. Furthermore, the main challenges in managing today' s IT application processes reside in the IT production environment in addition to the IT development environment.

Thus, all of these evaluation methods have serious flaws which avoid them to be used in IT production environment, also known as the service delivery environment of a company. Qualitative data and "gut feel" do not provide consistent checkpoints that would enable a ^λ Cie IT' to easily and rapidly build a technical target and a coherent IT transformation plan. Moreover, those approaches do not allow to have relevant

technical indicators in order to improve the current ^λ Cie IT' business .

Accordingly, more quantitative, objective methods are necessary for evaluating the internal IT production processes of a company. The present invention offers such method.

Summary of the Invention

The foregoing problems and shortcomings of the prior art are addressed and further advantageous solutions are provided by the present invention wherein is provided a method, system and program product for performing an analysis of a business entity to devise an IT solution to improve the service delivery of this entity.

According to the invention there is provided a method of evaluating the IT production environment of an entity, the method comprising the steps of:

(a) establishing the scope of the evaluation;

(b) determining objectives for the IT production environment; (c) categorizing the IT activities of the IT production environment;

(d) identifying and defining each activity within each category of the IT production environment;

(e) generating a set of personalized questions for each of the defined activities to match with the objectives and scope of the evaluation;

(f) conducting interviews using the set of personalized questions;

(g) analyzing results of the interviews; and (h) developing recommendations and targets for the entity on the basis of the analyzed results to enable the entity to achieve the objectives. The method may further comprise the step of presenting the

recommendations and targets to management line of the entity.

It will be appreciated by the man skilled in the art that the results, recommendations and targets may further allow the CIOs of the entity to rapidly build an IT transformation plan and to put in place control indicators to implement such.

It is also to be noted that the time required for an evaluation using the method of the present invention depends on the size of the ^λ Cie IT'. For a small ^λ Cie IT', the present invention will provide an evaluation in approximately one day, while for a large ^λ Cie IT' , the present invention will provide an evaluation in a range of ten to twenty days. Accordingly, the evaluation provided by the present invention is considerably cheaper for the ^λ Cie IT' than manual evaluation methods.

According to a second aspect of the invention there is provided a system for evaluating the IT production environment of a company, comprising:

- a categorization tool comprising a means of categorizing the activities of the IT production environment into a plurality of organizational areas and a means of specifying each activity within each of the organizational areas;

- a computer implemented interview guide comprising: -a means of selecting a set of questions from a question repository for each organizational area; and

-a means of adapting the set of questions to each activity to match with objectives and scope of the evaluation;

- an interview recording tool to record the results of interviews; and

- an evaluation and scoring tool which comprises a means of processing the results of interviews to produce a measure of the maturity of the IT production environment of the entity.

According to a third aspect of the invention there is provided a method of evaluating an IT production environment of a business entity, the method comprising the steps of: a) establishing the scope of the evaluation; b) determining the objectives of the IT production environment; c) categorizing the IT activities of the IT production environment; d) identifying and specifying particular activities within each of the categories of the IT production environment; e) searching for questions from a repository of evaluation questions, within the categories of the IT production environment, to match with the objectives and scope of the evaluation; f) adapting the questions with a vocabulary extracted from a pre-defined dictionary to best reflect the activities of the IT production environment; g) conducting interviews using the adapted questions; h) analyzing results from the interviews; i) developing recommendations and targets for the IT production environment on the basis of the analyzed results, to enable the IT production environment to achieve the objectives; wherein steps a) to f) and steps h) to i) are performed by a first party, and step g) is performed by a second party, the second party being different from the first party.

Further embodiments of the invention are provided in the appended Claims.

In another aspect of the invention, a computer program product is provided. The computer product comprises a computer usable medium having readable program code embodied in the medium and the computer program product includes at least one component

to operate the steps of the method as described in the appended claims.

The present invention thus provides an automatic interview questionnaire, which provides a mark value according to precisely described criteria. Such automatic interview device eliminates the risk of biased appraisals by interviewed line-personnel. Consequently, the automatic interview questionnaire reduces the risk of bias and errors associated with such appraisals.

In a preferred implementation, the present invention provides a powerful data analysis and visualization tool that automatically calculates performance metrics for five organizational areas, namely: the management, the framework (i.e. the general structure), the transverse programs (i.e. programs used by different entities - not dedicated to a single entity), the key processes and the operations. Furthermore, the tool allows a diagram representation of the same. Accordingly, the present invention provides a powerful tool for interpreting the data acquired from the interview questionnaires.

The present invention provides a mechanism for automatically grouping marks in order to bring to the fore two significant diagrams, namely the management profile (i.e. recommendations to modify the current management hierarchy) and the operation profile. This enables conclusions regarding potential improvements to be easily made.

The use of the automatic appreciation questionnaire enables the duration of interviews with line-personnel to be controlled. Furthermore, the automatic data analysis and visualization tools also enable the duration of the data analysis and conclusion formation period to be controlled.

The present invention further provides a strong base of data and technical gap analysis to quickly build new targets, to make a coherent IT transformation plan and to set up a consistent set of technical performance indicators.

Finally, the powerful data analysis and visualization tools of the present invention enable new consultants to be trained more easily on evaluation methods.

Brief Description of the Drawings An embodiment of the invention will now be described with reference to the accompanying Figures in which:

Figure 1 is a block diagram of the system architecture of the present invention;

Figure 2 is a flow diagram of the method of the present invention;

Figure 3 is a polar diagram of the management profile of a ^λ Cie IT' ;

Figure 4 is a polar diagram of the operation profile of a ^λ Cie IT' ; and Figure 5 illustrates a typical data processing environment wherein the present invention may be practiced.

Detailed Description of the Preferred Embodiment

1. Overview

The following description will provide a brief discussion of the rationale for ^λ Cie IT' evaluation.

For sake of clarity, the term ^λ Cie IT' used in the description will have the meaning of the IT production environment of a Company, while the IT production environment will encompass all the IT services provided by infrastructure and data processing environment for the end user.

Bearing in mind the importance of the CMMI model for evaluating IT application development, it is useful to briefly

describe this methodology in more detail, although the main distinguishing characteristics between the two methods is that the present invention is implemented in the IT production environment rather than the IT application development environment.

Having established the context for the present invention, the discussion will be oriented to a description of the architecture of the system for evaluating an IT production environment of a ^λ Cie IT' , and the operation of such method. For the sake of brevity, the system for evaluating an IT production environment will be known henceforth as the ^λ IT evaluation system' . Similarly, the method of evaluating an IT production environment will be known henceforth as the ^λ IT evaluation method' .

2. Rationale for a ^λ Cie IT' Evaluation

The three main characteristics of an IT application project are its cost, its schedule and its quality (i.e. how well the product satisfies the customer). An IT application project succeeds if it meets or exceeds the customer' s expectations on all the three fronts. Despite recent advances in IT application technology, the ability to develop and deliver reliable, usable IT application within budget and on schedule continues to elude most ^λ Cie IT' . Bad IT application plagues nearly every organization that uses computers, causing lost work hours during computer downtime, lost or corrupted data, missed sales opportunities, high IT support and maintenance costs, and low customer satisfaction.

Studies into how bugs get into an IT application process and why they seem to be so hard to prevent suggest that bugs are not the result of a single specific failure. Instead, bugs result from a series of disconnects and miscommunications among the IT specialists involved in the planning, development, testing and maintenance of a it application

project. In essence, the root cause of many of the problems with IT application projects is the inability of ^λ Cie IT' to manage the IT application process.

The ability of a ^λ Cie IT' to manage an IT application process is designated by the maturity of the IT organization. In an immature IT organization, IT application processes are generally improvised by practitioners and their managers during the course of a project. Even if an IT application process has been specified, it is not rigorously followed or enforced. Such IT organizations have no objective basis for assessing product quality or for solving product or process problems. Similarly, such organizations possess little understanding of how the steps of an IT application process affect quality. Thus, such organizations find it difficult to predict product quality. More generally, immature IT application organizations routinely exceed budgets and schedules and their customers typically have little insight into a product until delivery.

In contrast, a mature IT application organization possesses an organisation-wide ability for managing IT application development and IT maintenance processes. Such IT organizations accurately communicate the IT application process to their employees and carry out work activities accordingly. The process definitions are updated when necessary, and improvements are developed through controlled pilot-tests and/or cost benefit analysis. Furthermore, roles and responsibilities within the process are clear throughout the project and across the IT organization. Managers in mature IT organizations employ an objective, quantitative basis for judging product quality and analyzing problems with the product and process.

To enable an immature IT organization to improve, it is necessary to develop a specific improvement target therefor.

Similarly, it is necessary to establish a trajectory detailing the short and medium term interim targets leading towards the overall improvement target, the IT transformation plan. However, before such improvement targets and trajectories can be established, it is necessary to make an overall evaluation of the current state of the organization. The evaluation must be holistic considering all aspects of the IT organization and its IT application processes. Furthermore, the evaluation must be reliable and in view of the rapid rate of development and short time to market in the IT sector, the evaluation must be rapid. Finally, since IT application has pervaded almost all industrial sectors, a mechanism for evaluating a ^λ Cie IT' must be capable of assessing IT application projects across a wide variety of industrial sectors.

Similarly, the same quality and maturity steps are to be applied to the IT production environment in terms of management, organization, operation environments and IT services. The IT production environment delivers IT services to end users. The measurement of the quality of the IT services shows the maturity of the IT production environment, which determines the maturity of the IT services. The present invention is thus oriented towards providing tools and methods for evaluating the maturity of the IT production environment.

3. The Capability Maturity Model (CMM) (trade mark) in an IT application development

The Capability Maturity Model (CMM) (trade mark) was developed to give the US Department of Defense a yardstick to assess and describe the capability of IT application contractors to provide IT application on time, within budget, and to acceptable standards. From there, it quickly evolved into a powerful tool to guide process improvement, not only for IT application development, but for many related fields including systems engineering, product acquisition, team management and research and development.

The Capability Maturity Model Integration (CMMI) (trade mark) is a modification of the CMM that can be used to guide process improvement across a project, a division, or an entire organization. CMMI helps integrate traditionally separate organizational functions, set process improvement goals and priorities, provide guidance for quality processes, and provide a point of reference for appraising current processes.

The CMMI is best known for its five levels of organizational maturity. Each maturity level is associated with a number of related process areas. The process areas can be viewed as very detailed checklists of what goals need to be achieved, activities performed, and records created and maintained to satisfy the requirements of a specific part of an overall development process.

4. IT application development and production life-cycle

An IT application process can be defined as a set of activities, methods, practices and transformations that people employ to develop and maintain the IT application and associated products (e.g. project plans, design documents, code, test cases and user manuals) in accordance with the requirements of an IT application project.

An IT application life-cycle is a sequence of phased activities that represent the various stages of engineering through which an IT application project passes. There are many different IT application life-cycle models, of which the waterfall model is probably one of the most well-known.

The classical waterfall model comprises the steps of: performing a feasibility study to determine whether developing the product is financially worthwhile and technically feasible; determining the exact requirements of the IT

provider and drafting a specification; designing the software; translating the it application design into source code; - unit testing each module of the design; integrating the modules; system testing; deploying to an IT production environment; and maintenance.

The steps from performing the feasibility study to system testing represent the development stages of an IT application project wherein both unit and system testing are performed in a development environment (i.e. a test-laboratory environment with a configuration identical to the customer' s system) . Accordingly, during the development phase, errors in the IT application will not affect the operation or productivity of a client's system.

An IT production environment is the environment in which the IT application will finally run (i.e. the customer's system) . The IT production environment encompass any kind of IT application business users within any kind of telecommunication networking, i.e. from a single end-user computer to a network of computers connected to the Internet for example. Accordingly, once the IT application is deployed on a client's system, any errors in the IT application may have a negative effect on the operation of the customer' s system, thereby damaging the customer's productivity.

5. Architecture of the IT Production Environment Evaluation System To recall, it has been previously explained that the prior art IT development evaluation systems do not satisfy the IT production environment needs for performing an IT production evaluation. Main reasons reside in that the IT processes are

radically different to provide IT services to the end user. Thus, the inventors have created specific tools and methods to answer such needs which lead to decrease the time to market whereas increasing productivity.

Referring first to Figure 1, the IT production environment evaluation system 5 comprises the following tools: a categorization tool 7; an interview guide 9 coupled to a question repository 11 and a specialized dictionary 13; - an interview recording tool 15; a current status depiction tool 17; an evaluation and scoring tool 19; and a data output tool 21.

The categorization tool 7 is a tool for categorizing the IT activities of an IT production environment. In particular, the categorization tool 7 categorizes the activities in five main organizational areas, namely the management, the framework, the transverse programs, the key processes and the operations. The categorization tool 7 also identifies and specifies each activity within the above five organizational areas.

The question repository 11 contains question formulations based on well-established IT application process models and evaluations thereof. The specialized dictionary 13 comprises vocabulary specific to particular IT production environment activities. As the dictionary is repeatedly used for new evaluation processes, the value of its content is then enriched and its usefulness in maturity evaluation for other customers continues to grow. The specialized dictionary 13 is unique to the present invention and does not employ the same vocabulary as a CMMI model for example.

The interview guide 9 is a tool for assisting an interviewer 23 in conducting an interview according to the previous

categorization of the organization (by the organization categorization tool 7). In particular, the interview guide 9 selects interview questions from the question repository 11 and applies vocabulary selected from the specialized dictionary 13 to best reflect and analyze the specified activities of the organization.

The interview recording tool 15 enables the results of the interview to be recorded (in the form of appreciation questionnaires) and stored for later analysis. The interview recording tool 15 also includes the current status depiction tool 17 which depicts the current results of the evaluation. The results acquired with the depiction tool contains a set of information to devise a transformation plan and to define a set of control indicators for this plan, and further contains information to provide an IT gap analysis which reflects the lack of industrialization processes.

The evaluation and scoring tool 19 analyses the data acquired from the interview to provide a score to the organization according to the results of the analysis.

Preferably, a score from a five-level scale is provided. A score of between level zero and level one indicates that the organization is becoming aware of the need for improvement. A score of between level one and level two indicates that partial attempts have been made to control the IT production environment process, but that the procedures for controlling such are not implemented as industrialized-based procedures. A score of between level two and level three indicates that a global definition of the process exists in the organization and that the overall procedures are industrialized ones.

A score of between level three and level four indicates that the organization is bringing methods and procedures for controlling the IT production environment process into general

use and thus being partially under control. Finally, a score of between level four and level five indicates that the IT production environment process is comprehensively defined and under control and that there are continuously ongoing efforts in place for improving the process.

The data output tool 21 provides a report of the gap analysis. The data output tool 21 also provides graphical representations of the gap analysis in the form of polar diagrams of the management profile and the operation profile or scatter plots and histograms.

6. Operation of the IT Production Environment Evaluation Method

Referring to Figure 2 together with Figure 1, the evaluation method comprises the following steps: - establishing 25 the scope of the evaluation; determining 27 the objectives of the organization; categorizing 29 the IT activities of the organization; identifying and specifying 31 particular activities within each of the categories; searching 33 for questions from the question repository 11 within the categories to match the objectives and scope of the evaluation; adapting 35 the questions with vocabulary selected from the specialized dictionary 13 to best reflect the activities of the organization; conducting 37 interviews with key line personnel; analyzing 39 the data collected from the interviews; comparing 41 the data with the objectives of the organization to provide 43 a series of recommendations and improvement projects to enable the organization to achieve its objectives; and presenting 45 the results to the management of the

organization .

In the first and second steps (25, 27) of the evaluation method, an interview is held with the CIO of the organization to establish the scope of the appraisal and to determine the objectives of the organization. For example, the organization may require guidance as to how to learn from its customers to develop the quality of its IT services. Similarly, at an internal level, the organization may require guidance on how to comply with regulations. The organization may also require guidance as to set up problem determination process, changes IT process, or how to industrialized procedures and so on.

The activities of the organization are considered 29 within five main categories, namely the management, the framework, the transverse programs, the key processes and the operations. The IT activities of the organization within the above five categories are specified 31 and used as a basis for selecting 33 questions for the subsequent interview process.

In particular, the questions for the interview process are selected 33 from the question repository 11 to best reflect the activities and scope of the evaluation. The questions are presented 35 using a vocabulary selected from the specialized dictionary 13 to best reflect the activities of the organization. Collectively, the questions form a tailored questionnaire that is used to guide the interviews of line personnel. At the same time, appropriate data analysis and visualization tools are prepared.

Key line personnel are then identified and interviewed 37 using the previously developed questionnaire. The interviews take the form of guided discussions and in order to obtain statistically significant results, preferably ten to twenty of such interviews are conducted. The results from the guided interviews may be supplemented with additional free interviews

with product professionals to obtain further supporting factual data to explain the organization ^λ s status.

Once all the required data has been gathered, calculations are performed 39 thereon with the pre-defined data analysis tools and the resulting processed data are examined 41 using the agreed data visualization tools (e.g. of the management and operational profiles) . A series of recommendations and targets are then provided 43 to assist the IT production environment in achieving its desired objectives. The eventual recommendations and targets are then presented to the management of the organization (CIO and middle management, ... ) . In particular, a management profile and operation profile of the organization may be presented in polar graphic form as shown in Figures 3 and 4. The profiles may also be sent to the management over any type of transmission network.

The present invention relates to IT production environment processes and systems for enabling the described analysis. In a preferred implementation, the system is embodied in a data processing system such as that depicted as item 500 in FIG. 5. The system includes components typically associated with conventional data processing systems including without limitation an input device 501 such as a keyboard or other such input apparatus, and a display unit 502, both coupled either locally or via a network connection to a central processing unit (CPU) 503 for exchanging information therewith. The display unit 502 may be used to present to the customer the various surveys, and questionnaires required to elicit information from the customer, and the input device is utilized to gather from the customer responsive information required for implementing the inventive processes presented herein. The CPU 503 includes a local memory element 504 for storing a limited set of commands and data and is further coupled to a storage device 505 such as a direct access storage device (DASD) for storing and retrieving information

required for the processing of commands and data at the CPU. Finally, the CPU may be further coupled either directly or remotely to other peripheral devices 506 which may include additional computer systems via local or wide area networks and the Internet, and which may further include such peripheral devices as printers, facsimile machines, scanners, network connection devices, tape drive units, etc.. It will be appreciated that modifications and alterations may be made to the above without departing from the scope of the invention.