Approaches to the Ex-ante Evaluation of Information Systems: A Critical Review

TABLE OF CONTENTS

1. INTRODUCTION

2. THEMATIC BACKGROUND
2.1. Information Systems
2.2. Evaluation
2.3. Value

3. CRITICAL REVIEW OF EVALUATION APPROACHES
3.1. Antecedents to the Review
3.2. Effect-Assessing Approaches
3.3. Effect-Locating Approaches
3.4. Discussion of Usability of Approaches

4. EVALUATION IN PRACTICE
4.1. Review of Empirical Studies
4.2. Interpretation of Empirical Studies
4.3. Discussion and Future Research

5. CONCLUSIONS

APPENDIX

A. Overview of Evaluation Approaches

B. Literature Search Strategy

REFERENCES

LIST OF FIGURES

Fig. 1: Model of IS life cycle

Fig. 2: Classification of evaluation approaches

Fig. 3: Generic types of indicators

Fig. 4: Basic concept of the BSC

Fig. 5: Example of a BSC modified for IS evaluation

Fig. 6: IE factors for computing the project score

Fig. 7: The three stages of the NNC model

Fig. 8: Process chain map

Fig. 9: Example of effect chain map

LIST OF TABLES

Tab. 1: Overview of application systems

Tab. 2: Ideal type characterisations of approaches to IS evaluation

Tab. 3: Components of value

Tab. 4: Criteria for a characterisation of evaluation approaches

Tab. 5: Basic structure of a SMART table

Tab. 6: Description of IE factors

Tab. 7: Example of budget allocation

Tab. 8: Stages of the customer resource life cycle

Tab. 9: Overview of reviewed evaluation approaches

Tab. 10: General problems of IS evaluation

LIST OF ABBREVIATIONS

illustration not visible in this excerpt

ABSTRACT

This paper critically reviews approaches for the evaluation of investments in information systems prior to their implementation. First, the ground for the review is prepared by examining characteristics of evaluation, information systems and value. A classification of 54 evaluation approaches identified in English and German literature is then presented. Examples of each class are reviewed and their advantages and drawbacks are discussed. Their use in evaluation practice is analysed through the examination of empirical studies and directions for future research are given.

1. INTRODUCTION

Today, a significant share of corporate funds is spent on the implementation, upgrading and maintenance of an information system (IS). Recent studies show that in 2001 the IS budget of companies worldwide accounted for an averaged 8.8% of total corporate revenues (cf. CSC, 2001). Consequently, a thorough evaluation of investments in information systems before, during and after the implementation of a project is important. However, the normative literature reports a great deal of difficulty in the appraisal of these investments (cf. Irani, 2002:11). Although IS evaluation has been an issue for both academics and managers for more than three decades now, there are still serious concerns about how to select projects for investments, how to control the development and how to measure benefits after the implementation (cf. Farbey, 1999:189). This concern has been matched by increased research activity which prevailed through two broad streams. The first stream aimed to directly measure the payoff of IS investments for companies and came to mixed conclusions (cf. Dehning and Richardson, 2002:8). The second stream addressed the question of how IS investments can actually be assessed by decision-makers and particularly focussed on the research of evaluation criteria, evaluation methods and the very nature of the evaluation process (cf. Avgerou, 2000:570).

Of late, several deficiencies in the field of evaluation methods have induced calls for in-depth research. Academics have criticised the current state of the field as being immature and fragmented (cf. Mahmood and Szewczak, 1999:491) and have thus demanded “an overview of the whole panoply of evaluation methods, together with ... the assumptions they depend on ...[in order to enable]... the identification of gaps.” (Farbey, Land and Targett, 1999: 205). Another concern is a growing mismatch of theory and practice of IS evaluation (cf. Arribas and Inchusta, 1999:151). Although more than 50 techniques for the evaluation prior to implementation (ex-ante evaluation) have been suggested by the academic literature, managers still draw some of their IS investment decisions on so-called “acts of faith”, i.e. on their intuition and instincts (cf. Renkema and Berghout, 1997:1; Fitzgerald, 1998:16). These are worrying phenomena for proponents of rational decision-making.

Therefore, a critical analysis on the potential of the proposed methods, also with regard to their role in evaluation practice, is desirable. This is the purpose of this thesis which particularly addresses the following research problems:

(1) What approaches are available for the ex-ante evaluation of investments in information systems? What are their strengths and weaknesses and to what extent are they usable in practice?
(2) What role do evaluation approaches play for practical IS investment appraisal?

Essentially, answers to these questions are sought in a comprehensive review of English and German literature. The practical side of IS evaluation is explored through the analysis of published empirical studies. Thereby, this paper attempts to reflect the state-of-the-art of the literature on IS evaluation approaches. Investments into information systems, instead of entire information systems, are analysed due to the broad definition of such systems in this thesis.

This thesis is organised as follows. In Chapter 2, the ground is prepared through a delimitation and discussion of the central concepts “evaluation” as general activity, “information system” as evaluation object and “value” as key evaluation criterion. In particular, evaluation problems associated with each concept are identified. Chapter 3 presents the criteria for the selection, classification and characterisation of evaluation approaches within this paper. A review of representative approaches is given, their individual advantages and drawbacks are discussed and the usability of all reviewed techniques is examined on a general level. In Chapter 4, empirical studies on IS evaluation in practice are presented, interpreted and discussed with respect to the role of evaluation approaches. This reveals several directions for future research. Finally, conclusions are given in Chapter 5.

2. THEMATIC BACKGROUND

2.1. Information Systems

In the previous chapter, a brief introduction stated and justified the focal point of analysis in this thesis. On this ground, the paper continues with examining the background before which evaluation approaches are applied. Therefore, aspects of information systems, evaluation and value are subsequently discussed and especially problems associated with IS evaluation are identified.

The following working definition reflects the fundamental understanding of an IS within this thesis: an information system is an integrated socio-technical system with a certain life cycle that aims to provide information within organisations (cf. Krcmar, 2000:20; Alpar, 2002:28; Stickel, Groffmann and Rau, 1997:336). Being a part of this definition, the concepts of “information” and “system” require further delimitation due to their ambiguity. Information encapsulates knowledge for the purpose of taking effective action (cf. Stickel, 2001:2) and possesses five main attributes: actuality, accuracy, degree of aggregation, form of presentation and costs (cf. Alpar, 2002:10-11). A system is generally referred to as a “set of interrelated elements” (Ackoff, 1971:662) while an IS as a special type of system can be characterised as an open, dynamic and complex system: open, because its elements interact with their environment; dynamic, because this interaction can yield alterations of the elements’ features; complex, because of the high number of elements and their respective relations (cf. Krcmar, 2000:20).

The above definition raises the question of what the actual elements of an IS are. An answer to this question might be sought by looking at the disciplinary bases of this work: organisational theory and computer science. In organisational theory, an organisation possesses one IS (organisational IS) that comprises all organisational activities and processes related to information (cf. Krcmar, 2000:20). Information is seen as the foundation for organisational decision-making processes and it is therefore the system’s main function to enable and to support organisational planning and control (cf. Gutenberg 1983:268).

An organisational IS comprises a formal and an informal side. The formal side is designed by the organisation and includes for instance the official reporting system whereas the informal side refers to the informal actions of organisational members such as a phone call to obtain urgent information (cf. Zmud, 1983:62; Davis, 1974:197-200). In contrast, computer science has developed a more technical understanding of information systems as a super-class of application systems (cf. Krcmar, 2000:20). The function of these systems is to automate parts of the formal organisational IS, thus they can be understood as all hardware and software components in an organisation (cf. Stickel, 2001:4). As exhibited in Table 1, they address differently structured problems at different levels of an organisation.

illustration not visible in this excerpt

Tab. 1: Overview of application systems (Adapted from: Alpar, 2002:31)1

In this thesis, elements of an IS are understood as the technological infrastructure, i.e. hardware and software, as well as manual procedures for development and maintenance of the IS (cf. Davis, 1974:251). Thus, formal organisational IS based on application systems are referred to as information systems.

Another feature of an IS mentioned in the definition above is its life cycle character. Life cycle models were suggested in the IS literature for software applications (cf. Krcmar, 2000:110-114) and for entire systems (cf. Farbey, Land and Targett, 1993:16-21). Figure 1 shows a phase model of the IS life cycle that builds on prior work (cf. Farbey, Land and Targett, 1993:16-21; Boehm, 1976:1227; Macharzina, 1999:306). Since it primarily serves as^[1] a framework for discussing different functions of evaluation in Chapter 2.2, problems of identifying the present phase of an IS or the end of the life cycle are not further discussed.^[2]

The model depicts the IS lifetime as a repeating cycle comprising six phases. In the strategy development phase, an IS strategy is being developed using either a bottom-up, a top-down or a mixed approach and general objectives for the IS sector are formulated. For achieving these objectives, several investment opportunities, referred to as projects, might exist from which one or more are chosen in the project selection phase. After that, a detailed planning of the previously selected project (portfolio) takes place in the project specification phase and leads to a concrete project plan. In the project implementation phase, this plan is executed and an IS is developed, extended or modified. Before setting it into operation, outcomes of the project are tested. The final introduction and operation of the new or changed IS follows in the IS usage phase. Throughout this phase, the system is subject to alterations due to faults, insufficient capacity or changed requirements, so that follow-up projects might become necessary and the cycle might re-start at the project selection phase. The life cycle comes to an end when changes become overwhelming and replacement is needed. Then, the cycle is repeated starting either at the first or second stage.

illustration not visible in this excerpt

Fig. 1: Model of IS life cycle

Within its life cycle, an IS can be subject to different types of investments. An investment in general is characterised through the allocation of scarce resources, the expectation of a positive return during a certain period of time and a certain lifetime (cf. Renkema, 2000:100). Basically, three categories of IS investments can be distinguished: investments to set up a new system (installation investments), to extend (add-on investments) or to replace (replacement investments) an existing system (cf. Perridon and Steiner, 1999:29-30). Alternatively, IS investments can be classified according to the degree of initiated change, ranging from pure automation of existing routines to business transformations (cf. Farbey, Land and Targett, 1993:121-131). In a synthesis of both categorisations, substitutive, complementary and innovative IS are distinguished. Substitutive IS replace human labour in order to achieve calculable cost-savings whereas complementary IS support human labour to generate estimatable productivity and effectiveness increases. Finally, innovative IS aim to realise competitive advantage (cf. Parker and Benson, 1988:103; Alpar, 2002:77).

The different roles of information systems in corporations are another point to consider in IS investment appraisal. A study by Ragowsky et al. (1996:98) confirmed that different organisations can gain different benefits from the same IS application. Therefore, the normative literature increasingly recommends to adjust evaluation procedures to organisational characteristics (cf. Premkumar and King, 1992:101). In particular, the roles of information systems can differ with regard to two aspects. First, information systems can be of different strategic relevance for an organisation. While for some companies such as Amazon and Ebay IS activities are a part of crucial business processes, they play a mostly supportive role in other organisations e.g. in the handicraft sector (cf. Avison, Cuthbertson and Powell, 1999:422). Second, the degree of system integration varies across organisations and can range from stand-alone systems with no integration to sector-wide integration such as realised through the S.W.I.F.T. network in the banking sector (cf. Stickel, 2001: 140-142).^[3] Both aspects directly affect IS evaluation because they determine the reach of potential investment impacts and the locus of responsibility for system planning. However, as a general trend of the last few decades, computer-based information systems have penetrated organisations more and more due to the rising potential and shrinking prices of IS technology (cf. Premkumar and King, 1992:100). Simultaneously, a trend to the integration of isolated systems in and across organisations developed (cf. Smithson and Hirschheim, 1998:164).

A great deal of the difficulties associated with IS evaluation stems from characteristics of information and information systems. With regard to information, the determination of its value is particularly troublesome. One reason is that information has no intrinsic value because its value depends on the associated purpose (cf. Droste, 1986:90) and varies inter- subjectively. Defining the value of information in relation to the decisions that can be based upon it is also problematic due to intervening factors. For instance, the quality of managerial decision-making does not only depend on the quality of the underlying information but also on the individual competences of a manager to make use of this information (cf. Farbey, Land and Targett, 1993:13-14). Another reason is a fact known as “information paradox”: in order to assess ex-ante an information’s worth, its content has to be disclosed to determine the worth of the activities based on it after which there is no need to acquire the information anymore (cf. Walter, 1995:203).

Features of an IS account for further problems. First, the returns of investments in such systems can seldom be completely expressed in monetary terms because, being often of supportive nature, the system has a great distance to the cash flow generating activities of an organisation (cf. Huber, 1999:111). Second, the IS lifetime is hard to anticipate due to threats of technological obsolescence and changed organisational requirements (cf. Farbey, Land and Targett, 1993:13). Third, information systems are often difficult to distinguish from each other. Reasons for this are the growing integration of application systems (cf. Smithson and Hirschheim, 1998:164) and the evolution of systems through modifications and extensions over time (cf. Clegg et al., 1997:857). Fourth, the realised value after an IS project is seldom exclusively attributable to a particular IS investment, but is due to compound effects (cf. Anselstetter, 1984:10-11; Huber, 1999:112). For instance, major benefits and costs could not emerge from the IS investment per se, but more from the reorganisation induced by it (cf. Stefanou, 2001:206; von Dobschutz, 2000:446). Finally, IS investments often trigger not entirely predictable, organisation-specific processes of change (cf. Clegg et al., 1997:857). These processes complicate reliable estimations because they can prevent the realisation of expected benefits or can cause unforeseen costs (cf. Chircu and Kauffman, 2000:66-67). For instance, sales personal could hinder the value realisation of a CRM system by resisting to maintain the necessary data.

2.2. Evaluation

For the purposes of this thesis, the following working definition of IS evaluation is suggested: a process of determining by quantitative and/or qualitative means the value of an information system investment to an organisation (cf. Doherty and King, 2001). This definition implies that two fundamental activities of IS evaluation are the collection of evaluation-specific data and, based on this, the ascertainment of value (cf. Wittmann, 1985:263). It also indicates that on the whole the organisation as total, not individuals or groups alone, is supposed to be the primary beneficiary of any IS investment. As a shortcoming in the above definition, the “nicely ambiguous” (Veryard, 1991:3) character of the concept “value” necessitates further delimitations that are presented in detail in Chapter 2.3. Essentially, it is understood as the sum of positive (“benefits”) and negative consequences (“sacrifices”) of an IS investment that can be of both a financial and non- financial nature (cf. Renkema and Berghout, 1997:2).

Several taxonomies of IS evaluation are suggested in the academic literature. According to the timing, evaluation before (ex-ante evaluation) and after (ex-post evaluation) the implementation of an IS project is generally differentiated (cf. Remenyi, Money and Sherwood-Smith, 2000:25). A categorisation into formative and summative evaluation is also proposed. The former involves monitoring the processes and products of system development and gathering user feedback for use in the refinement and further system development whereas the latter focuses on assessing the impact, usability and effectiveness of the system (cf. Remenyi, Money and Sherwood-Smith, 2000:27).

Research and practice have developed a variety of evaluation approaches. To delimitate an evaluation approach from mere planning techniques, the following definition of ex-ante evaluation approaches is proposed. An ex-ante evaluation approach describes a procedure prior to the implementation of an IS in which criteria for the evaluation (as defined above) of an IS investment are prescribed or can be generated. Several requirements of evaluation approaches have been suggested in the IS literature (cf. Schumann, 1993:168-169). They should facilitate an analysis of cost-savings, productivity increases and competitive advantage associated with an IS investment, in order to make the approach applicable for appraising substitutive, complementary and innovate information systems. Different types of investment impacts should be envisaged including quantitative financial, quantitative non-financial and qualitative factors. In addition, risk as well as the indirect effects involved in IS investments should be considered.^[4]

Academics have approached the topic of IS evaluation from diverse research perspectives, ranging between two extreme positions: the formal-rational perspective and the interpretive perspective (cf. Serafeimidis, 2001:66-67). The formal-rational perspective seeks the worth of an IS in a system’s performance and financial profitability whereby it emphasises the economic and technical aspects. In contrast, the interpretive perspective focuses on the analysis and understanding of the social and subjective nature of IS evaluation. Particularly, the interaction of the technology with organisational structures, culture and stakeholders is addressed. Both perspectives correspond to distinct models of organisational decision-making (Daft, 1989 in Renkema, 2000:137; Schafer, 1988:192). In the rational model, evaluation is regarded as a means to optimise corporate actions by selecting efficient and effective choice options. The members of the organisation are commonly pursuing organisational goals and decision power is centralised. In contrast, the political model attributes pluralistic objectives to the organisational members and power is distributed across shifting interest groups. Evaluation is viewed as a disorderly process of mediating between heterogeneous coalitions that tend to use and withheld information strategically. As key difference, evaluation aims to determine objective value irrespective of context in the first model and context-specific understanding by the exploration of different views in the second model.

As Table 2 illustrates, different types of evaluation approaches are related to both perspectives. It should be noted that the views are given as ideal types for illustrative purposes and to draw the attention to their contrasts. In practice, IS evaluation may take place between both extremes, reflecting in varying proportions their characteristics.

Overall, this dichotomy of perspectives reflects in a sense the influence of two research paradigms on the field: the positivistic and the interpretive paradigm (cf. Serafeimidis, 2001:66).^[5] While the interpretive perspective follows the interpretive paradigm and considers the partly political context of evaluation, the formal-rational perspective disregards, like the positivistic paradigm, the context of evaluation.

illustration not visible in this excerpt

Tab. 2: Ideal type characterisations of approaches to IS evaluation (Adapted from: Jones and Hughes,

2001:193)

The usage of evaluation approaches induces the question of who to assign as evaluators. Several authors argue for an active participation of key stakeholders in the IS evaluation process (cf. Avgerou, 1995:435) by asserting that their involvement would create the right level of commitment needed for a successful project implementation (cf. Renkema, 2000:138; Markus, 1983:441).^[6] Essentially, three classes of key stakeholders are distinguished in the IS literature (cf. Remenyi, Money and Sherwood-Smith, 2000:17-19; Buss, 1983:120; Khalifa et al., 2001; Fitzgerald, 1998:25): (a) information technology (IT) professionals including internal staff, contractors and consultants, (b) business managers, and (c) end-users. However, the actual selection of evaluators is a research field in its own rights which is for reasons of space and time not further regarded.

The literature has acknowledged several reasons for evaluating IS investments prior to their implementation. From an formal-rational standpoint, it should be, at least in theory, a natural concern of management to estimate the worth of an investment in order to enable an optimal allocation of scarce resources (cf. Smithson and Hirschheim, 1998:160; Farbey, Land and Targett, 1993:12). With regard to the large amounts of organisational funding consumed by IS investments (cf. Farbey, Land and Targett, 1999:203), the determination of the relative merits of alternative projects competing for resources becomes even more important (cf. Ballantine, Levy and Powell, 1998:242). Furthermore, situation-specific evaluation is required because research has so far failed to provide general evidence that organisations can gain productivity increases through the use of information systems. Instead, empirical findings are very mixed, leading to the “productivity paradox” debate, where in some cases productivity increases, while in others it declines (cf. Smithson and Hirschheim, 1998:161). Moreover, ex-ante evaluation can provide insight to the interface between the technology and fundamental organisational processes (cf. Hirschheim and Smithson, 1988:21). It generates a set of measures to exercise control over the implementation and usage of an IS by estimating resource requirements and benefits to be gained (cf. Farbey, Land and Targett, 1992:110). Furthermore, measures for the degree of goal achievement by an IS investment can be produced through evaluation procedures (cf. Irani and Love, 2001:184). Additionally, it facilitates feedback and learning in order to improve IS evaluation practices and IS development capabilities (cf. Irani and Love, 2001:184; Farbey, Land and Targett, 1992:110).

Regarded from the interpretive perspective, evaluation can act as a mechanism for gaining commitment and, in highly politically influenced environments, for legitimisation (cf. Powell, 1992:33). The achievement of understanding and consensus among decision-makers can be promoted by making often implicit evaluation criteria explicit (cf. Farbey, Land and Targett, 1993:12). Evaluation procedures can also be a part of a regular justification process for investments (cf. Farbey, Land and Targett, 1992:110) and can function as a political instrument to rationalise already-made decisions (cf. Schafer and Wolfram, 1987:37). Furthermore, having a demonstrable record that a “legitimate” ex-ante evaluation procedure has been followed can mitigate the people involved if the system does not live up to expectations.

Evaluation can fulfil diverse functions within the IS life cycle presented in Figure 1 (p.5). While ex-ante evaluation takes place in the first three phases of the cycle, ex-post evaluation is conducted in the last three.^[7] In each phase, the focus of appraisal activities differs in terms of evaluation objectives, objects and criteria (cf. Farbey, Land and Targett, 1993:16-21;

Macharzina, 1999:306). Following the phase of strategy development, the congruency of IS objectives and business objectives is assessed in order to ensure a fit of both. Evaluation in the project selection phase aims at enabling an optimal choice of project (portfolio) by ranking all alternatives under consideration. Important criteria are the value and the risk of the respective alternatives. At the end of the project specification phase, the evaluation focus lies on scrutinising the feasibility and establishing efficiency of a concrete project plan for which value and risk are also the dominant criteria. Throughout the project implementation phase, costs and schedules are under constant review and compared to plan figures in order to support the steering of the project. Evaluation activities in the IS usage phase concentrate on monitoring the project’s impacts, comparing realised and hoped for value as well as locating unexpected value. Criteria for measuring IS performance should orientate at those criteria that were used in previous phases and reflect objectives associated with the overall IS investment (cf. Baumol and Frie, 1999:138). Once the end of the IS life cycle is reached, replacement options are, unless there is a change in the general IS strategy, sought and appraised like investment alternatives in the second phase.

The review of evaluation functions above matches the formal-rational perspective. An allocation of evaluation functions to life cycle phases was possible because in this view evaluation is regarded as a systematic, objective and, above all, explicit process. However, a corresponding presentation for the interpretive perspective is not possible by means of this thesis because this view depicts evaluation as an unsystematic, political and implicit process (cf. Jones and Hughes, 2001:193; Avgerou, 1995:427).

Ex-ante evaluation in general has to cope with a number of problems from which three problem types are primarily faced during formal-rational evaluation: problems of data collection, forecasting and judgemental bias. Data collection problems arise from the fact that gathering accurate and complete data to appraise investment alternatives often causes considerable costs or is impossible at all (cf. Anselstetter, 1984:10-14). Forecasting problems result from the future-orientation of ex-ante evaluation and the uncertainty involved in making predictions, e.g. on future cash flows accruing through an investment.^[8] Another problem type is judgemental bias that affects the evaluator’s faculty of objective judgement. On the one hand, this can occur consciously when, for instance, an evaluator is susceptible to

the persuasion of vendors and consultants (cf. Fowler and Walsh, 1999:9; Irani and Love, 2002:76) or when a project manager becomes too attached to her projects (cf. Farbey, Land and Targett, 1993:13). On the other hand, there could be unconscious bias where for example evaluation is used to rationalise already-made decisions instead of conducting an objective analysis (cf. Schafer and Wolfram, 1987:37). Such bias could be also based on the motivations of evaluators and their relationships to suppliers (cf. Anandarajan and Wen, 1999:331).^[9]

In addition, the interpretive perspective identifies other problems in ex-ante evaluation. Communication problems between evaluators can complicate the evaluation process due to divergent perspectives (cf. Ballantine, Levy and Powell, 1998:242; Smithson and Hirschheim, 1998:161). For instance, IT and business managers might have contrasting perspectives on a certain problem because of their different professional backgrounds (cf. Huber, 1999:111­112; Avison, Cuthbertson and Powell, 1999:442). Moreover, stakeholder groups such as customers and vendors can exert substantial influence on evaluation processes (cf. Farbey, Land and Targett, 1999:196) which requires the problematic definition, analysis and involvement of such groups (cf. Irani, 2002:12).

2.3. Value

Defining evaluation as “determination of value” in Chapter 2.2, automatically raises the question of what is meant by “value”. In most of the literature on IS evaluation, the term is treated as being self-evident, thus remaining undefined, or is defined in one of any number of ways (cf. Bannister and Remenyi, 2000:233). Since this poses the danger of misinterpretation, a definition of value is proposed in the following section. This is performed by subdividing the term “value” into single underlying components.

Table 3 shows components of IS value (cf. Renkema, 2000:99). Accordingly, the value of an IS investment consists of positive impacts (benefits) and negative impacts (sacrifices). These impacts are divided into monetarily measurable impacts (financial impacts) and non-monetary impacts (non-financial impacts). With respect to financial impacts, a further distinction is made between net cash flow and net profit. Net cash flows comprise the sum of cash inflows and cash outflows whereas net profit is defined as the accounting registration of income and expenditures. Regarding non-financial impacts, positive contributions of an IS investment and negative contributions are differed which are of either quantitative or qualitative nature.

illustration not visible in this excerpt

Tab. 3: Components of value (Adapted from: Renkema, 2000:99)

Several taxonomies have been developed to categorise value. Regarding investment impacts, a distinction between direct and indirect impacts can be made according to the attributability of cause and effect (cf. Anselstetter, 1984:2-3). Direct effects are benefits or sacrifices that are directly attributable to an IS investment whereas indirect effects depend on other influences. For instance, upgrading IS hardware might directly aim to realise reduced server response times in order to decrease the average time of order handling as an indirect effect. This, however, also depends on the employees’ working speed.

The IS literature recognises various benefits of IS investments that fall into three broad categories: efficiency, effectiveness and strategic advantage (cf. Silk, 1990:185; Ward, 1986:27).^[10] Efficiency means “to do the things right” (Drucker, 1968:85) and is essentially a measure for decreases of an input for a given output. Examples include cost-savings by workforce reduction, a faster retrieval or delivery of reports, and an improved accuracy or reliability of information through IS investments (cf. Lederer and Mirani, 1995:162).^[11] Moreover, effectiveness connotes “to do the right things” (Drucker, 1968:85) and centres on increases in the output for a constant input. Consequently, effectiveness measures the degree of goal achievement and is thus an attempt to identify the contribution of a particular action towards certain stated objectives (cf. Schafer, 1988:194-195).^[12] For instance, an IS investment could enhance employee productivity or could enable the organisation to respond quicker to changes in the business (cf. Lederer and Mirani, 1995:162). The third benefit type, strategic advantage, refers to the potential of an IS investment to improve the market position or to support success factors of a corporation (cf. Nagel, 1990:31; Krcmar, 2000:283-284).^[13] A popular example is American Airline’s flight reservation system SABRE (cf. Krcmar, 2000:201-202).

Costs classifications presented in the literature differ in terms of scope and detail. While they at least comprise costs for hardware, software and training (cf. Marsh and Flanagan, 2000:428), more comprehensive taxonomies also encompass additional items including environmental costs such as extensive cabling and additional furniture, running costs for electricity and phone bills, maintenance costs, costs for data back-ups and security, and wider organisational costs such as costs for temporary job interruptions during a system transition or installation (cf. Hochstrasser and Griffiths, 1991:181-185).

The choice of the analysis level is another important issue for the estimation of IS value (cf. Ahituv, 1980:62). Essentially, the following corporate levels of analysis were suggested in the literature: user, work group, department, business process, corporation and inter-corporation (cf. Davern and Kauffman, 2000:127; Alpar, 2002:78). While IS evaluation is per definition in the previous chapter supposed to determine the value of an IS to the organisation, several arguments were provided for including also or only levels below the corporate level into the analysis. The validity of IS evaluation could be increased when IS value is analysed at the locus of creation because the effect of intervening variables could be reduced (cf. van Wegen and de Hoog, 1996:249). Furthermore, benefits of IS investments are in many cases not directly captured by a corporation but are reaped by a variety of stakeholders instead (cf. Jurison, 1996:263). Therefore, they might be the focal point of analysis. The corporate level could also be used as a starting point in order to stepwise envisage underlying levels (cf. Schumann, 1993:176-177). However, general recommendations on the selection of an appropriate analysis level are difficult with regard to the fact that different types of IS investments have different effects on corporate levels. Thus, the choice of analysis level remains a context-specific issue that is not further analysed in this thesis.

The use of value as the central evaluation criterion entails several problems for IS evaluation. From the formal-rational viewpoint, identifying all cost and sacrifices involved in an IS investment is difficult due to the indirect and unplanned effects of such actions (cf. Farbey, Land and Targett, 1993:13). In addition, benefits and sacrifices are hard to estimate because they tend to evolve throughout the IS lifetime (cf. Smithson and Hirschheim, 1998:161) or they are delivered with a certain time lag (cf. Brynjolfsson and Hitt, 1998:52-53). Two other problems originate in the interpretive perspective. First, the ambiguous concept of value lacks a universally accepted definition which could lead to misunderstandings (cf. Bannister and Remenyi, 2000:233). Second, there can be conflicting perceptions of value in an organisation (cf. Smithson and Hirschheim, 1998:161). For instance, improved access to headquarters information might be seen as a benefit for those working in the field but as an erosion of the power base of managers at headquarters.

3. CRITICAL REVIEW OF EVALUATION APPROACHES

3.1. Antecedents to the Review

The previous chapter laid the foundation of this thesis by delimitating and discussing three central concepts: “information system”, “evaluation” and “value”. Chapters 3.2 and 3.3 can now proceed with a review and critique of selected ex-ante evaluation approaches. The review had to concentrate on samples for classes of techniques because a consideration of all identified approaches would have exceeded the scope of this thesis. A list of approaches that were reviewed but not presented is exhibited in Appendix A. Based on the review, Chapter 3.4 investigates the general usability of the approaches from a theoretical point of view. This chapter elucidates how the approaches were selected, classified and compared.

Each approach in the following review and in Appendix A was chosen from the body of existing work according to three selection criteria. First, it had to be published in a reliable source to exclude mere guidelines without scientific foundation. Thus, the literature review only included (a) peer-reviewed academic journals, (b) books based on academic articles, and (c) other publications that obtained academic appreciation by being referenced and accepted in one of the sources mentioned above. The general literature search strategy is explained in more detail in Appendix B. Second, the approaches had to comply with the definition of an evaluation approach presented in Chapter 2.2 and their use had to be explicitly suggested for the ex-ante evaluation of IS or IT in one of the sources mentioned above. Therefore, the respective references for each technique under review are given. Third, an approach had to facilitate a value analysis by corporate decision-makers. Consequently, the broad stream of research intending to actually measure the value added by IS investments was not included (cf. Brynjolfsson, 1993; Brynjolfsson and Yang, 1996; Potthoff, 1998).

A classification of all located approaches is suggested in Figure 2. It synthesises work of several authors (cf. Schumann, 1993:169-170; Irani and Love, 2002:78-79; Renkema and Berghout, 1997:2-6) and is orientated at the unit of analysis that each approach refers to.^[14] Due to the very different characteristics of the approaches, this classification is neither exhaustive nor mutually exclusive but fulfils its main function to construct a framework for in-depth discussion of the techniques.

illustration not visible in this excerpt

Fig. 2: Classification of evaluation approaches

At the first level, effect-assessing approaches and effect-locating approaches are distinguished. The first type assumes the availability of necessary data and focuses on the calculation and description of investment impacts. The second type aims to locate those impacts as well as the respective data (cf. Schumann, 1993:169). Effect-assessing approaches are further subdivided into financial, indicator and multi-criteria approaches. Financial approaches exclusively consider financial factors such as cash flows, income and expenditures and calculate how advantageous an IS investment is from the financial perspective. Indicator approaches combine financial and quantitative non-financial factors to provide several measures or surrogates for “IS value”. Multi-criteria approaches appraise IS investments by means of a score that is based on both financial and non-financial factors. Several approaches in the three classes can incorporate the analysis of risk by means of sensitivity analysis, scenario-techniques or probabilities (cf. Schumann, 1993:171-172).^[15]

Regarding effect-locating approaches, the following four types are distinguished according to their focus. Business objectives-related approaches pursue an alignment of IS objectives to business objectives or to critical success factors in order to ensure effectiveness of the potential IS investment. Thus, investment proposals are assessed according to their impact on the achievement of business goals. Corporate processes-related approaches analyse possible IS investments with respect to their impacts on processes inside a corporation whereas customer-related approaches investigate the potential of IS investments to enhance customers’ processes. Change management-related approaches aim to enable and support a process of change by promoting understanding and commitment of the involved individuals or groups.

[...]

---

^[1] See Alpar (2002:31-41) for a detailed description of each application system.

^[2] ' Such problems are reported, for instance, by Krcmar (2000:113) for a similar software life cycle model.

^[3] ' S.W.I.F.T. (Society for Worldwide Interbank Financial Telecommunication) runs an international network for the electronic data-exchange between banks.

^[4] Risk (incertitude) refers to situations in which decision-makers know (do not know) the probabilities for the occurrence of certain events (cf. Bamberg and Coenenberg, 2002:19).

^[5] The interpretive position assumes, in contrast to the positivistic side, that our knowledge of reality is a social construction of human actors and that value-free data cannot be obtained. Thus, it denies the assumption that hypotheses or theories can be tested by “objective” data (cf. Walsham, 1995:376).

^[6] Stakeholders can be defined as “all those parties who affect or are affected by a corporation’s actions, behavior, and policies” (Jurison, 1996:266).

^[7] Ex-post evaluation is considered here to show the complete life cycle and to indicate links between ex-ante and ex-post evaluation.

^[8] This paper disregards evaluation under certainty due to its extremely small relevance for practical IS evaluation (cf. Perridon and Steiner, 1999:98).

^[9] See Hogarth (1979:165-170) for more sources of judgemental bias.

^[10] See Farbey, Land and Targett (1992:111) and Parker and Benson (1988:250-260) for alternative classifications and examples of IS benefits.

^[11] See for example Mukhopadhyay, Kekre and Kalathur (1995) for a calculation of Chrysler’s cost-savings achieved by the introduction of EDI.

^[12] See Grover, Jeong and Segars (1996:182) for a review and classification of effectiveness indicators suggested in prior work.

^[13] ’ How sustainable such advantage finally is, may be another question. See Mata, Fuerst and Barney (1995) or Clemons and Row (1991) for resource-based analyses on resource conditions for sustained competitive advantage.

^[14] Alternative taxonomies were suggested by Bannister and Remenyi (2000:234-236), Remenyi et al. (2000:79), Powell (1992:30) and Nagel (1990:41).

^[15] See Perridon and Steiner (1999:97-133) for details on these risk analysis techniques.