University of Bielefeld

Approaches to the Ex-ante Evaluation of Information Systems: 
A Critical Review

Diplomarbeit / Diploma Thesis

by

Walter, Sascha G.

 2003

TABLE OF CONTENTS

1. INTRODUCTION ... 1

2. THEMATIC BACKGROUND ... 3
2.1. Information Systems ... 3
2.2. Evaluation ... 8
2.3. Value ... 13

3. CRITICAL REVIEW OF EVALUATION APPROACHES ... 17
3.1. Antecedents to the Review ... 17
3.2. Effect-Assessing Approaches ... 20
3.3. Effect-Locating Approaches ... 33
3.4. Discussion of Usability of Approaches ... 41

4. EVALUATION IN PRACTICE ... 51
4.1. Review of Empirical Studies ... 51
4.2. Interpretation of Empirical Studies ... 55
4.3. Discussion and Future Research ... 57

5. CONCLUSIONS ... 60

APPENDIX ... 61
A. Overview of Evaluation Approaches ... 61
B. Literature Search Strategy ... 64

REFERENCES ... 65

ABSTRACT
This paper critically reviews approaches for the evaluation of investments in information systems prior to their implementation. First, the ground for the review is prepared by examining characteristics of evaluation, information systems and value. A classification of 54 evaluation approaches identified in English and German literature is then presented. Examples of each class are reviewed and their advantages and drawbacks are discussed. Their use in evaluation practice is analysed through the examination of empirical studies and directions for future research are given.

1. INTRODUCTION
Today, a significant share of corporate funds is spent on the implementation, upgrading and maintenance of an information system (IS). Recent studies show that in 2001 the IS budget of companies worldwide accounted for an averaged 8.8% of total corporate revenues (cf. CSC, 2001). Consequently, a thorough evaluation of investments in information systems before, during and after the implementation of a project is important. However, the normative literature reports a great deal of difficulty in the appraisal of these investments (cf. Irani, 2002:11). Although IS evaluation has been an issue for both academics and managers for more than three decades now, there are still serious concerns about how to select projects for investments, how to control the development and how to measure benefits after the implementation (cf. Farbey, 1999:189). This concern has been matched by increased research activity which prevailed through two broad streams. The first stream aimed to directly measure the payoff of IS investments for companies and came to mixed conclusions (cf. Dehning and Richardson, 2002:8). The second stream addressed the question of how IS investments can actually be assessed by decision-makers and particularly focussed on the research of evaluation criteria, evaluation methods and the very nature of the evaluation process (cf. Avgerou, 2000:570).

Of late, several deficiencies in the field of evaluation methods have induced calls for in-depth research. Academics have criticised the current state of the field as being immature and fragmented (cf. Mahmood and Szewczak, 1999:491) and have thus demanded “an overview of the whole panoply of evaluation methods, together with … the assumptions they depend on ...[in order to enable]... the identification of gaps.” (Farbey, Land and Targett, 1999: 205). Another concern is a growing mismatch of theory and practice of IS evaluation (cf. Arribas and Inchusta, 1999:151). Although more than 50 techniques for the evaluation prior to implementation (ex-ante evaluation) have been suggested by the academic literature, managers still draw some of their IS investment decisions on so-called “acts of faith”, i.e. on their intuition and instincts (cf. Renkema and Berghout, 1997:1; Fitzgerald, 1998:16). These are worrying phenomena for proponents of rational decision-making.

Therefore, a critical analysis on the potential of the proposed methods, also with regard to their role in evaluation practice, is desirable. This is the purpose of this thesis which particularly addresses the following research problems:

1. What approaches are available for the ex-ante evaluation of investments in information systems? What are their strengths and weaknesses and to what extent are they usable in practice?
2. What role do evaluation approaches play for practical IS investment appraisal?

Essentially, answers to these questions are sought in a comprehensive review of English and German literature. The practical side of IS evaluation is explored through the analysis of published empirical studies. Thereby, this paper attempts to reflect the state-of-the-art of the literature on IS evaluation approaches. Investments into information systems, instead of entire information systems, are analysed due to the broad definition of such systems in this thesis.

This thesis is organised as follows. In Chapter 2, the ground is prepared through a delimitation and discussion of the central concepts “evaluation” as general activity, “information system” as evaluation object and “value” as key evaluation criterion. In particular, evaluation problems associated with each concept are identified. Chapter 3 presents the criteria for the selection, classification and characterisation of evaluation approaches within this paper. A review of representative approaches is given, their individual advantages and drawbacks are discussed and the usability of all reviewed techniques is examined on a general level. In Chapter 4, empirical studies on IS evaluation in practice are presented, interpreted and discussed with respect to the role of evaluation approaches. This reveals several directions for future research. Finally, conclusions are given in Chapter 5.

2. THEMATIC BACKGROUND
2.1. Information Systems
In the previous chapter, a brief introduction stated and justified the focal point of analysis in this thesis. On this ground, the paper continues with examining the background before which evaluation approaches are applied. Therefore, aspects of information systems, evaluation and value are subsequently discussed and especially problems associated with IS evaluation are identified.

[...]