Virtual Balls: Interactive Games for Theatre Audiences

TABLE OF CONTENTS

Abstract

1. Introduction
1.1 Motivation
1.2 Aims
1.3 Intention

2. Fundamentals
2.1 Fundamentals of computer graphics
2.2 Fundamentals for 2D motion detection
2.3 Fundamentals of depth extraction from images
2.4 Mathematical fundamentals
2.5 Current state of the art

3. Game design and development
3.1 Requirements analysis
3.2 Modelling and calibrating the scene
3.3 Detecting and processing motion
3.4 Creating the game
3.5 Real life experiences

4. Conclusion
4.1 Summarisation
4.2 Future prospects
4.3 Evaluation
4.4 Personal prospects

Sources

Abstract

How do you design an interactive game for more than a hundred people sitting in a theatre?

During my twenty-week internship project as visiting researcher at the Computer Science Department of the University of Otago I was involved in the development of an on motion recognition basing interactive virtual ballgame for a crowd of people sitting in a theatre (→ Figure 0-1 on p.5).

My tasks were remodelling the theatre as a virtual 3D scene, calibration of the scene geometry for different theatres, implementation of motion detection and the extraction of three-dimensional data from it. I was also engaged in designing the ballgame (appearance, sub games, rules, sound).

The game can roughly be described as follows: the players in the seat rows of the theatre see themselves displayed on a silver screen (as if looking in a mirror), as well as a couple of virtual balls that appear to be slowly bouncing down the seat plane. Yet whenever a ball strikes a moving person it acts like a real ball and bounces away. Hence people can play with the balls by hitting them in the designated direction.

The instruction for developing the ballgame was given by the chairman of a computer graphics conference. Prof. Wyvill wished to welcome and entertain the guests of the electronic theatre (an event at the conference) while they arrive and take a seat. Since already a small motion-detecting programme with squares moving in two dimensions (vertically and horizontally) existed, Prof. Wyvill assigned a further student and me to enhance the idea to ball objects behaving as if moving in all three dimensions.

After a very successful presentation at the conference we were ordered to advance the developed interactive ballgame to present and test it at a few future theatre events.

Generally the game we created turned out to be very successful in terms of entertainment and fun, as well as easily and intuitively playable.

In particular the remodelling of the theatre as virtual scene emerged as useful realisation of the game and the calibration of the scene geometry allowed an appliance of the game to other theatres. Also the used motion-detection method worked out very well and the extraction of 3D data resulted in expedient information to establish a realistic behaviour of the balls.

In this report I will portray in detail the development of the interactive ballgame, its implementation and the achieved results.

CHAPTER 1:
Introduction

In this chapter I will explain in which subject area our game project is resident, which motivation leaded to it, which aims were set, how I intend to reach these and outline my personal tasks.

Overview:

1.1 Motivation

1.2 Aims

1.3 Intention

1.1 Motivation

Before the beginning of the project Simon McCallum^[1] had developed a two- dimensional interactive game, allowing one to two players to watch themselves on a screen (as if looking in a mirror) together with a number of floating coloured squares (→ Figure 1-1). His programme detects their motion and uses a web camera to capture it.

Thereupon it calculates if any of the found motion has been close to the squares displayed on the screen. If so, moving direction and moving velocity of these squares will be changed by the programme in a way that the players percept their own motion as influencing the behaviour of the squares.

illustration not visible in this excerpt

Figure 1-1

Screenshot of Simon McCallum‟s game

© Simon McCallum 2006

A few of weeks before the Graphite 2005 conference^[2] took place Prof. Geoff Wyvill^[3] was in need of an entertainment programme for the guests of the electronic theatre being part of the conference. These guests would be sitting in a theatre-like room, comparable to a cinema.

He knew about Simon McCallum‟s game and was inspired by it. Therefore he assigned Kerstin Kellermann^[4] and me with the development of a version of Simon McCallum‟s game, having balls as play objects instead of squares. These balls should appear to be moving not only in height and width dimensions as the squares did but also in depth^[5] and be playable in a context of differing sub games with different rules and kinds of balls.

1.2 Aims

Overall aim of the project was the development of an interactive ballgame for a crowd of people sitting in a theatre-like locality. We wanted this game to be very entertaining – the players should enjoy playing with the balls.

It was also important to design the game as intuitively playable since the game already runs when the first persons enter the auditorium and no explanations how the game is to be played will be provided.

Yet another ambition was to design the diverse sub games in a way that these appear not too difficult but sensible. The goal of each sub game should be obvious to the players without many explanations.

Most important aim was to achieve as much realism as possible: the balls should look like real balls and behave like real balls.

To reach these aims we planned to continuously test our game and its latest changes (so called “hot testing”) on its current fun factor and its playability (in terms of sense and difficulty of rules and sub-game goals), and furthermore to interview the players for criticism and suggestions for improvements.

My personal objectives generally deal with the recreation of the theatre locality as virtual, three-dimensional scene within the game, the adjustment of this scene depending on different theatre shapes and especially with the more complex and more difficult task of detecting motion in the continuous images provided by the web camera capturing the players and the conversion of this motion information into data which will be used to change the behaviour of the displayed balls.

I wanted the scene “copy” of the theatre to be accurate and at the same time as simple as possible. It had to represent the real theatre locality as a small virtual version in order to create a “world” for the balls to “exist” in.

To achieve this aim I decided to examine which geometry factors would be minimally necessary and to use scaling that can visibly be checked for accuracy.

Almost the same ambitions I had for the calibration of this virtual scene: not needlessly complex, but still appropriate. Also I designated to enable the calibration act (the input of the different geometry values) to be easily accomplishable.

To realise an elegant, simple calibration I would have to find out the smallest possible number of measurable geometry variables^[6] in order to maximize the values that will be computed internally^[7]. To easily input the measured values we decided to add those to a widget (which also serves for other setting opportunities). Furthermore a possibility to store the calibration values would be necessary to prevent the calibration from being laborious.

As for the more complicated motion detection part I intended the searching process to be a simple and fast computation. It also had to be sufficiently correct to provide information about where exactly a ball would be affected.

To put this in practice we decided to use a simple motion-detection technique. To make it fast I planned to examine possibilities to limit the areas of motion detection and thereby shorten the computing time.

Yet the most challenging aim was to convert the information about motion into appropriate three-dimensional data having the purpose to determine the behaviour of the balls when “hit“. This influence had to be accurate enough to create the illusion of almost playing with a real ball – a ball behaving as expected by the player according to his/ her interaction with it.

To achieve this appropriateness I would “tinker” with the gained data and afterwards test the results by playing the game and studying the changes in the behaviour of the balls until it would fulfil our demands.

1.3 Intention

The overall intention of the project of my partner and me can be summarised in the following three parts: Design, Implementation and Test of a virtual interactive ballgame basing on real-time motion recognition.

The Design of the ballgame consisted mainly in the conceptual generation of a number of different sub games containing differing rules. They should have different goals, different kind of balls (in appearance and behaviour) and differing possibilities to play with them by movement.

As for the implementation part we had to put the design in practise by transferring the tasks gained from a requirements analysis into a software architecture and arrange them in modules. In order to lay a fundament to build up on a suitable computer graphics method had to be chosen that is responsible for displaying the game on a screen. Also suitable algorithms had to be explored to implement the given tasks.

Last part of the overall intention was the testing process which describes the presentation of the game and has the purpose of estimating the quality of the developed game. Originally only one real-life test run was scheduled but due to a following great positive feedback a proceeding with the game was initiated and we were able to present it four more times – each with an improved, advanced version of the ballgame.

Since it is not possible to evaluate a game mathematically interviews with players took place. They judged the playability and appearance of the ballgame and provided improvement suggestions; the running stability and other programme features we were able to determine ourselves.

We also recorded the game a couple of times to later be able to watch and examine it in order to meliorate it.

My personal intentions included four main parts: recreation the real theatre as a three- dimensional virtual scene, development of a way to calibrate the virtual scene according to difference theatre sizes, and particularly the detection of relevant occurring motion and extracting data about the effect on the balls from it.

It follows a short description of each part.

Firstly modelling the presentation room means to rebuild all important geometrical features of it as a small virtual version (→ Figure 1-2 on p.12).

For this purpose I planned to use 3D computer graphics providing opportunities to model a 3D scene and transform it into a 2D image in order to create a graphical user interface for the game.

It also contains the creation of the balls and their appearance as well as the embedding of the video stream from the capturing camera into the virtual scene.

illustration not visible in this excerpt

Figure 1-2

Typical shape of a theatre showing the view of a player

© cinetech 2005

Since dimensions vary from theatre to theatre I had to examine ways to calibrate the virtual scene with changing presentation localities. I had to develop a way to let the programme recalculate the scene basing only on a few number of variables that have to be measured by hand for every theatre once and handed over to the programme. These heights-, widths- and other necessary variables and the recalculation of the scene had to be stored to be available with every new start of the programme.

To ease the interaction with the programme these values had to be fed into the programme outside the programme code via a widget that is accessible during the running game and which I had to construct.

The third intention was to the find motion in the video stream provided by the web camera that records the seat plane with the crowd. I only want to search for “interesting” motion. I therefore had to which motion is relevant and for which reasons motion can be discarded. Furthermore it was essential to find out the three-dimensional position of the detected motion.

Last and most challenging of my personal intentions was to extract data from the detected motion and convert it so that is suitable to affect the usual behaviour of the balls (when there is no contact with a player). Here I had to take in account the implementation of the ball physics ^[8]. To this procedure also belonged to determine the intensity of motion and its direction. At last I had to examine and test different possibilities to manipulate the resulting data for the purpose of affecting the balls‟ behaviour in a realistically appearing way.

CHAPTER 2:
Fundamentals

Within this chapter I will present the general and specific basics of the subject area necessary to understand my motivations and my work described later on in chapter 3.

Also an insight in the current state of the art will be given.

Overview:

2.1 Fundamentals of computer graphics

2.2 Fundamentals of 2D motion detection

2.3 Fundamentals of depth extraction from images

2.4 Mathematical fundamentals

2.5 Current state of the art

2.1 Fundamentals of computer graphics

illustration not visible in this excerpt

Table 2-1

Major realms of computer graphics

Though the computation of the two 2D approaches is cheaper and faster the 3D computer graphics are more suitable for us because we want our balls to look three- dimensionally and photo-realistically.

Yet the 3d voxel graphics are not suitable to represent simple balls and cannot easily be used to apply the physical laws of real balls. Therefore we chose to implement the game with 3D vector graphics. Also the available number of open source libraries is very useful and helpful for our ambitions.

Its basics are described in the following:

a) Modelling of ballgame scene
b) Rendering of ballgame scene

a) Modelling of ballgame scene

Scenes are mostly modelled by hand such as ours.

The ballgame scene contains the following components: spheres (as balls), planes (e.g. for displaying the video stream), a few light sources (representing the light in the theatre coming from above) and a virtual camera in the centre of the scene coordinate-system. Furthermore I used textures to “dress” objects with the look of real-world objects (e.g. the texture of a beach ball).

A more detailed description of modelling the scene will be given in paragraph 3.2.

b) Rendering of ballgame scene

The ballgame scene with its three-dimensional content has to be rendered to generate an image of it that will be displayed to the players.

The rendering process is unlike the modelling done automatically.

Within the rendering process the suitable lighting model I configured is taken in account, the position of the clipping planes and of the virtual camera that I set as well as the chosen perspective projection-type^[9] and the shading^[10] features I defined.

The rendering is a projection procedure that transforms the three-dimensional points of the scene into a coordinate system of two dimensions (the displayed image).

Thereby run projection rays which start from a projection centre through every point of the object and intersect the projection plane. The points on the projection plane represent the resulting picture.

This correlation will represent a very important matter in paragraph 3.3.

illustration not visible in this excerpt

Figure 2-5

The GL frustum (viewing volume)

© D. Easterly

illustration not visible in this excerpt

Figure 2-6

The resulting raster graphic, rendered from a 3D scene

© D. Easterly

The sources [08] [09} [10] [11] and [34] [35] [36] will provide a range of detailed information about modelling and rendering 3D vector graphics.

2.2 Fundamentals of 2D motion detection

illustration not visible in this excerpt^[11],^[12]

Table 2-2

Some potential methods to detect motion in images

The most suitable method for our purposes is the image subtraction because it has no disadvantages but only advantages according to our requirements and suffices our demands.

Therefore I will now elucidate the necessary fundamentals difference-imaging.

a) Difference imaging generally
b) Difference imaging as indicator for motion

a) Difference imaging generally

Difference Imaging – also known as image subtraction or pixel subtraction - is an image processing technique that allows determining changes between two images.

The difference between the pixel of the one image and the pixel of the other image at the same position is calculated by subtraction their pixel values (done for all pixels) (→ Figure 2-7 and Figure 2-8). The result is used to generate a new image – called difference image – to make the differences between two images visible.

Subtracted identical pixels appear as black (pixel value 0), subtracted non-identical pixels as colour ≠ black (pixel value ≠ 0) in the resulting absolute difference image (→ Figure 2-9 to Figure 2-11).

illustration not visible in this excerpt

Figure 2-7

Pixel subtraction

© 2003 R. Fisher, S. Perkins, A. Walker and E. Wolfart

illustration not visible in this excerpt

Figure 2-8

Calculation of absolute difference

© 2003 R. Fisher, S. Perkins, A. Walker and E. Wolfart

illustration not visible in this excerpt

Figure 2-9 © HouCiel

Grey-value image with the numerical values of the pixels

illustration not visible in this excerpt

Figure 2-10 © HouCiel

Another grey-value image

illustration not visible in this excerpt

Figure 2-11

Difference image resulting from a subtraction of Figure 2-9 and Figure 2-10

© HouCiel

b) Difference imaging as indicator for motion

The image subtraction algorithm is suitable for recognizing a moving object.

For this purpose the region of interest is captured by a digital camera which provides a video stream consisting of a digital image sequence.

Reason for the indication of motion is the assumption that the colour value of a pixel in two consecutive images does not change if the object at the pixel position did not move during the recording of the two frames and the fact that changes in pixel values can be interpreted as motion.

No colour (black) in the difference image represents motion, colour indicates motion^[13].

illustration not visible in this excerpt

Figure 2-12

Digital image of a collection of screws and bolts

© 2003 R. Fisher, S. Perkins, A. Walker and E. Wolfart

illustration not visible in this excerpt

Figure 2-13

Digital image of the same scene but with two differences

© 2003 R. Fisher, S. Perkins, A. Walker and E. Wolfart

illustration not visible in this excerpt

Figure 2-14

Difference image (contrast-stretched) resulting from the subtraction of Figure 2-12 and Figure 2-13;

(only the two moved pieces are visible)

© 2003 R. Fisher, S. Perkins, A. Walker and E. Wolfart

Also inevitable background noise must be mentioned since it will be interpreted as false motion. When a digital camera digitalises an image it carries on disturbances caused by vibration of camera or environment, lighting fluctuations, signal amplifying and other factors. Using a threshold to reduce the acceptance of every difference value helps relieving this problem.

[...]

---

^[1] Artificial Intelligence Laboratory, Department of Computer Science, University of Otago

^[2] Graphite 2005: 3rd International Conference on Computer Graphics and Interactive Techniques in Australasia and Southeast Asia, 29 November - 2 December 2005, Dunedin, New Zealand; http://www.cs.otago.ac.nz/graphite/

^[3] Prof. Geoff Wyvill: Graphics and Vision Research Laboratory, Department of Computer Science, University of Otago; http://www.cs.otago.ac.nz/graphics/

^[4] Kerstin Kellermann: further student of Computational Visualistics at the Otto-von-Guericke- University Magdeburg searching for an internship project at the Department of Computer Science of the University of Otago

^[5] Depth movement: forward in terms of downwards and backwards in terms of upwards the seat plane

^[6] Measurable variables: e.g. height of theatre, height of web camera, length of seat plane

^[7] Non-measurable values: e.g. depth of theatre, ascent of seat plane, height of seat plane

^[8] The complete ball physics containing velocity and direction change calculations, interaction with its environment and rotation were developed by Kerstin Kellermann, my project partner. The ball physics are expatiated in her report Designing ballgames for virtual auditoriums: responding to player expectations.

^[9] For the perspective projection the rays are not parallel and the projection centre is in visible distance. The projection centre of an orthographic projection lies in unlimited distance and therefore the rays are parallel (orthographic). Since the perspective projection represents the way of human perception we used this setting for the rendering

^[10] The simulation of surface characteristics of objects is called Shading.

^[11] The XOR (eXclusive OR) function is only true if just one (and only one) of the input values is true, and false otherwise.

^[12] Fourier-transformation: integral-transformation calculating the frequency spectrum of a signal

^[13] Admittedly motion always causes pixel value changes but in reverse changed pixel values are not necessarily caused by motion. Light-source changes or camera shaking produce pixel value changes falsely interpretable as motion and should be avoided.