The Player-Composer. Interactive Video Games as a Medium for Collaborative Composition

Abstract

In this dissertation I explore how players' interaction and engagement with video games affects their musical content. I build principally upon previous research undertaken by Elizabeth Medina-Grey concerning video game music's modular compositional process. I claim that the players of video games, though their vital role in the final realisation of video game scores, act not merely as instruments in the composer's vision but as co­composers in themselves. The dissertation is in three parts. In the first part I will be explicating relevant scholars' work, with some discussion and critique. The second part is devoted to an in-depth case study of the compositional process of the 2014 puzzle game Hexcells, to which I will apply theories and concepts explained in the first part. The final part is an extended discussion based on the findings in part two, ending with conclusions and implications.

INTRODUCTION

Imagine that you are in a modern art museum. As you pass through the museum, you notice a special exhibition confined to a single room off to one side. As you go to open the door into the room, an attendant by the door into the room informs you that you will have to wait a few minutes before entering. They go on to explain that only one visitor can enter the room at a time, and that the room is currently occupied. You wait for your turn. Finally, a light above the door changes colour to green. The attendant opens the door for you and you head inside, with the door closing behind you. You are struck by the size of the room - about 400 square metres in size - which makes the fact only one visitor is allowed in at a time all the more strange to you. Arranged around the room are a variety of artworks, arranged in a seemingly haphazard manner. Separating the artworks, and also oddly placed, are sections of mirrored wall, that make navigating the room slightly confusing, with no natural set path apparent. You notice immediately also that there is a persistent ambient wash of warmly synthesised chords providing a soundtrack to your experience in this room. After taking this all in, you head further into the room, picking a nearby piece of art to consider: an abstract painting. As you walk up to it, you hear that something has changed in the music: a synthesised melody of sorts has begun to play. Once you reach the painting, however, the melody halts, settling on two fading tones. The unobtrusive ambient chords continue to sound as you look at the painting, and eventually you decide to move on to another item. Again, as you navigate around the mirrored walls, a melody begins to play, different this time. As you reach a sculpture, once again the melody comes to a stop, and you take notice this time. A coincidence? It seems unlikely. After finishing looking at the sculpture, you begin your advance to the next, now paying close attention to the music playing through the speakers. Sure enough, as you walk, the melody plays, and when you stop, so does it. You soon realise that the floor tiles are what causes the melody to sound. This continues throughout your time in the room. As you leave you see a large panel with text, and you stop to read it. It is written by the designer of the room, and makes everything very clear. The individual artworks arranged around the room were not themselves important, as they were mainly there to draw the occupant towards them. The confusing arrangement was deliberate, intending to lead each individual visitor to navigate the room in different ways. The background ambience was composed and created by the designer, and the pitches that the floor tiles trigger were chosen by them. Some of the tiles have set pitches that will always play upon triggering, but the majority pick a tone randomly from a carefully chosen set of pitches. The result, then, is that a unique set of pitches will play based on an individual's path through the room. The point of the room, according to the designer, was to make you, the visitor, into a performer - and even perhaps into a composer.

While this scenario is purely hypothetical, and a somewhat idealised analogy for the topics I will be discussing later on, an interview conducted with a visitor to the Victoria and Albert Museum in October 2018 demonstrates that real-life scenarios not too dissimilar to that described above are in fact already occurring:

I went to the Opera exhibition at the V&A with a friend and although we both saw the same exhibition, we actually had completely separate experiences because it was obligatory for the visitors to have a headset that provided the soundtrack for what we were going to see. This meant that we could move through the exhibition at our own pace from one room or zone into another — which in this case was a chronological exposition of the development of opera from early to late to later. As we moved from one area to the other and the soundtrack played arias from operas of different eras, there were times in the transitional areas where one aria morphed into another. But by taking a step forward or a step back, or changing the direction I was facing, it would flip the arias back and forth. Proceeding through the exhibition, looking around it was clear that other people were exploring their own sound world whilst reading information boards and looking at video. By the end when I met up with my friend, we had both experienced the same exhibition but we had each had a completely unique version of it because of the choices that we had made as we were moving through in terms of position and pace and direction chosen. (Moseley-Gore, 2019)

According to the interviewee's account of their visit to the V&A, every visitor's auditory experience of the opera exhibition is fundamentally different from any other visitor's. No one visitor will take the same amount of time to go through the exhibition as any other, just as no one will be positioned identically during the entirety of their time within it. Ultimately, it could be said that the visitor has contributed to the creation of their own musical experience.

Similarly, my own experience at another V&A exhibition, this time dedicated to the progressive rock band Pink Floyd, presents another contrasting, yet no less intriguing example of this phenomenon. At a certain point within the exhibition, visitors would come across a small mixing desk, with attached headphones that they were encouraged to put on. With these headphones over their ears, visitors heard what would doubtless be for many attendees a very familiar piece of music: the track 'Money', from the band's album Dark Side of the Moon. However, this was not 'Money' in a form most are familiar with: something was missing. In my case, the drums were extremely faint. I soon realised that the mixing desk was not merely on display, but it was meant to be interacted with. Using the faders corresponded directly with raising or lowering the volume of each of the 10 or so tracks present within the song, allowing the visitor to gain an appreciation for the way that the song was composed and arranged. An alternative interpretation, however, might be that this mixing desk served to provide visitors with the opportunity to test out their own versions of the track, or to interpret it in their own way. In theory, of course, Money exists in a definitive form: the album recording. Interacting with this mixing desk, for example by reducing the volume of the drums, felt like it had a significant impact on the the track, changing it from Money to an alternate version of Money. In other words, this was not merely a demonstrative tool, but also a creative outlet, whether or not the user was aware of this or not. Each individual visitor's experience using the mixing desk provided them with their own unique version of the track.

These scenarios are useful because they present us with real-world demonstrations of phenomena that have been taking place within the world of video games for many years. Of course for video game music to be successful, it must be adaptable. Unlike in linear entertainment media such television and film, events on­screen within a video game are highly changeable, and so for the music to remain appropriate, it must be similarly flexible. This is what has been referred to by scholars as 'dynamic' music, and it has become the norm for video game scoring (Collins 2008: 184). This is generally achieved through the use of a game 'engine', which is a type of computer program that is designed to control and arrange the parts of a game's world and in-game environment. Music engines specifically exist to deal with a game's musical content, and are designed to fit the in-game score to what the player is doing within the game, accessing and playing back audio files in a way that always remains suitable. This helps to create a sense of musical interactivity, which 'establishes a direct connection' between in-game actions and soundtrack (Van Elferen 2016: 37). So what then is the definitive version of a video game's score? As in the case of the examples given above, there can never be such a version, as the dynamic nature of such experiences generates a large variance of musical scores. The existence of 'original soundtrack' albums for video games, that feature a set of fixed tracks, solidifies this truth. These albums feature prearranged versions of a game's music, either representing amalgamations of various segments in an approximation of a game's 'playthrough' in music, or merely showcasing the original recordings from which segments are taken for use within the game. These versions may be structurally and aesthetically satisfying, but they are not true representations of the game score, as, out of necessity, they have lost their authentic individual nature. In other words, music in video game soundtrack albums 'ceases to be video game music and becomes music from a video game' (Reale 2014: 77).

Music engines allow the music to be supremely dynamic and moment-to- moment responsive, with the potential for it to change instantaneously upon changes within the game. The responsibility for each person's experience of the game's music is placed in their hands, usually in such a way that they will not consciously focus on it, or even be aware of it happening. Composers, writers and programmers work together to ensure that as players interact with and explore an in-game world, the engine matches these actions up with appropriate musical content, such as scoring combat sequences with fast, energetic music, or quiet, reflective explorative sequences with calm and gentle music. Of course, an individual's 'playstyle' - the unique ways in which each player makes decisions - has a crucial role in the eventual output of the music. For example, different players will find certain tasks easier or more difficult, and some players may choose to take things more slowly or faster than others. The reasons for this vary tremendously. Some players may simply have more experience playing certain types of video games than others, and some players may simply have less time in which they can play. Additionally, playstyle can be a reflection of players' personalities, as in the case of patience and aggression. The results of this are fascinating. For example, a faster player may not hear as much of the possible musical content of a game, or, just as they miss a visual clue to solving a problem, they may miss an auditory or musical clue to solving one. Some ludomuscicologists in the last decade or so have discussed many of these phenomena, creating and applying novel techniques and methods to rise to the task of discussing and analysing an area of music that has for so many years been largely understudied. Much has been said concerning the unique role that video game players play in the formation of their own experience of the score, and the individual nature of what is produced. Within this dissertation, I aim to establish that the role that the player takes goes beyond merely participating in the creation of the score, but that in many cases they can be seen as becoming a co-composer of the musical results of their own playthrough. During the first part of the dissertation, I will be providing an overview of and discussing various crucial pieces of literature, which provide the context for my research and establish a theoretical framework for the second and third parts. The second part of the dissertation consists of an in-depth case study and analysis, during which I will be applying some theories and ideas of the scholars I introduced in the first part. For this, I will be examining in detail the 2014 minimalist puzzle game Hexcells, which I believe provides a window into the subject, presenting us with many fascinating implications. I will then use the analysis in part two as a springboard for my own conclusions, which come in the form of a final discussion in part three.

PART ONE: THEORY

Musicologist Tim Summers, in a chapter from the highly influential 2016-edited volume Ludomusicology: Approaches to Video Game Music, recognises the discipline's relative youth, saying that it is still undergoing an 'exciting period of growth' (Summers 2016: 28). The result of this, however, is that it is a field of study that is constantly attempting to find its feet, without a solid canon of writings on which to build. This book is the product of this recognition, and is an attempt to contribute to this groundwork. Summers notes that it is crucial that those working within ludomuscology work towards agreeing upon methodology and technique when approaching their work, and it is with this aim that he writes his chapter. The chapter itself is titled simply 'Analysing Video Game Music: Sources, Methods, and a Case Study' and is concerned therefore with presenting researchers with a wide palette of analytical routes with which one can approach video game music. As part of this overview, I will be addressing some methodological concerns for the case study in the second part of this dissertation.

The first of these analytical routes provided is what Summers refers to as 'analytical play' (Summers 2016: 10). He points out that an essential part of the process of analysing video game music is the actual playing. The analyst must play the games, thus occupying a dual role as a player and as a ludomusicologist in the process. Summers also discusses here the choice to play his case study game on a laptop computer, listening to the audio through headphones. I will be doing the same, for the same reasons that he provides. Playing games on a computer as opposed to a games console allows for far easier capture of audio and video footage of the game, using screen recording and audio recording software. For my purposes, I will be using MacOS's built- in Quicktime Player, which features its own screen recording tool. However, as it does not by default record the audio as well as the video, I have used another application called Soundflower, which allows me to record screen captures complete with audio. Summers explains the use for such capturing by pointing out that it allows the analyst to separate the tasks of playing the game and analysing it. Indeed, this would most often prove to be a hopeless task - video games are rarely so simple that one can split the mind during play. During his discussion of his own analytical play of Halo: Combat Evolved (2001), Summers raises some important points of consideration: firstly, as has already been discussed, that video game music acts in a dynamic manner depending on the player's specific in-game decision-making, and also that these decisions can vary level duration wildly. What is meant by this is that depending on how a player chooses to navigate through a 'level' - a clearly-defined section of gameplay, usually with a single overall goal - how long it takes before they reach the end will change accordingly. However, importantly, he introduces the idea of playing 'against the grain' (Summers 2016: 12). Game design involves anticipating a player's thought processes and steering them towards various goals. This usually means that during an average individual's first playthrough of a video game, they will tend to stick roughly to what the game's programmers expect them to do, as they are being manipulated to do so. However, it is always possible to try, as best one can, to play in an unexpected manner, for example by deliberately ignoring directions, and even trying to escape the constraints of the game's world. Summers notes that this can be a useful means of determining how a game is constructed, as often game designers will not have accounted for every possible scenario. The game I will be exploring in my case study, Hexcells, is a simple example, but there are still opportunities to play against the grain in this way. Most importantly, analytical play 'reveals the musical mechanics and material of the game', and I will be employing this later on. Following this in Summer's chapter, there is a short discussion on use of video games' 'engines' and the code therein as tools for analysis, although much of what is said in the chapter concerning how music is handled within these engines is covered in far more detail in Elizabeth Medina-Grey's investigation of musical modularity, which I will be examining shortly.

The final part of Summers' chapter that is relevant to this dissertation is how to best approach representation of musical material found within a video game. He notes here that transcription is a problematic task when looking at video game music, as its dynamic nature means that no definitive single version of a game's score exists. Therefore, theoretically, there can be no definitive written score. However, he does agree that transcription can be a useful tool, dependant on the circumstance. He specifically mentions how transcribing excerpts that 'form part of the modular construction of the audio output' has the potential to be informative. However, Steven Beverburg Reale, writing in Music in Video Games: Studying Play, disagrees. The difficulties in representing video game music highlighted previously are such that he believes that in video games, the game world itself should be seen as the score. Instead of stave notation on the page, Reale proposes that the 'levels are abstracted notation' in themselves (Reale 2014: 97). While this approach has some merit, it has far more drawbacks. It could be argued that the game itself acting as the score is of limited use, in that while much of the music is triggered by events and actions within the game world, there may be long periods of time where nothing within the game world is having any impact on the game's musical score, such as during exploration in a vast open environment, where the player may choose to simply walk around aimlessly. In addition to this, it simply cannot provide the analyst with nearly enough useful information, and will always be hard or even impossible to understand without intimate knowledge of the video game's mechanics and rules, something that cannot be demanded of a reader. Stave notation is universally understood amongst musicologists, and has the potential to be expanded upon and adjusted to fit any kind of special requirements that a video game score may require. It is for this reason that I will be employing the use of transcription in the form of stave notation in my analysis of Hexcells in the second part of this dissertation.

Elizabeth Medina-Grey, writing within the same volume, provides much of the theoretical framework on which I will base my work within this dissertation. The chapter itself is titled 'Modularity in Video Game Music' and is also primarily concerned with providing an analytical framework for the highly dynamic and changeable medium of video game music, whilst also situating it as part of a lineage within the broader art music realm of 'modular' music. Medina-Grey claims that techniques associated previously with composers of avant-garde music during the 20th century have been and are being applied to video game scores as well, specifically referring to the interrelated areas of 'indeterminate', 'open form', and aleatoric musics. The most important of these concepts is 'open' music. What Medina-Grey means by this is based on the work of

Umberto Eco, who defined such music as being those cases where someone besides the composer in some way takes part in the final form of the work, being granted the opportunity to take part in a 'theoretical, mental collaboration' with it (Eco 1989: 12). Most relevant to this discussion is the idea of 'works in movement'. These works, or pieces, are not set in stone, and do not have a clearly laid out structure or order. Instead, structural units of various levels of musical 'completeness' are laid out for a third party to arrange for themselves. Consequently the final arrangement is changeable, and for an audience, unpredictable, and is highly unlikely to be repeated exactly from one performance of the work to another. A prominent, important example is Karlheinz Stockhausen's Klavierstück XI, the score of which consists of 19 unordered segments of music. Each one of these also contains information pertaining to how the pianist must perform the next segment, with tempo, articulation and dynamics all indicated following the stave. Most importantly however, the choice of which segment to play first, and which segment to play after that (and so on), is left to the performer to make at random. The piece can only finish once a single segment has been sounded three times, at which point the performance comes to a close. Considering that each segment can have 19 different versions, based on instructions for the previous segment played, and that the maximum number of segments that could theoretically be played in one performance is 39, there are a great deal of possible performances of Klavierstück XI. Evidently, a result of this is that there can be no definitive example of the work. KlavierstückXI exists as all of its permutations.

This kind of 'work in movement' has more recently come to be known as 'modular' in construction. Medina-Grey highlights the work of composer and musicologist James Saunders, whose 2008 article Modular Music provides us with some useful groundwork for examining such music. He states that modular music must contain first a set of 'standardised units', or modules, and also a 'procedure for fitting them together': effectively, a set of rules for the arrangement (Saunders 2008: 153). In the example of Klavierstück XI, for instance, the modules are the 19 individual segments of music, and the rules are a combination of the instructions provided at the start of the score, and also those specific performance directions printed following each musical segment. Both are required for the piece to be seen as a modular work. Without the rules, one would simply play through the segments in the order that they are printed on the page, and end upon playing the 19th. The combination of modules and rules, then, allows a process of assembly to occur, where within the restrictions of the rules an individual chooses the arrangement of the modules. The final step of this modular process, of course, is the actual sounding object: the finished product. This three-step process, from module and rule creation to assembly to the actual production of sound, is fundamental in the analysis and understanding of modular music, and is displayed by Medina-Grey in a diagram, which I have reproduced below in ex. 1:

Abbildung in dieser Leseprobe nicht enthalten

Ex. 1: The three-step modular process, as represented by Elizabeth Medina-Grey

Someone must cause each of these steps to occur in order for a piece of modular music to come about. Someone, or something: while in the majority of cases a human being will take on the task of enacting each step, it is important, especially in this current age, to acknowledge the possibility of machine involvement in the form of computer programs and artificial intelligence. Medina-Grey sets out in the chapter the three different types of participants that correlate to the steps above: 'Creators' compose the modules themselves, and devise the rules by which they must be arranged; 'Assemblers' decide when and how the modules sound in light of these rules; and 'Producers' make the final sounding object. Each one of these roles must be fulfilled for the process to occur, but it is important to note that they do not necessarily have to be fulfilled by different individuals. For example, the creator of Klavierstück XI is the composer, Karlheinz Stockhausen. The assembler and the producer roles, however, will always be fulfilled by the same individual: the pianist performer of the piece. But of course, these roles could also theoretically be taken on by Stockhausen himself, making him be creator, assembler, and producer all at once. This stands in contrast to traditional determinate composition, where the composer will always act as both creator and assembler. It is also worth noting that all three roles can be fulfilled by multiple individuals. Medina-Grey here raises another central issue with this topic: the question of how far the assembler can be said to hold some authorship when someone besides the composer has higher levels of involvement in the final sounding product. Janet Murray, whilst admitting that the assembler does take part in 'artistic creation' through this process, nevertheless labels this as 'agency', and not authorship (Murray 1997: 152-3). Medina-Grey seems inclined to agree, also admitting that the assembler may have a significant impact on the final product, but concluding that due to the rules that they must operate within, authorship 'belongs overall to the system's creators.' (Medina- Grey 2016: 56). However, this is something that I believe should not be so strictly stated, as I will be arguing later on.

Ultimately, the purpose of Medina-Grey's chapter is to link these ideas to video game composition, and the latter part of the chapter is occupied with this. Medina-Grey states that music in video games shares much of its compositional process with that of the art music discussed previously. Indeed, the majority of what has already been said here can be applied directly to video game music. Medina-Grey stresses that these similarities have come about due to 'different practical and aesthetic goals', and although she does not state the difference here, it is plain to see what these are (Medina-Grey 2016: 54). Simply put, modularity within video games is first and foremost a practical necessity, rather than an aesthetic choice. Arguably, in the world of art music, the opposite is true, with modular music serving little practical purpose beyond that of the audience's aesthetic appreciation. Video game music belongs to a larger whole, which is an important distinction when considering how its composition is approached. Naturally, as a result of this, despite such similarity, there are many differences that set it apart. In art music the performers, as assemblers, fulfil their role with the final sounding object at the forefront of their mind, as it is in fact their only goal. Decision­making is entirely musically motivated. In video game music, the assembler's goals are those which the video game demands, and they can involve a huge range of in-game tasks. In some cases, in 'music games' such as the Rock Band and Guitar Hero series, musical choice is still the primary motivator. However, in the majority of cases, it is not. Modular assembly takes place instead as a secondary result of the in-game situational decision-making of the player, whether that be fighting monsters, diving underwater, engaging in conversations, and so on. This is what Medina-Grey refers to as 'non-musical choice' (Medina-Grey 2016: 61).

[...]