The Appropriateness of the Keyboard-Visual Display

Contents

1 Introduction

2 Interaction Modalities
2.1 Usability of input devices
2.2 Interaction in pervasive contexts

3 Classes of interaction
3.1 Entertainment
3.2 Communication and messaging
3.3 Social interaction

4 Pervasive environment interaction
4.1 Pheremone based interaction
4.2 Real-world environment

5 Conclusion

1 Introduction

The combination of keyboard and screen is arguably the most well-established I/O model in computing but it is no longer the most ubiquitous. In the UK alone, there were 81.6 million subscriptions to a mobile phone in the last quarter of 2011 [?]. While some of the latest mobile devices can be connected to a hardware input device (such as a keyboard), this affects the mobility of the device as well as it’s ability to act as a peripheral- free computing device. Therefore, with mobile technology there is a great need for sophisticated user interaction models. This essay tackles the question of whether or not the ‘keyboard & visual screen’ combination is appropriate in the worlds of mobile and pervasive computing.

The essay begins with discussion of the various types of computer input and how they affect user experience. It then goes on to describe different classes of interaction, and how the input device can influence the overall user experience. Different deployment environments are described as the essay addresses issues in the pervasive computing paradigm in both a mobile and im-mobile context. In each section, various solutions that have appeared in the literature over the years are presented and commented upon. The author’s own opinions on the solutions are discussed. The essay concludes with the author’s perception of the appropriateness of the keyboard and screen in ubiquitous computing paradigms.

2 Interaction Modalities

Ever since the mass adoption of the personal computer in the 1980’s, there has been extensive research into how humans can interact with such devices. Over the years many different hardware peripherals were implemented to tackle this problem. The most universally adopted input device is the keyboard. The digital keyboard was adopted from the early typewriter as the primary device after the punch card system in the early days of industrial computing. For decades, it remained the only input until the mouse came to light [?]. With the advent of this new input mechanism, the realm of computing was moving towards a more user-accessible context.

Today, there exists a variety of devices, each with their own characteristics such as usability, portability, input dimensionality (the size of the ‘input-space’) and the technology required to leverage such varying modality. In the mobile realm, there is the need for more ‘effective’ ways to interact with devices. As stated previously, it may be undesirable to use a traditional keyboard with a tablet or a mobile phone.

In the following subsections, various interaction issues and contexts that manifest in mobile and pervasive computing are discussed, arguing the appropriateness of the physical keyboard and visual display seperately as well as their combination.

2.1 Usability of input devices

A device’s usability is a very important characteristic. As mentioned in [?], adapting desktop keyboard-mouse interaction to a mobile context does not fit well. There are usability issues that need to be resolved when considering any form of input device.

Initially, mobile devices followed the traditional route of standard ‘physical key’ based input. Each was equipped with a set of hard buttons with which the user interacts with the device. Modern mobile technology has advanced since then and new ways to provide input to the device have been developed. Efforts have been made in the creation of virtual keyboards such as [?]. However, projected keyboards are incapable of providing tactile feedback; an inherent characterisitc of a physical keyboard. The touch-screen offers the haptic feedback, simulating the presence of a physical button. It also can come in many forms, some of which provide a virtual keyboard. Virtual keyboards can provide tactile feedback in the form of actuators behind the touch interface, reacting to user input. There are usability issues with virtual keyboards on small handheld devices however. Studies show that some user groups, namely the elderly, have complained about the small size of some mobile devices [?].

Another modality of the touch screen is gesture-based input. This form has also been used in combating usability issues associated with users suffering from hand injuries. Mankoff et. al used a pen-based interface for input of ASCII characters in the desktop environment[?]. [?] used multi-touch screen interfaces coupled with audio display to enhance the usability of mobile devices for the blind. They comment on how the touch screen offers advantages over physical buttons; mainly the ability to directly manipulate items on the screen through gestures. Kane et. al attempted to extend this with the addition of audio feedback. Their design proved to be more error prone than a physical button alternative. The authors attribute this to unfamiliarity with the interface. They conjecture that, with more time, users would become accustomed to the touch screen interface and perform better than the study results. Perhaps the fact that almost all smartphones today adopt a ‘multi-touch’ interface for their screens is evidence of the validity of this conjecture.

2.2 Interaction in pervasive contexts

An example of a pervasive environment is the standard setup of a public visual display. Public displays, be it visual, audio or tactile, allow an augmented experience of various environments. For example, in a public garden park, there may be the advent of an audio guide, which users can wirelessly interact through their mobile device. This has been implemented in [?], where a pervasive installation in a park provided interactivity with the surroundings of the park. Participants of the study stated that they would have ‘preferred more audio than text’. The user group surveyed clearly preferred the audio display over the visual. This is further evidence for the unappropriateness of visual displays in pervasive contexts.

Other pervasive environments include visual public displays. If these were coupled with a physical keyboard, the scalability of the interaction would be greatly limited my the number of input ports the system has for keyboards. [?] explored mobile phone based interaction techniques with visual displays. Their results showed that their ‘point & shoot’ technique proved to be just as good as the comparative and they recognised the addition of the mobility that their technique provided. If their prototype were to be implemented with today’s technology, the results may be better, and would certainly scale with regards the number of interacting users.

An important consideration in the argument of appropriateness of visual displays is that, at times, a mobile context may prove inconvenient for a visual display. Imagine a user driving a car; it would be dangerous for a device to actively demand their atten- tion to a visual display. [?] make a novel investigation into mobile interfaces. Shoogle employs the notion of sonification, where audio is used as the main perceptualisation of information, coupled with tactile feedback to provide ‘eyes-free’ interaction with a mobile device. The authors suggest a number of scenarios exemplifying non-visual inter- action that is more intuitive than the visual counterpart. This is further evidence that visual displays are not always the most convenient or appropriate medium. Pervasive systems and interaction with them and visual displays are discussed more thoroughly in section 4.

3 Classes of interaction

Following on from the previous section, it is important to consider the varying classes of interaction between people and machines. An accountant in an office has a very different expectation of his/her device than a user who is playing a computer game. These different interactions present varying possiblities for input. Again, a keyboard and screen setup may not always be appropriate.

3.1 Entertainment

Paelke et. al explored the use of computer vision as an input modality to a mobile computer game. They argue that the entertainment sector mainly consists of users who wish to dive straight in to the game, as ‘the only incentive...is the pleasure derived from it [the interaction]’ [?]. Playability of a game can be seen as the ease of first time use, coupled with the incentive to play again. For games, unless there is a signinficant incentive to learn a complicated interface, the fun and playability is affected. In classic 2-player fighting games, the user is rewarded for becoming familiar with the interface, as they can improve their ‘combo’ score if they string many moves together against their opponent. Paelke et al. give evaluation results of their simple computer vision based game. They report how the users claimed the game was more fun due to the physical interaction required, as opposed to a traditional keyboard and mouse setup widely used on desktop games.

Games on mobile platforms are also susceptible to the ‘2-minute lifespan’ problem. Users may stop playing the game if they are not satisfied usually within the first 2 minutes. If the interface of the game is intuitive enough that no manual is required, the game is more likely to surpass the two minute lifespan. This is easy to do in gesture-based interaction; for example, platform games usually involve the avatar jumping around. An accelerometer in a mobile device can detect the upward motion and cause the virtual character to perform the action.