Cognitive Semantics. Embodied Cognition and Dynamic Mental Representations in Language Comprehension

Table of contents

1. Introduction

2. Theoretical Background
2.1. Embodied Cognition
2.2. Dynamic Representations in Language Comprehension

3. Methodology
3.1. Research Instrument & Material
3.2. Procedure
3.3. Informants

4. Results
4.1. Descriptive Statistics
4.2. Inferential Statistics

5. Discussion

6. Conclusion

References

1. Introduction

It has been proposed by researchers that every interaction humans are part of or merely perceive leaves traces in their memory (cf. Zwaan & Madden 2005). It is the embodiment hypothesis which is the basis for this claim, saying that everything human beings perceive is imprinted, and thus embodied, in their neural circuits. This means that everything humans feel, think, perceive, conceive inside derives from their experiences and interactions with the outside world. Several studies (e.g. Zwaan 2004) have provided evidence for the assumption that these traces and neural circuits in humans’ memories are dynamically represented mentally when they experience a similar interaction the traces have once been stored with. Studies (e.g. Loftus & Palmer 1974, Zwaan et al. 2002, Vinson & Reed 2002, Zwaan 2004) have provided evidence that linguistic input alone can suffice to manipulate and generate dynamic mental representations in language comprehension.

This study aims to add to the current body of research by assuming, as Zwaan (2004), that dynamic mental representations are perceptual traces that can be temporarily reactivated. Again as Zwaan (2004), this study hypothesizes that the comprehension of a sentence auditorily presented describing a motion event (e.g. Du schießt den Fussball zu den Kindern, engl.: You kick the football to the kids) should facilitate the perception of a matching motion event presented visually (e.g. implied motion of a football moving away from the participant) compared to a mismatching motion event (e.g. implied motion of a football moving towards the participant).

To investigate dynamic representation in language comprehension an experiment was created in which 20 participants had to listen to a sentence which was followed by two pictures after which they had to judge whether the objects presented in the pictures were the same or not. Both the sentence and pictures implied the motion of a ball toward or away from the participants. In critical trials, the two pictures always presented the same ball (e.g. tennis ball, golf ball, etc.) which was also mentioned in the previous sentence and further either matched or mismatched the direction implied in the previous sentence. To imply motion in the pictures, the object in the first picture was either slightly smaller or larger than the second picture, thus implying a motion away or toward the participant.

The paper will start with a comprehensive overview of embodiment theory followed by an introduction to research and findings regarding dynamic mental representations in language comprehension. Afterwards, the methodology of this study will be described including information on the instrument and material used, experimental procedure and informants. The paper will then present most important findings which are relevant to answer the research questions posted. In the following discussion the results will be discussed in the light of previous findings and method used. At last, the paper will summarize all noteworthy results and limitations of the study and will then give an outlook for further related research.

2. Theoretical Background

2.1. Embodied Cognition

How human beings experience and process external input, i.e. their environment, is deeply connected and embodied in their mental representations (cf. Pecher & Zwaan 2007). It is especially information containing patterns of change in the environment that humans are particularly receptive of (cf. Freyd 1987). It is the realm of cognitive linguistics to investigate such relationships between "human language, the mind and socio-physical experience" (Evans et al. 2007: 263). The cognitive linguistics framework proposes that how humans remember, think, produce or understand language is based on physical interactions between humans and their environment (cf. Pecher & Zwaan 2005). This means that everything humans have stored in their mind is directly linked to their perception and action. Cognitive linguistics cannot be seen as one unified approach but aims at combining various principles and theories all together.

One such sub-field is that of cognitive semantics, which is one of the most elaborate areas of cognitive linguistics (cf. Evans et al. 2007) and which is most relevant for the present study. As a framework with different foci, cognitive semantics is based on four guiding principles: (i) conceptual structure is embodied, (ii) semantic structure is conceptual structure, (iii) meaning representation is encyclopedic and (iv) meaning construction is conceptualization (cf. Evans et al. 2007: 6). Although all of the these principles can be discussed in great detail, this paper will only discuss the embodiment hypothesis proposed in (i). The embodiment hypothesis is based on the assumption that due to the way our brain and body works, humans have a "species-specific view of the world" (Evans et al. 2007: 7). This means that our perception of the world and environment is mediated by our embodiment. This has a direct impact on our cognition and it has been argued that "we can only talk about what we can perceive and conceive, and the things that we can perceive and conceive derive from our embodied experience" (Evans et al. 2007: 7). The central idea of this thesis is that everything humans perceive and experience is imprinted and embodied in their neural circuits. How our conceptual structure is organized and placed is, thus, a direct consequence of our embodiment (Evans et al. 2007: 7). Therefore, an interdependency between humans’ bodies and psyche needs to be assumed which are deeply connected and influence each other. Early studies (e.g. Schachter 1959, Griffit & Veitch 1971) from social psychology have already provided evidence for this position showing that our bodily actions affect our psychological condition and that, hence, "people think, feel, and act inside their bodies" (Meier et al. 2012: 707). These studies have illustrated how internal cognitive processes influence our emotions, thoughts and behavior.

Although the link between embodiment and language does not seem obvious on first sight, it is this approach which "views meaning as centrally involving the activation of perceptual, motor, social and affective knowledge that characterizes the content of utterances" (Bergen 2005: 258). This implies that humans draw on their embodied knowledge which depends on individual experiences in the actual world in which those experiences are recreated in response to linguistic input to make meaning out of it (cf. Bergen 2005). As humans are exposed to language, they learn to link chunks of language, such as kick, to perceptual, motor, social and affective experiences (cf. Bergen 2005). To put it simply, how language is processed and comprehended is based on the recreation of previous experiences. Bergen et al. (2003) have described this kind of phenomenon as a ‘mirror system’ in which mirror neurons are involved. These neurons are not only activated while specific motor functions are performed, but also when an individual perceives another person performing the same function (cf. Bergen et al. 2003). This means that the function of these kinds of neurons is two-fold in that they are activated while e.g. an action is performed but also when the same action is perceived. Several studies (e.g. Pulvermüller et al. 2001, Hauk et al. 2004, Kaschak et al. 2005) have provided evidence for this position showing, for instance, that specific areas in the brain associated to specific body parts are not only active when they are used but also when they are merely referred to in motor language (cf. Bergen 2005). Zwaan and Madden (2005) further support this position by arguing that sensorimotor representations are routinely activated when meaning is construed in sentence comprehension. They argue that "interactions with the world leave traces of experience in the brain" (Zwaan & Madden 2005: 224) and so does linguistic input. These traces are reactivated when humans experience the same interaction again both in action but also in theory, for instance by reading about this interaction.

2.2. Dynamic Representations in Language Comprehension

The previous chapter has argued for the position that how humans experience and make meaning out of linguistic input is strongly connected to their embodiment. It has been demonstrated that specific areas in humans’ neural networks are activated not only when humans perform an action but also when they merely hear or read about the actions being performed. This involves not only the activation of static visual representations, but also the activation of dynamic representations called ‘representational momentum’ (Zwaan 2004).

First documented by Freyd and Finke (1984), ‘representational momentum’ is defined as "the systematic tendency for observers to remember an event as extending beyond its actual ending point" (Thornton & Hubbard 2002: 1). This means that when observers view a moving object they imagine and remember this object to have moved further than it actually did. In a study by Freyd and Finke (1984) they provide evidence for this position. In their study, informants viewed three rotating rectangles with each slightly more rotated than the one before to imply rotation. Afterwards, a fourth rectangle, a probe, was presented. Informants then were asked to judge whether the probe was at the same orientation as the third rectangle. Their findings showed that informants were more likely to judge ‘same’ when the probe was in line with the implied rotation, and thus rotated a little further in the same direction as the rectangles seen before. Another study by Hubbard and Bharucha (1988) presented informants with a target continuously moving vertically or horizontally across a monitor which would randomly vanish at some point. Informants had to indicate with their computer mouse where the target had vanished on the monitor. Again, they tended to judge the vanishing point to be more in front of the actual vanishing point in line with the moving direction. Findings from studies by Freyd and Finke (1984) as well as Hubbard and Bharucha (1988) support the assumption that humans actually seem to continue the movement of an observed object in their mind.

However, other studies (e.g. Freyd and Finke 1985) have also revealed a number of factors which can influence the effect size of representational momentum, as, for instance, acceleration, velocity, observers’ expectations and also context (cf. Thornton & Hubbard 2002, Zwaan 2004). This was greatly illustrated in Vinson and Reed’s (2002) study in which representational momentum was reported to be stronger when a moving object described as a rocket ship was also moving upwards compared to when the informants were told it was a steeple. Due to the amount of factors which may influence representational momentum, no unified explanation as to why it occurs in the first place has been consequently proposed. Findings of studies described above, nevertheless, show that mental representations can by dynamic and are thus commonly interpreted in favor for the existence of dynamic mental representations (cf. Zwaan 2004). Zwaan (2004) among other researchers, however, suggests that dynamic object representations are due to spatio-temporal associations between visual patterns which humans acquire during the actual experience with their environment. This is a suggestion towards a visual connection between space and time in which an object is moving. Based on the findings from the studies presented above and the embodiment hypothesis, which argues for the position that our conceptual knowledge is embodied, this paper assumes the same perceptual- simulation hypothesis as Zwaan (2004).

Taking this as a starting point, it has further been suggested that people routinely activate perceptual representations instead of amodal propositions during language comprehension (cf. Barsalou 1999) which is affirmed by Zwaan (2004: 613) stating that "language comprehension is a perceptual simulation of the described situation". As mentioned in the previous chapter, it has been argued that humans’ interaction with the world leaves traces in their memory. In connection to language comprehension this means that these traces become associated with words (cf. Zwaan 2004, Zwaan & Madden 2005), which involve an automatic engagement of neural systems reflecting the perceptual, motor and affective features of these words (cf. Lupyan & Bergen 2016). Word meanings are, hence, grounded or, again, embodied in sensorimotor experiences. The idea is that these traces are reactivated during language comprehension to produce perceptual simulation and let the mind mimic what is comprehended. Hence, fully comprehending a word involves a representation of the real-world referents of that word (cf. Lupyan & Bergen 2016). Evidence for this position can be deduced from a study by Zwaan et al. (2002) in which informants had to read a sentence and were afterwards presented with a picture which they had to simply recognize or name. The pictures included the object described in the sentences on experimental trials and either matched the described shape implied in the sentence or mismatched it. For instance, a picture could either depict an eagle with its wings outstretched or drawn preceded by a sentence saying "he saw an eagle in the sky". Zwaan et al. (2002) demonstrated that reaction times were significantly faster when the object in the picture matched the implied shape of the object in the sentence than when it mismatched the implied shape. This implies that comprehending eagle activates neural visual circuits linked to shape at the minimum but also to other visual properties of the object (e.g. location) (cf. Lupyan & Bergen 2016). In another study by Zwaan (2004), he further found implied direction to be activated in language comprehension as well. Informants had to first read a sentence and were then presented with two images. On critical trials the images depicted the same object mentioned in the sentence (e.g. a softball). The object in the first picture was always bigger or smaller then the one in the second picture, thus implying movement toward or away from the informant. Participants then had to decide whether the objects in the pictures were the same. Depending on the preceding sentence already including an implied direction (e.g. The shortstop hurled the softball at you.) the pictures’ implied movement either matched the sentence or mismatched it. The findings support his previous findings, namely that reaction times were again significantly faster in the match condition compared to the mismatch condition. Both studies show that comprehenders spontaneously represent motion both when reading the sentence and looking at the pictures. Slower reaction times to trials in which the implied direction in the pictures mismatched the direction implied in the sentence add further evidence to this position. Nevertheless, the findings provided by Zwaan (2004) demonstrate that the representational motion is generated entirely by linguistic stimuli, i.e. a sentence describing a motion. Compared to the study by Vinson and Reed (2002) which demonstrated that context plays an important role in the manipulation of dynamic mental representations, Zwaan’s (2004) design was much more implicit and naturalistic in terms of experimental task. While Vinson and Reed (2002) used short verbal stimuli, Zwaan (2004) used complete sentences, which makes it on the one hand surely more naturalistic but also might be a target for confounding variables. For example, what cannot be eliminated to have an influence on the participants’ behavior is the sentence construction used in Zwaan (2004) study. In some cases the sentences included a locative expression as in You tossed the beach ball over the sand toward the kids, but in other cases locative expressions were not used as in You hurled the softball at the shortstop (cf. Zwaan 2004). It has been argued that neural circuits would actually automatically activate typical locations for a softball or beach ball to appear. In cases where locative expressions were added it cannot be ruled out that this may have actually affected and influenced the outcome, meaning reaction times

Nevertheless, Zwaan (2004) argues that amodal theories cannot predict these findings and would represent eagle as a simple list of features which does, however, not include the fact that its shape my change depending on the context. The perceptual-simulation hypothesis can, on the other hand, predict these findings as it suggests that visual traces of an object are stored in memory which are reactivated during language comprehension. Faster reaction times in match conditions, thus, result from the fact that the trace already activated in the sentence matches the trace activated by the picture, therefore, not interfering with the previous trace in terms of implied direction. Longer processing times are needed if implied directions in sentence and pictures are not the same and lead to a delay in reaction times.

More evidence for the existence of representational momentum can be derived from the study conducted by Loftus and Palmer (1974). In their experiment participants watched car accidents and then were asked how far the car was driving. The question was manipulated using motor verbs implying different levels of speed as in smashed, bumped, hit or collided. One week later, participants were questioned if they saw any broken glass in the films. Participant’s which were primed with smashed were more likely to say ‘yes’ compared to the other conditions, although broken was not even present in the car accidents they watched. With this experimental design, Loftus and Palmer’s (1974) illustrated that linguistic input can indeed cause a reconstruction in one’s memory of the event generating dynamic mental representations.

3. Methodology

3.1. Research Instrument & Material

An individually programmed experiment was designed with OpenSesame to gather data for the study. Overall, the experiment consisted of 43 trials. Out of these trials, 20 were experimental, another 20 were fillers and 3 were practice trials. Practice trials were used to ensure that participants understood the introduction and instructions of the experiment. Practice trials are further useful to minimize learning curves during the experiment. One trial consisted of one sentence which was auditorily presented and then followed by two pictures depicting two different or the same object. The experiment was conducted in German. As this study is heavily based on the study conducted by Zwaan (2004), it was tried to keep the methodology as similar as possible.

[...]