Natural language is a complicated thing. When processing a sentence, the human parser has to keep track of the structure of the sentence; this requires remembering the input string, integrating new words into already built structures, and many other things, – and everything has to be done on-line. If the sentence becomes too difficult, the parser will lose control, and processing becomes slow, or may eventually break down.
There have been a number of complexity measures for natural language; the most influential one at the moment is Gibson’s (2000) Dependency Locality Theory (DLT). However, in a recent experiment, Konieczny and Döring (2003) found that reading times on clause-final verbs were faster, not slower, when the number of verb arguments was increased. This was taken as evidence against DLT’s integration cost hypothesis and for the anticipation hypothesis originally developed by Konieczny (1996): During language processing, a listener / reader anticipates what is about to come – he is “ahead of time“.
This paper presents a series of simulations modeling anticipation. Due to the fact that Simple Recurrent Networks (SRNs; Elman 1990) seem to be the most adequate device for modeling verbal working memory (MacDonald & Christiansen 2002), neural networks were used for the simulations.
In seven series of simulations, I managed to model the anticipation effect. Next to a deeper understanding of anticipation, insights into the way SRNs function could be gained.
The paper is organized as follows. First I will give an overview of different complexity measures; then the experiment mentioned above will be described. Third, I will briefly discuss existing models for verbal working memory. After a short introduction into neural network modeling, the core part of the paper, the seven simulation series, will be presented in detail.
Finally, in the Discussion I will argue that SRNs represent a good model for anticipation; implications for the anticipation hypothesis as well as implications for SRNs in general will be considered. Finally, predictions for further experiments will be discussed.
Inhaltsverzeichnis (Table of Contents)
- 1. Introduction
- 2. Complexity Measures
- 2.1. Why complexity measures?
- 2.2. Yngve
- 2.3. Bottom-up parsing
- 2.4. Left-corner parsing
- 2.5. Chomsky and Miller
- 2.6. Fodor and Garret
- 2.7. Other complexity measures
- 2.8. Gibson
- 3. The idea of anticipation
- 3.1. Shannon and Weaver
- 3.2. Entropy and anticipation
- 3.3. SOUL
- 3.4. New developments.
- 3.5. My concept of anticipation .....
- 4. Empirical evidence for anticipation
- 5. Language processing and Working Memory
- 5.1. Baddeley's model of Working Memory
- 5.2. Individual differences: reading span
- 5.3. The one-resource model
- 5.3.1. King and Just (1991).
- 5.3.2. Just and Carpenter (1992): CCREADER
- 5.4. The two-resource model
- 5.4.1. Waters and Caplan (1996)
- 5.4.2. Caplan and Waters (1999)
- 5.5. The alternative: Neural Networks
- 5.5.1. The model by MacDonald and Christiansen (2002)
- 5.5.2. Postscript to MacDonald and Christiansen's model
- 6. A short break: Integration – anticipation – neural networks
- 7. The simulations
- 27.1. Introduction to Neural Network modeling
- 7.1.1. Simple Recurrent Networks
- 7.1.2. Training and testing the network
- 7.1.3. Evaluating performance
- 7.1.4. The grammars
- 7.1.5. Details of the simulations
- 7.1.6. Overview of the seven series
- 7.2. First series
- 7.3. Second series: Anticipation is modeled
- 7.4. Third series: Even distribution of verbs
- 7.5. Fourth series: Easy versus difficult context
- 7.5.1. Subject and object relative clauses
- 7.5.2. Simply- and doubly-embedded sentences
- 7.5.2.1. With determiners
- 7.5.2.2. Without determiners
- 7.6. Fifth series: Positive and negative evidence.
- 7.6.1. The Restricted GPE
- 7.6.2. A simple grammar
- 7.6.3. A complicated grammar.
- 7.7. Sixth series: Early and late evidence.
- 7.7.1. A simple grammar
- 7.7.2. A complicated grammar
- 7.7.2.1. Was the grammar too easy?
- 7.7.2.2. Two effects?
- 7.8. Seventh series: “\"pure\" anticipation
- 27.1. Introduction to Neural Network modeling
- 8. General Discussion
- 8.1. Series 1 to 3
- 8.2. Series 4
- 8.3. Series 5 and 6
- 8.4. Series 7
- 8.5. Implications for the anticipation hypothesis .....
- 8.6. Connection to other theories
- 9. Conclusion
Zielsetzung und Themenschwerpunkte (Objectives and Key Themes)
This paper explores the phenomenon of anticipation in language processing, specifically focusing on the anticipation of clause-final heads. The research uses neural network simulations to model and investigate this anticipatory behavior.
- Complexity measures in language processing
- The anticipation hypothesis
- Modeling language processing with Simple Recurrent Networks (SRNs)
- Empirical evidence for anticipation
- Implications of the anticipation hypothesis for language processing theories
Zusammenfassung der Kapitel (Chapter Summaries)
The paper begins by providing an overview of different complexity measures used in language processing, highlighting the limitations of existing theories and introducing the anticipation hypothesis. Chapter 3 delves into the concept of anticipation, exploring its theoretical foundations and its application in language processing.
Chapter 4 presents empirical evidence supporting the anticipation hypothesis, while Chapter 5 discusses various models of verbal working memory, leading to the introduction of neural networks as a potential alternative.
The core of the paper lies in the detailed presentation of seven simulation series conducted using Simple Recurrent Networks (SRNs). Each series explores a specific aspect of anticipation, providing insights into the mechanisms of SRNs and the nature of anticipation in language processing.
The General Discussion analyzes the results of the simulations, drawing conclusions about the effectiveness of SRNs as a model for anticipation and discussing the implications of the findings for current theories of language processing.
Schlüsselwörter (Keywords)
The main keywords and focus topics of this work include: anticipation, clause-final heads, language processing, complexity measures, Simple Recurrent Networks (SRNs), neural network simulations, verbal working memory, integration cost, Dependency Locality Theory (DLT).
- Citation du texte
- Philipp Döring (Auteur), 2004, Being ahead of time. A number of neural network simulations exploring the anticipation of clause-final heads, Munich, GRIN Verlag, https://www.grin.com/document/303808
