This thesis overviews selected forecast evaluation tests and attempts to link the concept of testing equal mean squared error and forecast encompassing within a common simple regression framework. A Monte Carlo analysis provides size and power properties for both a model-free and model-based environment. In particular, the encompassing regression based test assessing the null hypothesis of equal mean squared error offers beneficial size and power properties compared to the Diebold-Mariano test, at least in a conditional homoskedastic small sample framework without autocorrelation. A simple application of several tests is provided by comparing different interest rate prediction models like a time series model, a linear model with macroeconomic indicators and a dynamic yield curve model. It turns out that simple time series specifications are hard to outperform for most of the comparisons. However, indicators like the German stock market index or the ifo expectation indicator provide useful information for future German government bond yields.
Table of Contents
1 Introduction
2 Forecast evaluation methods
2.1 Selected tests in a model-free environment
2.1.1 Forecast encompassing
2.1.2 Equal mean squared error
2.2 Selected tests in a model-based environment
2.2.1 Exemplary forecasting framework
2.2.2 Non-nested model structure
2.2.3 Nested model structure
2.2.4 Inference à la Giacomini and White (2006)
3 Monte Carlo evidence
3.1 Model-free framework
3.1.1 Size properties
3.1.2 Power properties
3.2 Model-based framework
3.2.1 Non-nested models
3.2.1.1 Size investigation
3.2.1.2 Power investigation
3.2.2 Nested models
3.2.2.1 Size investigation
3.2.2.2 Power investigation
4 Application: Predicting interest rates
4.1 Competing prediction models
4.1.1 Macro-indicator model
4.1.2 Pure time series model
4.1.3 Dynamic Svensson (1994) model
4.2 Results of pairwise comparison
4.3 Out-of-sample Granger causality
5 Conclusion
A Appendix
A.1 Fixed regressor bootstrap of Clark and McCracken (2012)
A.2 Derivations
A.3 Tables
B Bibliography
Research Objectives and Themes
This thesis examines and compares various hypothesis tests for evaluating competing forecast series, focusing on the distinction between forecast encompassing and equal mean squared error (MSE). The primary objective is to develop a unified regression-based framework for these tests and to analyze their size and power properties, particularly in small samples, through Monte Carlo simulations and an empirical application to interest rate forecasting.
- Comparison of forecast encompassing and equal MSE evaluation criteria.
- Assessment of hypothesis test performance in model-free and model-based environments.
- Investigation of small sample size and power properties using Monte Carlo evidence.
- Empirical application of predictive models for German government bond yields.
- Analysis of macroeconomic indicator predictive power and out-of-sample Granger causality.
Excerpt from the Book
2.1.1 Forecast encompassing
The concept of forecast encompassing is based on the concept of forecast combination, i.e. the idea to combine different forecasts and thereby exploit the whole predictive power of the individual forecasts. If there is no gain in predictive power from combining forecast 1 with forecast 2, the former is said to encompass the latter forecast. This idea was originally proposed by Granger and Newbold (1973), who used the terminology of forecast 1 being conditionally efficient with respect to forecast 2. More formally, consider the set of errors arising from the two competing forecast models (u1,t, u2,t) for t = 1, . . . , n and a combined forecast error
uc,t = (1 − λ)u1,t + λu2,t with 0 ≤ λ ≤ 1. (2.1)
Forecast 1 encompassing forecast 2 relates to an optimal weight of λ* = 0 in the sense that the optimal weight minimizes the mean square error of the combined forecasts (see Harvey and Newbold, 2005):
λ* = argminλ Euc,t^2. (2.2)
Summary of Chapters
1 Introduction: Outlines the three main research issues: the link between forecast evaluation criteria, the Monte Carlo analysis of test properties, and the empirical application to interest rate models.
2 Forecast evaluation methods: Details the theoretical foundation for forecast encompassing and equal MSE tests in both model-free and model-based (nested/non-nested) environments, including the incorporation of parameter estimation uncertainty.
3 Monte Carlo evidence: Provides simulation results evaluating the size and power properties of the examined tests across different model structures and data generating processes.
4 Application: Predicting interest rates: Applies the discussed forecast evaluation methods to compare various interest rate prediction models for Germany, including macro-indicator and dynamic yield curve models.
5 Conclusion: Synthesizes the main findings regarding the performance of encompassing tests in small samples and the predictive capability of the considered interest rate models.
Keywords
Forecast evaluation, forecast encompassing, mean squared error, MSE, Monte Carlo study, small sample properties, interest rate forecasting, nested models, non-nested models, macroeconomic indicators, Granger causality, parameter estimation uncertainty, Diebold-Mariano test, forecast combination.
Frequently Asked Questions
What is the core focus of this research?
The research focuses on summarizing and comparing selected hypothesis tests used to evaluate competing forecast series, specifically linking the concepts of forecast encompassing and equal mean squared error within a regression framework.
What are the primary topics covered in this thesis?
The thesis covers forecast evaluation theory, Monte Carlo simulation evidence regarding test size and power, and an empirical application involving the prediction of German government bond yields using various models.
What is the main objective of the thesis?
The primary objective is to establish a framework that links different evaluation criteria and to demonstrate that specific modified encompassing tests offer superior size and power properties compared to traditional tests in small samples.
Which scientific methods are employed?
The research employs econometric modeling for forecast evaluation, analytical derivations of test statistics, and extensive Monte Carlo simulations to assess test properties under various data generating processes.
What does the main body address?
The main body systematically presents the theoretical framework for forecast evaluation methods, provides Monte Carlo evidence for model-free and model-based environments, and performs an empirical application to interest rate forecasting.
What keywords characterize the work?
Key terms include forecast evaluation, encompassing, mean squared error, Monte Carlo study, small sample properties, and interest rate forecasting.
How does the research address nested models?
The thesis investigates the out-of-sample Granger causality framework for nested models, noting the necessity of specialized critical values due to the violation of standard distribution assumptions.
What are the findings regarding the German bond market?
The empirical analysis suggests that simple time series models are robust benchmarks, while certain indicators like the German stock market index and the ifo expectation index provide useful predictive information for specific yield maturities.
What role does parameter estimation uncertainty play?
The work demonstrates that parameter estimation uncertainty can lead to serious size distortions in tests, necessitating robust estimation or appropriate adjustments to ensure valid statistical inference.
- Quote paper
- Frank Undorf (Author), 2017, Forecast evaluation methods: A Monte Carlo investigation and an application to the predictability of interest rates, Munich, GRIN Verlag, https://www.grin.com/document/439428
