The statistical technique called bootstrap is usable with a lot of inferential problems and it is the main topic of this paper. Since the bootstrap provides material for a whole series of books
it is essential to pick one special aspect of the bootstrap and investigate it in depth, otherwise the analysis would inevitably become too general. This aspect is the topic of regression. Hence, this paper will introduce the bootstrap and compare the performance of the new inference methods which it provides with some classical methods of judging a regression which were used in the years before the bootstrap.
Therefore the remainder of this paper is as follows: First there will be a description of the basic model in which all of the following investigations will be done, chapter two. The next chapter will describe the different regression techniques which try to solve the model.
The fourth chapter is going to show the behavior of these regression techniques in large samples, i.e. shows some classical methods of statistical inference. Following chapter five will give an introduction to the bootstrap which will be succeeded by a description of the bootstrap in regression problems, chapter six. The seventh chapter will show how inference is done with the help of the bootstrap. The eighth chapter is going to compare the
performances of classical and bootstrap inference in regressions. Before the concluding remarks of chapter ten, there will be a practical application in chapter nine which tries to prove some observations of the preceeding chapters.
Table of Contents
1 Introduction
2 The model
2.1 The basic form
2.2 The disturbance term
3 Regression techniques
3.1 The method of least squares
3.1.1 Ordinary Least Squares
3.1.2 Generalized Least Squares
3.2 Alternative regression methods
4 Classical measures of performance
4.1 Bias
4.2 Variances
4.2.1 The variance of OLS
4.2.2 The variance of GLS
4.2.3 A remark on the variances
4.3 Confidence intervals
4.3.1 A remark on the critical values
4.3.2 A confidence interval for OLS
4.3.3 A confidence interval for GLS
4.4 Rate of convergence
5 The bootstrap
5.1 How does the bootstrap work?
5.2 When does the bootstrap work?
5.3 The non-parametric bootstrap
5.4 The parametric bootstrap
5.5 Why does the bootstrap work?
5.6 How many bootstrap repetitions?
5.7 On the size of each repetition
6 Regressions with the bootstrap
6.1 Case resampling
6.2 Residual resampling
6.3 Wild bootstrap
6.4 When to use which method?
7 Inference with the bootstrap
7.1 Variances with the bootstrap
7.2 Confidence intervals with the bootstrap
7.2.1 The percentile interval
7.2.2 The bootstrap-t interval
7.2.3 Other bootstrap intervals
7.3 Convergence with the bootstrap
8 Classical or bootstrap inference?
8.1 Which variance estimate?
8.2 Which confidence interval?
8.3 When the bootstrap fails
9 A practical test for the bootstrap
9.1 The datasets
9.1.1 The homoscedastic data
9.1.2 The heteroscedatic data
9.2 The simulations
9.2.1 Results simulation one
9.2.2 Results simulation two
9.3 Resumé of simulations
10 Concluding remarks
A Tables simulation one
A.1 Table of coefficient β̂1
A.2 Table of coefficient β̂2
B Tables simulation two
B.1 Table of coefficient β̂1
B.2 Table of coefficient β̂2
C Friedman test values
C.1 Test values simulation one
C.2 Test values simulation two
D Some further formulae
E Bibliography
Objectives and Topics
The primary objective of this work is to evaluate the performance of different bootstrap methods within the context of linear regression models. The paper investigates whether bootstrap inference provides meaningful improvements over classical statistical methods for measuring regression performance and constructing confidence intervals, particularly when model assumptions are violated.
- Theoretical foundations of linear regression models and least squares estimators.
- Introduction to various bootstrap resampling techniques (Case, Residual, and Wild bootstrap).
- Comparative analysis of bootstrap-based inference versus classical asymptotic inference.
- Practical simulation testing under both homoscedastic and heteroscedastic conditions.
Excerpt from the Book
6.3 Wild bootstrap
Compared to the two bootstrap techniques above the wild bootstrap lies somewhere in between, like a hybrid. On the one hand it relies on the residuals of an estimation as in residual resampling. On the other hand it has a more widespread field of application just like the case resample since it does not depend on iid errors. The wild bootstrap achieves this through imposing a special structure on the residuals. This is done by multiplying the residuals with a random variable ti. Here the residuals stay with their respective covariates (multiplied by the original least squares estimates). Hence the random variable ti is an instrument which creates new samples but preserves most of the structure of the data.
Contrary to residual resampling the randomness this time is not induced in the residuals but in ti. The probabilities and characteristics of ti are determined by the need to properly represent the error term. The most common form for ti which was used through most of its past is the following two point distribution:
y* = Xβ̂ + t · ε̂, ti = {1−√5/2 with probability 5+√5/10; 1+√5/2 with probability 5−√5/10}
However, recently it was shown that the Rademacher distribution has quite better results, so that the variable ti would have the distribution:
ti = {-1 with probability 1/2; 1 with probability 1/2}
Summary of Chapters
1 Introduction: This chapter introduces regression analysis as a core statistical challenge and outlines the motivation for using bootstrap methods as an alternative to classical asymptotic inference.
2 The model: This section defines the standard linear regression model in matrix form and establishes the fundamental assumptions, including the properties of the disturbance term.
3 Regression techniques: This chapter covers OLS and GLS estimation methods and briefly touches upon alternative regression techniques such as Least Absolute Deviations and Least Median of Squares.
4 Classical measures of performance: This chapter details classical metrics for regression evaluation, focusing on bias, variances of estimators, and the construction of confidence intervals.
5 The bootstrap: This chapter provides an introduction to the bootstrap, detailing its working mechanism, the difference between parametric and non-parametric approaches, and practical considerations for simulation.
6 Regressions with the bootstrap: This section describes three specific bootstrap resampling methods developed for regression settings: Case resampling, Residual resampling, and the Wild bootstrap.
7 Inference with the bootstrap: This chapter explains how to use bootstrap results to calculate variances and construct confidence intervals, including percentile and bootstrap-t intervals.
8 Classical or bootstrap inference?: This chapter compares classical and bootstrap inference, discussing the justification for additional computational effort and the conditions under which the bootstrap may fail.
9 A practical test for the bootstrap: This chapter presents a simulation study comparing classical standard errors with three different bootstrap methods under homoscedastic and heteroscedastic data conditions.
10 Concluding remarks: The final chapter summarizes the findings, noting that while the bootstrap is a powerful tool, the simulation results did not consistently demonstrate superior performance over classical methods in all scenarios.
Keywords
Bootstrap, Linear Regression, Ordinary Least Squares, Generalized Least Squares, Resampling, Case Resampling, Residual Resampling, Wild Bootstrap, Asymptotic Inference, Homoscedasticity, Heteroscedasticity, Variance Estimation, Confidence Intervals, Simulation, Statistical Inference.
Frequently Asked Questions
What is the core focus of this diploma thesis?
The work focuses on analyzing the performance of different bootstrap resampling methods for statistical inference in the context of linear least squares regression models.
What are the primary regression techniques discussed?
The paper primarily examines Ordinary Least Squares (OLS) and Generalized Least Squares (GLS), while also discussing alternative methods like Least Absolute Deviations (LAD) and Least Median of Squares (LMS).
What is the main objective of the bootstrap in this study?
The bootstrap is employed as a tool for statistical inference, specifically to estimate variances of regression coefficients and construct confidence intervals when classical assumptions or asymptotic approximations may not yield satisfactory results.
Which bootstrap methods are analyzed in detail?
The study analyzes three specific non-parametric bootstrap methods: Case resampling, Residual resampling, and the Wild bootstrap.
What methodology is used to evaluate the bootstrap?
The methodology combines theoretical analysis of bootstrap properties with a practical simulation study, using the Friedman test to compare the standard error estimates produced by different methods.
What are the key findings regarding the bootstrap?
The simulation results indicate that the bootstrap does not always outperform classical methods, and its performance is highly dependent on the nature of the data, such as homoscedasticity versus heteroscedasticity.
Why is the "Wild Bootstrap" considered a hybrid method?
It is considered a hybrid because it utilizes residuals like residual resampling, but it introduces randomness through a multiplicative factor to better handle non-iid errors, similar to case resampling.
Does the bootstrap always work as expected?
No, the paper explicitly notes circumstances where the bootstrap fails, such as with autocorrelated time series data, when the parameter-to-sample-size ratio is too high, or when the incorrect bootstrap method is matched with the data structure.
- Quote paper
- Diplom Volkswirt Jonas Böhmer (Author), 2009, Least Squares Regressions with the Bootstrap, Munich, GRIN Verlag, https://www.grin.com/document/135688
