Understanding Bootstrap Confidence Intervals

Introduction

Recently I set out to understand Bias-Corrected Accelerated (BCa) bootstrap confidence intervals. I was aware that it was the “gold standard” for bootstrapping and that many statistical packages used it as their default. However, I never understood the need for it relative to simpler approaches like the percentile method, which seemed more intuitive

However, this intuitive understanding was misplaced, it turns out the BCa method can be framed as a natural progression from the percentile method. This progression can actually be shown to improve the accuracy of the confidence interval from first order to second order, meaning that the confidence interval coverage converges to the nominal rate quicker. Specifically, the convergence of a reported level confidence intervals goes from

to a rate of

which makes a big difference for smaller sample size problems.

Bootstrap

First, lets have a quick refresher on what bootstrapping is. Suppose you have some fixed size data sample from an unknown data generation process , and you use this sample to generate a statistic . Now suppose we want to investigate the statistical properties of . For example, we might be interested in , or more generally . However, our statistic’s distribution is a function of our unknown distribution . therefore without the unknown distribution we are at a dead end.

This is where the ingenuity of the bootstrap method comes into play. What if instead of using we used ? In other words, what if instead of trying to calculate these expected values relative to the unknown distribution, we instead use our empirical distribution associated with adding point masses at each of the data points in the sample. The motivation for this being that, by the Glivenko–Cantelli theorem, .

The power of this approximation is that it means we can estimate the expected values via simulation. The key insight, is that sampling from is the same as resampling our original sample with replacement to generate a new size sample. We can then use this resampled data to generate a bootstrap replicate of our statistic . If we generate such bootstrap replicates then, by the law of large numbers, the sample mean of these will converge to the expected value of our statistic under .

Importantly this means that we can estimate moments such as:

Which in turn means that we can estimate the variance as:

Furthermore, we can estimate the quantile function using

To summarise, bootstrapping allows us to estimate expected values of functions of our statistic via simulation of the empirical distribution. This can be seen as a two stage approximation procedure:

Confidence Intervals

Naturally we might now ask whether we can leverage our toolkit, namely being able to estimate bootstrap quantiles and variances, to calculate confidence intervals for estimators.

if we can generate a confidence interval of the form:

Then it follows that in the limit of , by virtue of and , we would naturally get an approximation for a confidence interval of the form:

Naturally this leads to the observation that we can general make the correspondence

when moving between and . This is sometimes referred to as the “bootstrap principle”.

However, for finite , under certain assumptions different confidence interval generation processes will converge to this asymptotic correspondence faster than others, and therefore reach their nominal level of coverage faster. The methods below have been grouped as either making assumptions about pivotal quantities or normality.

Pivotal Assumptions

Each of the interval construction methods below will work best when their proposed transformation on gives rise to a distribution that does not depend on any unknown parameters. In other words we have transformations of the form:

Rather than:

Importantly, under these assumptions, we can use our bootstrap distribution to calculate approximations for quantiles of our non-bootstrap distribution. Obviously this is highly unlikely to be exactly true, but for certain problems some of these bootstrap confidence interval estimation procedure’s assumptions will be approximately true to a greater degree than others.

Basic Bootstrap

For basic bootstrap we assume that the difference between the quantity is pivotal, specifically this means that

Which means that

i.e. the following is a confidence interval

However, as has been stressed, we cannot directly calculate the distribution of our statistic . However, under our assumptions that the bootstrap pivots follow the same distribution , it follows that we can use the bootstrap quantile function to estimate the above quantities. Specifically, we can calculate bootstrap replicates of our pivotal quantiles of the form

And then get the and quantiles. Furthermore, we can simplify the form of this confidence interval purely in terms of our bootstrap resamples quantiles. Notice that is monotonically increasing in , which means that the estimated quantiles of the pivot are related to estimated quantiles of the statistic via

Which means we can restate our bootstrap confidence interval as

Bootstrap-t Intervals

The key assumption of the pivot bootstrap, in terms of rapid convergence, is that the quantity is pivotal. However, we know from classical statistics problems that this is rarely the case, it is far more common to deal with pivotal quantities of the form

This leads to a pivotal quantity that is akin to the t-statistic. However, unlike a classical t-test, which usually makes strong distributional assumptions, the bootstrap-t method only demands that this quantity is pivotal. The key assumption therefore is that

Which means that

Therefore a confidence interval for is

It should be stressed, we are not reading these quantile values from a t-table, they are estimated from the bootstrap distribution using the bootstrap ECDF \hat{H} for the pivotal quantity, i.e. bootstrap-t statistic.

Normality Assumptions

The below methods all make normality assumptions in some way, whether this is that some statistic is normally distributed or that there exists some procedure that we can perform to make the statistic normally distributed. The methods have been grouped together, because they can be seen as building on each other, at each stage loosening the assumptions and therefore having optimal convergence for a wider set of problems.

Standard Intervals

The simplest method to generate a bootstrap confidence interval is to use

However, this makes very strong assumptions. Specifically that the quantity below is pivotal and follows a normal distribution:

Which is obviously a much stronger assumption than the bootstrap-t method, and will not work in a variety of circumstances, especially for small

Percentile Intervals

The most intuitive way of deriving a confidence interval is

And indeed, this will asymptotically converge to a confidence interval at the nominal level as the bootstrap distribution converges to the true statistic sampling distribution. However, further analysis can be performed to give further justifications for such an interval, and conditions for which we expect minimal approximation error.

Suppose there exists a monotone transformation , which can be used on our raw statistic as follows:

Such that

We do not claim to know m, we only require that such a transformation exists.

Which means we can form pivotal quantities

Therefore, since

It follows that

is a confidence interval for .

Importantly, due to the symmetry of the normal distribution, we can also write this:

This is the critical step as to why the percentile method works, i.e. the symmetry of the normal distribution. In fact we could restart our analysis with:

Where is symmetric. Therefore the percentile method actually has optimal convergence properties for a wider range of scenarios than this analysis would imply.

We now make the observation that the bootstrap distribution’s quantiles for the transformed distribution are equal to

which means that the confidence interval above can be restated

Furthermore, since we are dealing with a monotone transform it follows that

Which means that

Therefore when we map our confidence interval back to space via , the result on applying this transform to quantiles in is to get back the corresponding quantiles in , i.e. our confidence interval in space is

Finally, we can use our trick of estimating quantiles using our bootstrap distribution to get an estimated confidence interval

thus giving justification for the percentile method and conditions under which it should converge optimally.

Bias Corrected (BC) Intervals

Our previous approach worked on the assumption that there was a monotone normalising transformation. However, it turns out that if our statistic is median biased, then we know for certain that no such transformation exists. This is relatively simple to show. If we truly have a transformation producing then it follows that:

Monotonicity would then also imply that

Fortunately, we can use simulation on our bootstrap distribution to estimate the median bias. Specifically, we can use the bootstrap estimate the CDF of our statistic

and check whether it is equal to 0.5.

If we find that our estimator is median biased then we can be certain that there does not exist a monotone normalising transformation around mean zero. However, we could now relax our condition and allow there to be a monotone normalising transformation around some offset .

We can now construct a confidence interval (again using the symmetry of the normal distribution):

However, unlike before we cannot immediately identify these as the quantiles of , these would have the form

Instead we notice that confidence interval above is of the form

Therefore, we expect the bias correction to have shifting effect on the quantiles

Lets now investigate this more precisely. We want to know the quantile associated with each of the above confidence interval bounds.

Therefore, our confidence interval is of the form

Which can be transformed to space via the transform to reveal

However, currently is a nuisance parameter and is stopping us from mapping our confidence interval back to space. Lets see if we can estimate it by revisiting the test we performed at the start of this analysis, now with the caveat that we have this added parameter.

Which means that:

By monotonicity:

Which can be estimated using the bootstrap distribution as

Finally using our bootstrap estimated and we can get estimates for these confidence intervals

Similar to percentile intervals we could have replaced the normal assumption to a symmetric assumption. However, in this case, this would change the final form of our confidence interval transformation to to involve non-normal quantiles. The popularity of the normal assumption reflects the fact that these quantiles are easily calculable via statistical software.

Bias Corrected Accelerated Confidence Intervals

As we saw from the BC analysis, loosening our ideal transformation allowed us to use our procedure in a wider set of problems while enjoying optimal convergence properties. Indeed the procedure detailed below actually enjoys second order accuracy, which makes it especially powerful for bootstrapping on smaller sample sizes.

Bias-Corrected Accelerated bootstrap (BCa) loosens the requirement for constant variance, allowing the standard deviation to depend linearly on the unknown parameter. Specifically, assume there exists some monotone transformation that produces

Where BC can be seen as the special case of and the percentile method can be seen as the special case of and .

After an even more elaborate analysis than the BC method, we arrive at an estimated confidence interval:

For non-parametric problems can be estimated using jackknife resampling:

and is estimated as before:

For best results |z\_0| and |a| should be small

References

• Wasserman L. All of statistics: a concise course in statistical inference. Springer Science & Business Media; 2013 Dec 11.
• Efron B, Hastie T. Computer age statistical inference, student edition: algorithms, evidence, and data science. Cambridge University Press; 2021 Jun 17.
• Efron B. The jackknife, the bootstrap and other resampling plans. Society for industrial and applied mathematics; 1982 Jan 1.
• Efron B. Better bootstrap confidence intervals. Journal of the American statistical Association. 1987 Mar 1;82(397):171-85.