Asymptotics

Outline

Topics

  • The notion of asymptotic analysis.
  • The two asymptotic regimes considered in this class.

Rationale

In statistics, asymptotic analysis is an important tool to understand any estimator, including Bayesian estimators and Monte Carlo methods.

A first type of asymptotics: consistency of Monte Carlo methods

Example: suppose you are trying to estimate the slope parameter in last week’s regression model.

  • Typical situation:
    • Fix one Bayesian model and one dataset.
    • You are interested in \(g^* = \mathbb{E}[g(X) | Y = y]\)
      • e.g., \(g(\text{slope}, \text{sd}) = \text{slope}\) in last week’s regression example.
    • You try simPPLe with \(M = 100\) iterations, get a Monte Carlo approximation \(\hat G_{M} \approx \mathbb{E}[g(X) | Y = y]\)
    • Then you try \(M = 1000\) iterations to gauge the quality of the Monte Carlo approximation \(\hat G_{M}\).
    • Then you try \(M = 10 000\), etc.
  • Mathematical guarantee: we proved (Monte Carlo) consistency,
    • i.e. that \(\hat G_{M}\) can get arbitrarily close to \(g^*\).

Monte Carlo consistency: the limit \(M\to\infty\) gives us a first example of asymptotic regime: “infinite computation.”

A second type of asymptotics: “big data”

Question: even after \(M \to \infty\), can there still be an error in the “infinite compute” limit \(g^*\)?

  1. No, since there is no computational error.
  2. Yes, because of model mis-specification.
  3. Yes, because of limited data.
  4. Yes, because of model mis-specification and limited data.
  5. None of the above.

Yes, because of model mis-specification and limited data.

We talked briefly about the mis-specification issues last week.

This week we will talk more about the other one: limited data—even if the model were correct and we used infinite computation, because the observations are noisy, our 24 datapoints are not enough to exactly pin-point the slope parameter.

You will explore this question in this week’s exercises. To help you preparing for this exercise, we will first go back to the first type of asymptotics (Monte Carlo consistency) in more depth.