Checking correctness

Outline

Topics

  • Using calibration to check correctness of model code.

Rationale

Code implementing Bayesian models can get complex in real applications. Complex code invariably means software defects (bugs) will creep in. We review a powerful method to detect bugs in Bayesian inference software.

The topic covered here is typically known as “software testing”. However we avoid the terminology “testing” here as it is already used in the statistical literature.

From goodness-of-fit to correctness check

  • In the last page, we developed a procedure for goodness-of-fit.
  • However we identified several factors that can lead to a “warning”.

Question: how can we modify last page’s check to exclude “model mis-specification” as a potential cause?

  1. Use simulated data.
  2. Use a smaller dataset.
  3. Use a larger dataset.
  4. Repeat the test several times.
  5. None of the above.

Best answer is to use simulated data. By construction, simulated data leads to the well-specified case—that is, unless there is a software defect!

Using a larger dataset will lead to only approximate calibration. Hence when we hit a warning we cannot be completely sure if it is because we need to use even more data or if there is an actual bug. Larger datasets also mean the test will take a long time to run, which is undesirable.

Additional references

Note: there are more sophisticated method for checking correctness of MCMC code, see for example:

  • The Exact Invariance Test in Bouchard-Côté, 2022, Section 10.5.
  • This line of work was initiated in Geweke, 2004. Reverse citation search on that article provides a comprehensive view of the literature on checking correctness of MCMC algorithms.