Bayesian Statistics

An Introduction

Fourth Edition

PETER M. LEE

(ISBN 978-1-118-33257-3)

Table of Contents

• Preface
• Preface to the First Edition

1. Preliminaries
   1. Probability and Bayes’ Theorem
      1. Notation
      2. Axioms for probability
      3. ‘Unconditional’ probability
      4. Odds
      5. Independence
      6. Some simple consequences of the axioms; Bayes’ Theorem
   2. Examples on Bayes’ Theorem
      1. The Biology of Twins
      2. A political example
      3. A warning
   3. Random variables
      1. Discrete random variable
      2. The binomial distribution
      3. Continuous random variables
      4. The normal distribution
      5. Mixed random variables
   4. Several random variables
      1. Two discrete random variables
      2. Two continuous random variables
      3. Bayes’ Theorem for random variables
      4. Example
      5. One discrete variable and one continuous variable
      6. Independent random variables
   5. Means and variances
      1. Expectations
      2. The expectation of a sum and of a product
      3. Variance, precision and standard deviation
      4. Examples
      5. Variance of a sum; covariance and correlation
      6. Approximations to the mean and variance of a function of a random variable
      7. Conditional expectations and variances
      8. Medians and modes
   6. Exercises on Chapter 1
2. Bayesian Inference for the Normal Distribution
   1. Nature of Bayesian inference
      1. Preliminary remarks
      2. Post is prior times likelihood
      3. Likelihood can be multiplied by any constant
      4. Sequential use of Bayes’ Theorem
      5. The predictive distribution
      6. A warning
   2. Normal prior and likelihood
      1. Posterior from a normal prior and likelihood
      2. Example
      3. Predictive distribution
      4. The nature of the assumptions made
   3. Several normal observations with a normal prior
      1. Posterior distribution
      2. Example
      3. Predictive distribution
      4. Robustness
   4. Dominant likelihoods
      1. Improper priors
      2. Approximation of proper priors by improper priors
   5. Locally uniform priors
      1. Bayes’ postulate
      2. Data translated likelihoods
      3. Transformation of unknown parameters
   6. Highest density regions (HDRs)
      1. Need for summaries of posterior information
      2. Relation to classical statistics
   7. Normal variance
      1. A suitable prior for the normal variance
      2. Reference prior for the normal variance
   8. HDRs for the normal variance
      1. What distribution should we be considering?
      2. Example
   9. The role of sufficiency
      1. Definition of sufficiency
      2. Neyman’s Factorization Theorem
      3. Sufficiency Principle
      4. Examples
      5. Order Statistics and Minimal Sufficient Statistics
      6. Examples on minimal sufficiency
   10. Conjugate prior distributions
       1. Definition and difficulties
       2. Examples
       3. Mixtures of conjugate densities
       4. Is your prior really conjugate?
   11. The exponential family
       1. Definition
       2. Examples
       3. Conjugate densities
       4. Two-parameter exponential family
   12. Normal mean and variance both unknown
       1. Formulation of the problem
       2. Marginal distribution of the mean
       3. Example of the posterior density for the mean
       4. Marginal distribution of the variance
       5. Example of the posterior density of the variance
       6. Conditional density of the mean for given variance
   13. Conjugate joint prior for the normal
       1. The form of the conjugate prior
       2. Derivation of the posterior
       3. Example
       4. Concluding remarks
   14. Exercises on Chapter 2
3. Some Other Common Distributions
   1. The binomial distribution
      1. Conjugate prior
      2. Odds and log-odds
      3. Highest density regions
      4. Example
      5. Predictive distribution
   2. Reference prior for the binomial likelihood
      1. Bayes&rsquop; postulate
      2. Haldane’s prior
      3. The arc-sine distribution
      4. Conclusion
   3. Jeffreys’ rule
      1. Fisher’s information
      2. The information from several observations
      3. Jeffreys’ prior
      4. Examples
      5. Warning
      6. Several unknown parameters
      7. Example
   4. The Poisson distribution
      1. Conjugate prior
      2. Reference prior
      3. Example
      4. Predictive distribution
   5. The uniform distribution
      1. Preliminary definitions
      2. Uniform distribution with a fixed lower endpoint
      3. The general uniform distribution
      4. Examples
   6. Reference prior for the uniform distribution
      1. Lower limit of the interval fixed
      2. Example
      3. Both limits unknown
   7. The tramcar problem
      1. The discrete uniform distribution
   8. The first digit problem; invariant priors
      1. A prior in search of an explanation
      2. The problem
      3. A solution
      4. Haar priors
   9. The circular normal distribution
      1. Distributions on the circle
      2. Example
      3. Construction of an HDR by numerical integration
      4. Remarks
   10. Approximations based on the likelihood
       1. Maximum likelihood
       2. Iterative methods
       3. Approximation to the posterior density
       4. Examples
       5. Extension to more than one parameter
       6. Example
   11. Reference Posterior Distributions
       1. The information provided by an experiment
       2. Reference priors under asymptotic normality
       3. Uniform distribution of unit length
       4. Normal mean and variance
       5. Technical complications
   12. Exercises on Chapter 3
4. Hypothesis Testing
   1. Hypothesis testing
      1. Introduction
      2. Classical hypothesis testing
      3. Difficulties with the classical approach
      4. The Bayesian approach
      5. Example
      6. Comment
   2. One-sided hypothesis tests
      1. Definition
      2. P-values
   3. Lindley’s method
      1. A compromise with classical statistics
      2. Example
      3. Discussion
   4. Point null hypotheses with prior information
      1. When are point null hypotheses reasonable?
      2. A case of nearly constant likelihood
      3. The Bayesian method for point null hypotheses
      4. Sufficient statistics
   5. Point null hypotheses (normal case)
      1. Calculation of the Bayes’ factor
      2. Numerical examples
      3. Lindley’s paradox
      4. A bound which does not depend on the prior distribution
      5. The case of an unknown variance
   6. The Doogian philosophy
      1. Description of the method
      2. Numerical example
   7. Exercises on Chapter 4
5. Two-sample Problems
   1. Two-sample problems – both variances unknown
      1. The problem of two normal samples
      2. Paired comparisons
      3. Example of a paired comparison problem
      4. The case where both variances are known
      5. Example
      6. Non-trivial prior information
   2. Variances unknown but equal
      1. Solution using reference priors
      2. Example
      3. Non-trivial prior information
   3. Variances unknown and unequal (Behrens-Fisher problem)
      1. Formulation of the problem
      2. Patil’s approximation
      3. Example
      4. Substantial prior information
   4. The Behrens-Fisher controversy
      1. The Behrens-Fisher problem from a classical standpoint
      2. Example
      3. The controversy
   5. Inferences concerning a variance ratio
      1. Statement of the problem
      2. Derivation of the F distribution
      3. Example
   6. Comparison of two proportions; the 2 x 2 table
      1. Methods based on the log odds-ratio
      2. Example
      3. The inverse root-sine transformation
      4. Other methods
   7. Exercises on Chapter 5
6. Correlation, Regression and ANOVA
   1. Theory of the correlation coefficient
      1. Definitions
      2. Approximate posterior distribution of the correlation coefficient
      3. The hyperbolic tangent substitution
      4. Reference prior
      5. Incorporation of prior information
   2. Examples on correlation
      1. Use of the hyperbolic tangent transformation
      2. Combination of several correlation coefficients
      3. The squared correlation coefficient
   3. Regression and the bivariate normal model
      1. The model
      2. Bivariate linear regression
      3. Example
      4. Case of known variance
      5. The mean value at a given value of the explanatory variable
      6. Prediction of observations at a given value of the explanatory variable
      7. Continuation of the example
      8. Multiple regression
      9. Polynomial regression
   4. Conjugate prior for bivariate regression
      1. The problem of updating a regression line
      2. Formulae for recursive construction of a regression line
      3. Finding an appropriate prior
   5. Comparison of several means – the one way model
      1. Description of the one way layout
      2. Integration over the nuisance parameters
      3. Derivation of the F distribution
      4. Relationship to the analysis of variance
      5. Example
      6. Relationship to a simple linear regression model
      7. Investigation of contrasts
   6. The two way layout
      1. Notation
      2. Marginal posterior distributions
      3. Analysis of variance
   7. The general linear model
      1. Formulation of the general linear model
      2. Derivation of the posterior
      3. Inference for a subset of the parameters
      4. Application to bivariate linear regression
   8. Exercises on Chapter 6
7. Other Topics
   1. The likelihood principle
      1. Introduction
      2. The conditionality principle
      3. The sufficiency principle
      4. The likelihood principle
      5. Discussion
   2. The stopping rule principle
      1. Definitions
      2. Examples
      3. The stopping rule principle
      4. Discussion
   3. Informative stopping rules
      1. An example on capture and recapture of fish
      2. Choice of prior and derivation of posterior
      3. The maximum likelihood estimator
      4. Numerical example
   4. The likelihood principle and reference priors
      1. The case of Bernoulli trials and its general implications
      2. Conclusion
   5. Bayesian decision theory
      1. The elements of game theory
      2. Point estimators resulting from quadratic loss
      3. Particular cases of quadratic loss
      4. Weighted quadratic loss
      5. Absolute error loss
      6. Zero-one loss
      7. General discussion of point estimation
   6. Bayes linear methods
      1. Methodology
      2. Some simple examples
      3. Extensions
   7. Decision theory and hypothesis testing
      1. Decision theory and classical hypothesis testing
      2. Composite hypotheses
   8. Empirical Bayes methods
      1. Von Mises’ example
      2. The Poisson case
   9. Exercises on Chapter 7
8. Hierarchical Models
   1. The idea of a hierarchical model
      1. Definition
      2. Examples
      3. Objectives of a Hierarchical Analysis
      4. More on Empirical Bayes Methods
   2. The hierarchical normal model
      1. The model
      2. The Bayesian analysis for known overall mean
      3. The empirical Bayes approach
   3. The baseball example
   4. The Stein estimator
      1. Evaluation of the risk of the James-Stein estimator
   5. Bayesian analysis for an unknown overall mean
      1. Derivation of the posterior
   6. The general linear model revisited
      1. An informative prior for the general linear model
      2. Ridge regression
      3. A further stage to the general linear model
      4. The one way model
      5. Posterior variances of the estimators
   7. Exercises on Chapter 8
9. The Gibbs Sampler
   1. Introduction to numerical methods
      1. Monte Carlo methods
      2. Markov chains
   2. The EM algorithm
      1. The idea of the EM algorithm
      2. Why the EM algorithm works
      3. Semi-conjugate prior with a normal likelihood
      4. The EM algorithm for the hierarchical normal model
      5. A particular case of the hierarchical normal model
   3. Data augmentation by Monte Carlo
      1. The genetic linkage example revisited
      2. Use of R
      3. The Genetic Linkage Example in R
      4. Other possible uses for data augmentation
   4. The Gibbs sampler
      1. Chained data augmentation
      2. An example with observed data
      3. More on the semi-conjugate prior with a normal likelihood
      4. The Gibbs sampler as an extension of chained data augmentation
      5. An application to change-point analysis
      6. Other uses of the Gibbs sampler
      7. More about convergence
   5. Rejection sampling
      1. Description
      2. Example
      3. Rejection sampling for log-concave distributions
      4. A practical example
   6. The Metropolis-Hastings Algorithm
      1. Finding an invariant distribution
      2. The Metropolis-Hastings algorithm
      3. Choice of a candidate density
      4. Example
      5. More Realistic Examples
      6. Gibbs as a special case of Metropolis-Hastings
      7. Metropolis within Gibbs
   7. Introduction to WinBUGS and OpenBUGS
      1. Information about WinBUGS and OpenBUGS
      2. Distributions in WinBUGS and OpenBUGS
      3. A Simple Example using WinBUGS
      4. The Pump Failure Example Revisited
      5. DoodleBUGS
      6. coda
      7. R2WinBUGS and R2OpenBUGS
   8. Generalized Linear Models
      1. Logistic Regression
      2. A general framework
   9. Exercises on Chapter 9
10. Some Approximate Methods
    1. Bayesian importance sampling
       1. Importance Sampling to find HDRs
       2. Sampling importance resampling (SIR)
       3. Multidimensional applications
    2. Variational Bayesian methods: simple case
       1. Independent Parameters
       2. Application to the normal distribution
       3. Updating the mean
       4. Updating the variance
       5. Iteration
       6. Numerical example
    3. Variational Bayesian methods: general case
       1. A mixture of multivariate normals
    4. ABC: Approximate Bayesian Computation
       1. The ABC rejection algorithm
       2. The genetic linkage example
       3. The ABC Markov Chain Monte Carlo algorithm
       4. The ABC Sequential Monte Carlo algorithm
       5. The ABC local linear regression algorithm
       6. Other variants of ABC
    5. Reversible Jump Markov Chain Monte Carlo
    6. Exercises on Chapter 10

1. A Common Statistical Distributions
   1. Normal distribution
   2. Chi-squared distribution
   3. Normal approximation to chi-squared
   4. Gamma distribution
   5. Inverse chi-squared distribution
   6. Inverse chi distribution
   7. Log chi-squared distribution
   8. Student’s t distribution
   9. Normal/chi-squared distribution
   10. Beta distribution
   11. Binomial distribution
   12. Poisson distribution
   13. Negative binomial distribution
   14. Hypergeometric distribution
   15. Uniform distribution
   16. Pareto distribution
   17. Circular normal distribution
   18. Behrens’ distribution
   19. Snedecor’s F distribution
   20. Fisher’s z distribution
   21. Cauchy distribution
   22. Difference of beta variables
   23. Bivariate normal distribution
   24. Multivariate normal distribution
   25. Distribution of the correlation coefficient
2. Tables
   1. Percentage points of the Behrens-Fisher distribution
   2. HDRs for the chi-squared distribution
   3. HDRs for the inverse chi-squared distribution
   4. Chi-squared corresponding to HDRs for log c2
   5. Values of F corresponding to HDRs for log F
3. R Programs
4. Further Reading
   1. Robustness
   2. Nonparametric methods
   3. Multivariate estimation
   4. Time series and forecasting
   5. Sequential methods
   6. Numerical methods
   7. Bayesian Networks
   8. General reading

• References
• Index

• Preface to the Fourth Edition
• Preface to the Third Edition
• Preface to the Second Edition
• Preface to the First Edition
• Answers to Exercises (LaTeX source file)
• Answers to Exercises (PostScript file) [Broken link; use PDF file below SPE 2017/06/16]
• Answers to Exercises (pdf file)
• Further Exercises (not in the book) (LaTeX source file)
• Further Exercises (not in the book) (PostScript file) [Broken link; use PDF file below SPE 2017/06/16]
• Further Exercises (not in the book) (pdf file)
• Computer programs relating to the book
  • Versions in C++ which can be run on Windows using http://www.cygwin.com/.
  • Versions in the R language which is available free.
  • Versions in WinBUGS which is available free.
• Send a message to the maintainer of these web pages
• Return to the welcome page for the book

Peter M. Lee

2 July 2012