# Bayesian and Frequentist Regression Methods by Jon Wakefield By Jon Wakefield

This booklet offers a balanced, sleek precis of Bayesian and frequentist equipment for regression analysis.

Cover

Bayesian and Frequentist Regression Methods

ISBN 9781441909244 ISBN 9781441909251

Preface

Contents

Chapter 1 advent and Motivating Examples

1.1 Introduction
1.2 version Formulation
1.3 Motivating Examples
1.3.1 Prostate Cancer
1.3.2 final result After Head Injury
1.3.4 Pharmacokinetic Data
1.3.5 Dental Growth
1.3.6 Spinal Bone Mineral Density
1.4 Nature of Randomness
1.5 Bayesian and Frequentist Inference
1.7 Bibliographic Notes

Part I

bankruptcy 2 Frequentist Inference
2.1 Introduction
2.2 Frequentist Criteria
2.3 Estimating Functions
2.4 Likelihood
o 2.4.1 greatest chance Estimation
o 2.4.2 variations on Likelihood
o 2.4.3 version Misspecification
2.5 Quasi-likelihood 2.5.1 greatest Quasi-likelihood Estimation
o 2.5.2 A extra advanced Mean-Variance Model
2.6 Sandwich Estimation
2.7 Bootstrap Methods
o 2.7.1 The Bootstrap for a Univariate Parameter
o 2.7.2 The Bootstrap for Regression
o 2.7.3 Sandwich Estimation and the Bootstrap
2.8 number of Estimating Function
2.9 speculation Testing
o 2.9.1 Motivation
o 2.9.2 Preliminaries
o 2.9.3 rating Tests
o 2.9.4 Wald Tests
o 2.9.5 chance Ratio Tests
o 2.9.6 Quasi-likelihood
o 2.9.7 comparability of try out Statistics
2.10 Concluding Remarks
2.11 Bibliographic Notes
2.12 Exercises
bankruptcy three Bayesian Inference
3.1 Introduction
3.2 The Posterior Distribution and Its Summarization
3.3 Asymptotic houses of Bayesian Estimators
3.4 previous Choice
o 3.4.1 Baseline Priors
o 3.4.2 great Priors
o 3.4.3 Priors on significant Scales
o 3.4.4 Frequentist Considerations
3.5 version Misspecification
3.6 Bayesian version Averaging
3.7 Implementation
o 3.7.1 Conjugacy
o 3.7.2 Laplace Approximation
o 3.7.4 built-in Nested Laplace Approximations
o 3.7.5 significance Sampling Monte Carlo
o 3.7.6 Direct Sampling utilizing Conjugacy
o 3.7.7 Direct Sampling utilizing the Rejection Algorithm
3.8 Markov Chain Monte Carlo 3.8.1 Markov Chains for Exploring Posterior Distributions
o 3.8.2 The Metropolis-Hastings Algorithm
o 3.8.3 The city Algorithm
o 3.8.4 The Gibbs Sampler
o 3.8.5 Combining Markov Kernels: Hybrid Schemes
o 3.8.6 Implementation Details
o 3.8.7 Implementation Summary
3.9 Exchangeability
3.10 speculation checking out with Bayes Factors
3.11 Bayesian Inference in response to a Sampling Distribution
3.12 Concluding Remarks
3.13 Bibliographic Notes
3.14 Exercises
bankruptcy four speculation trying out and Variable Selection
4.1 Introduction
4.2 Frequentist speculation Testing
o 4.2.1 Fisherian Approach
o 4.2.2 Neyman-Pearson Approach
o 4.2.3 Critique of the Fisherian Approach
o 4.2.4 Critique of the Neyman-Pearson Approach
4.3 Bayesian speculation trying out with Bayes elements 4.3.1 review of Approaches
o 4.3.2 Critique of the Bayes issue Approach
o 4.3.3 A Bayesian View of Frequentist speculation Testing
4.5 trying out a number of Hypotheses: common Considerations
4.6 checking out a number of Hypotheses: fastened variety of Tests
o 4.6.1 Frequentist Analysis
o 4.6.2 Bayesian Analysis
4.7 trying out a number of Hypotheses: Variable Selection
4.8 methods to Variable choice and Modeling
o 4.8.1 Stepwise Methods
o 4.8.2 All attainable Subsets
o 4.8.3 Bayesian version Averaging
o 4.8.4 Shrinkage Methods
4.9 version construction Uncertainty
4.10 a realistic Compromise to Variable Selection
4.12 Bibliographic Notes
4.13 Exercises

Part II

bankruptcy five Linear Models
5.1 Introduction
5.2 Motivating instance: Prostate Cancer
5.3 version Specifiation
5.4 A Justificatio for Linear Modeling
5.5 Parameter Interpretation
o 5.5.1 Causation as opposed to Association
o 5.5.2 a number of Parameters
o 5.5.3 info Transformations
5.6 Frequentist Inference 5.6.1 Likelihood
o 5.6.2 Least Squares Estimation
o 5.6.3 The Gauss-Markov Theorem
o 5.6.4 Sandwich Estimation
5.7 Bayesian Inference
5.8 research of Variance
o 5.8.1 One-Way ANOVA
o 5.8.2 Crossed Designs
o 5.8.3 Nested Designs
o 5.8.4 Random and combined results Models
5.10 Robustness to Assumptions
o 5.10.1 Distribution of Errors
o 5.10.2 Nonconstant Variance
o 5.10.3 Correlated Errors
5.11 overview of Assumptions
o 5.11.1 assessment of Assumptions
o 5.11.2 Residuals and In uence
o 5.11.3 utilizing the Residuals
5.12 instance: Prostate Cancer
5.13 Concluding Remarks
5.14 Bibliographic Notes
5.15 Exercises
bankruptcy 6 common Regression Models
6.1 Introduction
6.2 Motivating instance: Pharmacokinetics of Theophylline
6.3 Generalized Linear Models
6.4 Parameter Interpretation
6.5 chance Inference for GLMs 6.5.1 Estimation
o 6.5.2 Computation
o 6.5.3 speculation Testing
6.6 Quasi-likelihood Inference for GLMs
6.7 Sandwich Estimation for GLMs
6.8 Bayesian Inference for GLMs
o 6.8.1 earlier Specification
o 6.8.2 Computation
o 6.8.3 speculation Testing
o 6.8.4 Overdispersed GLMs
6.9 evaluation of Assumptions for GLMs
6.10 Nonlinear Regression Models
6.11 Identifiabilit
6.12 probability Inference for Nonlinear versions 6.12.1 Estimation
o 6.12.2 speculation Testing
6.13 Least Squares Inference
6.14 Sandwich Estimation for Nonlinear Models
6.15 The Geometry of Least Squares
6.16 Bayesian Inference for Nonlinear Models
o 6.16.1 past Specification
o 6.16.2 Computation
o 6.16.3 speculation Testing
6.17 overview of Assumptions for Nonlinear Models
6.18 Concluding Remarks
6.19 Bibliographic Notes
6.20 Exercises
bankruptcy 7 Binary information Models
7.1 Introduction
7.2 Motivating Examples 7.2.1 final result After Head Injury
o 7.2.2 plane Fasteners
o 7.2.3 Bronchopulmonary Dysplasia
7.3 The Binomial Distribution 7.3.1 Genesis
o 7.3.2 infrequent Events
7.4 Generalized Linear versions for Binary info 7.4.1 Formulation
7.5 Overdispersion
7.6 Logistic Regression versions 7.6.1 Parameter Interpretation
o 7.6.2 probability Inference for Logistic Regression Models
o 7.6.3 Quasi-likelihood Inference for Logistic Regression Models
o 7.6.4 Bayesian Inference for Logistic Regression Models
7.7 Conditional probability Inference
7.8 evaluate of Assumptions
7.9 Bias, Variance, and Collapsibility
7.10 Case-Control Studies
o 7.10.1 The Epidemiological Context
o 7.10.2 Estimation for a Case-Control Study
o 7.10.3 Estimation for a Matched Case-Control Study
7.11 Concluding Remarks
7.12 Bibliographic Notes
7.13 Exercises

Part III

bankruptcy eight Linear Models
8.1 Introduction
8.2 Motivating instance: Dental progress Curves
8.3 The Effciency of Longitudinal Designs
8.4 Linear combined types 8.4.1 the overall Framework
o 8.4.2 Covariance versions for Clustered Data
o 8.4.3 Parameter Interpretation for Linear combined Models
8.5 probability Inference for Linear combined Models
o 8.5.1 Inference for mounted Effects
o 8.5.2 Inference for Variance elements through greatest Likelihood
o 8.5.3 Inference for Variance elements through constrained greatest Likelihood
o 8.5.4 Inference for Random Effects
8.6 Bayesian Inference for Linear combined versions 8.6.1 A Three-Stage Hierarchical Model
o 8.6.2 Hyperpriors
o 8.6.3 Implementation
o 8.6.4 Extensions
8.7 Generalized Estimating Equations 8.7.1 Motivation
o 8.7.2 The GEE Algorithm
o 8.7.3 Estimation of Variance Parameters
8.8 evaluation of Assumptions 8.8.1 assessment of Assumptions
o 8.8.2 techniques to Assessment
8.9 Cohort and Longitudinal Effects
8.10 Concluding Remarks
8.11 Bibliographic Notes
8.12 Exercises
bankruptcy nine basic Regression Models
9.1 Introduction
9.2 Motivating Examples
o 9.2.1 birth control Data
o 9.2.2 Seizure Data
o 9.2.3 Pharmacokinetics of Theophylline
9.3 Generalized Linear combined Models
9.4 chance Inference for Generalized Linear combined Models
9.5 Conditional chance Inference for Generalized Linear combined Models
9.6 Bayesian Inference for Generalized Linear combined types 9.6.1 version Formulation
o 9.6.2 Hyperpriors
9.7 Generalized Linear combined versions with Spatial Dependence 9.7.1 A Markov Random box Prior
o 9.7.2 Hyperpriors
9.8 Conjugate Random results Models
9.9 Generalized Estimating Equations for Generalized Linear Models
9.10 GEE2: attached Estimating Equations
9.11 Interpretation of Marginal and Conditional Regression Coeffiients
9.12 creation to Modeling established Binary Data
9.13 combined versions for Binary info 9.13.1 Generalized Linear combined types for Binary Data
o 9.13.2 probability Inference for the Binary combined Model
o 9.13.3 Bayesian Inference for the Binary combined Model
o 9.13.4 Conditional probability Inference for Binary combined Models
9.14 Marginal versions for based Binary Data
o 9.14.1 Generalized Estimating Equations
o 9.14.2 Loglinear Models
o 9.14.3 additional Multivariate Binary Models
9.15 Nonlinear combined Models
9.16 Parameterization of the Nonlinear Model
9.17 probability Inference for the Nonlinear combined Model
9.18 Bayesian Inference for the Nonlinear combined Model
o 9.18.1 Hyperpriors
o 9.18.2 Inference for capabilities of Interest
9.19 Generalized Estimating Equations
9.20 overview of Assumptions for normal Regression Models
9.21 Concluding Remarks
9.22 Bibliographic Notes
9.23 Exercises

Part IV

bankruptcy 10 Preliminaries for Nonparametric Regression
10.1 Introduction
10.2 Motivating Examples
o 10.2.1 gentle Detection and Ranging
o 10.2.2 Ethanol Data
10.3 The optimum Prediction
o 10.3.1 non-stop Responses
o 10.3.2 Discrete Responses with ok Categories
o 10.3.3 normal Responses
o 10.3.4 In Practice
10.4 Measures of Predictive Accuracy
o 10.4.1 non-stop Responses
o 10.4.2 Discrete Responses with ok Categories
o 10.4.3 basic Responses
10.5 a primary examine Shrinkage Methods
o 10.5.1 Ridge Regression
o 10.5.2 The Lasso
10.6 Smoothing Parameter Selection
o 10.6.1 Mallows CP
o 10.6.2 K-Fold Cross-Validation
o 10.6.3 Generalized Cross-Validation
o 10.6.4 AIC for normal Models
o 10.6.5 Cross-Validation for Generalized Linear Models
10.8 Bibliographic Notes
10.9 Exercises
bankruptcy eleven Spline and Kernel Methods
11.1 Introduction
11.2 Spline tools 11.2.1 Piecewise Polynomials and Splines
o 11.2.2 common Cubic Splines
o 11.2.3 Cubic Smoothing Splines
o 11.2.4 B-Splines
o 11.2.5 Penalized Regression Splines
o 11.2.6 a short Spline Summary
o 11.2.7 Inference for Linear Smoothers
o 11.2.8 Linear combined version Spline illustration: probability Inference
o 11.2.9 Linear combined version Spline illustration: Bayesian Inference
11.3 Kernel Methods
o 11.3.1 Kernels
o 11.3.2 Kernel Density Estimation
o 11.3.3 The Nadaraya-Watson Kernel Estimator
o 11.3.4 neighborhood Polynomial Regression
11.4 Variance Estimation
11.5 Spline and Kernel equipment for Generalized Linear Models
o 11.5.1 Generalized Linear versions with Penalized Regression Splines
o 11.5.2 A Generalized Linear combined version Spline Representation
o 11.5.3 Generalized Linear types with neighborhood Polynomials
11.7 Bibliographic Notes
11.8 Exercises
bankruptcy 12 Nonparametric Regression with a number of Predictors
12.1 Introduction
12.2 Generalized Additive versions 12.2.1 version Formulation
o 12.2.2 Computation through Backfittin
12.3 Spline tools with a number of Predictors
o 12.3.1 average skinny Plate Splines
o 12.3.2 skinny Plate Regression Splines
o 12.3.3 Tensor Product Splines
12.4 Kernel tools with a number of Predictors
12.5 Smoothing Parameter Estimation 12.5.1 traditional Approaches
o 12.5.2 combined version Formulation
12.6 Varying-Coefficien Models
12.7 Regression timber 12.7.1 Hierarchical Partitioning
o 12.7.2 a number of Adaptive Regression Splines
12.8 Classificatio
o 12.8.1 Logistic versions with ok Classes
o 12.8.2 Linear and Quadratic Discriminant Analysis
o 12.8.3 Kernel Density Estimation and Classificatio
o 12.8.4 Classificatio Trees
o 12.8.5 Bagging
o 12.8.6 Random Forests
12.10 Bibliographic Notes
12.11 Exercises

Part V

Appendix A Differentiation of Matrix Expressions
Appendix B Matrix Results
Appendix C a few Linear Algebra
Appendix D chance Distributions and producing Functions
Appendix E features of standard Random Variables
Appendix F a few effects from Classical Statistics
Appendix G uncomplicated huge pattern Theory

References

Index

Similar probability books

Applied Bayesian Modelling (2nd Edition) (Wiley Series in Probability and Statistics)

This publication offers an obtainable method of Bayesian computing and information research, with an emphasis at the interpretation of genuine information units. Following within the culture of the profitable first version, this ebook goals to make quite a lot of statistical modeling purposes available utilizing confirmed code that may be without difficulty tailored to the reader's personal functions.

Fuzzy analysis as alternative to stochastic methods -- theoretical aspects

A practical and trustworthy numerical simulation calls for appropriate computational versions and appropriate facts versions for the structural layout parameters. Structural layout parameters are normally non-deterministic, i. e. doubtful. the alternative of an acceptable uncertainty version for describing chosen structural layout parameters will depend on the attribute of the to be had info.

Stochastic Hybrid Systems

Cohesively edited by means of major specialists within the box, Stochastic Hybrid platforms (SHS) introduces the theoretical fundamentals, computational equipment, and functions of SHS. The booklet first discusses the underlying rules in the back of SHS and the most layout boundaries of SHS. development on those basics, the authoritative participants current equipment for laptop calculations that follow SHS research and synthesis concepts in perform.

Linear Stochastic Systems: A Geometric Approach to Modeling, Estimation and Identification

Maximizes reader insights into stochastic modeling, estimation, procedure identity, and time sequence analysis
Reveals the options of stochastic nation house and country area modeling to unify the idea
Supports extra exploration via a unified and logically constant view of the subject

This e-book provides a treatise at the idea and modeling of second-order desk bound procedures, together with an exposition on chosen program components which are very important within the engineering and technologies. The foundational matters concerning desk bound tactics handled at first of the publication have an extended historical past, beginning within the Nineteen Forties with the paintings of Kolmogorov, Wiener, Cramér and his scholars, particularly Wold, and feature because been subtle and complemented by way of many others. difficulties in regards to the filtering and modeling of desk bound random indications and platforms have additionally been addressed and studied, fostered by way of the appearance of recent electronic desktops, because the primary paintings of R. E. Kalman within the early Sixties. The ebook deals a unified and logically constant view of the topic in line with basic rules from Hilbert house geometry and coordinate-free considering. during this framework, the strategies of stochastic country area and nation house modeling, in accordance with the proposal of the conditional independence of previous and destiny flows of the appropriate signs, are published to be essentially unifying principles. The publication, according to over 30 years of unique examine, represents a priceless contribution that might tell the fields of stochastic modeling, estimation, process identity, and time sequence research for many years to return. It additionally presents the mathematical instruments had to snatch and examine the constructions of algorithms in stochastic platforms conception.

Additional info for Bayesian and Frequentist Regression Methods

Example text

13) where ∂ Gn (θ) ∂θ T An = E Bn = E [Gn (θ)Gn (θ)T ] = var [Gn (θ)] . 9). 13)—the covariance of the estimating function, flanked by the expectation of the inverse of the Jacobian matrix of the transformation from the estimating function to the parameter—is one that will appear repeatedly. Estimators derived from an estimating function are invariant in the sense that if we are interested in a function, φ = g(θ), then the estimator is φn = g(θn ). The delta method (Appendix G) allows the transfer of inference from the parameters of the model to quantities of interest.

We assume conditional independence so that p(y | θ) = n er–Rao lower bound for any unbiased estimator φ of a i=1 p(yi | θ). 3) n where l(θ) = i=1 log p(yi | θ) is the log of the joint distribution of the data, viewed as a function of θ. If T (Y ) is a sufficient statistic of dimension 1, then, under suitable regularity conditions, there is a unique function φ(θ) for which a UMVUE exists and its variance attains the Cram´er–Rao lower bound. Further, a UMVUE only exists when the data are independently sampled from a one-parameter exponential family.

9) k=1 Suppose we only measure xi1 , . . 8). Viewing the error terms as sums of random variables and considering the central limit theorem (Appendix G) naturally leads to the normal distribution as a plausible error distribution. There is no compelling reason to believe that the variance of this normal distribution will be constant across the space of the x variables, however. 9), denoted βj . In general, βj = βj , because of the possible effects of confounding which occurs due to dependencies between xij and elements of zi = [zi1 , .