Model comparison

We can outline three general principles guiding model-based inference in science:

• Simplicity and parsimony (Occam’s razor). Model selection is a classic bias-variance trade-off.

• Multiple working hypotheses (Chamberlin, 1890). At any point in time there must be several hypotheses (models) under consideration, but number of alternatives should be kept small.

• Strength of evidence. Providing quantitative information to judge the strength of evidence is central to science (e.g., see Royall, 1997).

Steps of the model selection approach consist in establishing a set of R relevant models, then, based on data, ranking models, choosing the best model, and making inferences from this best model. To rank models, we need tools that account for the basic principles that makes a model a good model.

Information criteria

Akaike weights and evidence ratios

The individual AIC values are not interpretable in absolute terms as they contain arbitrary constants and are much affected by sample size. Then it is imperative to rescale these criteria. Usually, this is done by substracting to the AIC of each model the AIC of the model with the minimum one:

Δ_AIC = AIC_i − AIC_min

where AIC_min is the minimum of the R different AIC_i values. This transformation forces the best model to have Δ_AIC = 0, while the rest of the models have positive values.

For the AIC, the simple transformation exp(−Δ_i/2), for i = 1, 2, ..., R, provides the likelihood of the model given the data. This is a likelihood function over the model set in the sense that ℒ(θ|data, g_i) is the likelihood over the parameter space (for model g_i) of the parameter θ, given the data (x) and the model (g_i).

It is convenient to normalise the model likelihoods such that they sum to 1 and treat them as probabilities. Hence, we use:

$$w_{i}=\dfrac{exp(-\Delta_{i}/2)}{\sum_{r=1}^{R}exp(-\Delta_{r}/2)}.$$

The w_i, called Akaike weights, are useful as the weight of evidence in favor of model g_i(⋅|θ) as being the actual Kullback-Leibler best model in the set. The ratios w_i/w_j are identical to the original likelihood ratios, ℒ(g_i|data)/ℒ(g_j|data), and so they are invariant to the model set, but the w_i values depend on the full model set (because they sum to 1).

Evidence can be judged by the relative likelihood of model pairs as ℒ(g_i|x)/ℒ(g_j|x) or, equivalently, the ratio of Akaike weights w_i/w_j. Such ratios are called evidence ratios and represent the evidence about fitted models as to which is better in a Kullback-Leibler information sense (i.e., the best predictive model).

Consistency versus Efficiency (true versus best model)

The Sequential Bayes Factor (SBF) procedure discussed above is useful in addressing the question of whether we have accumulated enough evidence for a model (relatively to another model), given the data we observed and the prior predictions of each model. Moreover, the SBF is a consistent procedure for model identification. In other words, if the true model is in the set of models, the Bayes Factor (or equivalently, the BIC) will select it when the sample size increases.

However, one could question whether it is sensible to assume a true model (an oxymoron) in “real life” (i.e., except in simulations), especially in the social sciences (e.g., Burnham & Anderson, 2002; 2004). As Findley (1985) notes: “[…] consistency can be an undesirable property in the context of selecting a model”. In other words, the goal of identifying the true model works fine… when there is a true model to be identified.

The AIC is not consistent for model identification, but it is an efficient estimator of the expected K-L information loss.

As a consequence, when using information criteria (e.g., AIC, WAIC) for sequential testing, we try to answer a different question. The question is not anymore when do I have enough evidence for this model over this other model being the true model. Instead, the question is: when do I have enough evidence for this model being the best model ?

Observed data

On the other hand, ESTER can be used to do sequential testing on your own data. You can study the evolution of sequential ERs using the seqtest function.

data(mtcars)

mod1 <- lm(mpg ~ cyl, mtcars)
mod2 <- lm(mpg ~ cyl + disp, mtcars)

plot(seqtest(ic = aic, mod1, mod2, nmin = 10) )

In addition, seqtest allows studying the behavior of sequential ERs computed on your own data, along with sequential ERs computed on permutation samples. This feature might be useful to study to what extent the evolution of evidence ratios you observed on the original sample is dependant to the order of incoming observations.

plot(seqtest(ic = aic, mod1, mod2, nmin = 10, nsims = 10) )

When data involve repeated measures (and so multiple lines by subject), you should help ESTER to figure out how the data is structured by indicating in which column are stored subject or observations numbers. For instance, below we use data from lme4, and indicate that the subject number is stored in the "Subject" column of the dataframe, by passing it to the id argument of seqtest.

library(lme4)

## Loading required package: Matrix

data(sleepstudy)

mod1 <- lmer(Reaction ~ Days + (1|Subject), sleepstudy)
mod2 <- lmer(Reaction ~ Days + I(Days^2) + (1|Subject), sleepstudy)

plot(seqtest(ic = aic, mod1, mod2, nmin = 10, id = "Subject", nsims = 10) )

If no information is passed to the id argument, seqtest will suppose that there is only one observation (i.e., one line) per subject.