Models in HTA: a tradeoff between goodness of fit and generalizability?

Abstract:
Analysis of data from clinical studies is often based on models with a simple structure that aim to “let the data speak for themselves”.  Model selection criteria aim for a balance between goodness of fit to the sample data and predictive ability. Decision models aim to be explanatory and predictive, and often have a complex structure. “By marvelous coincidence” these models have one data point per parameter and model selection criteria rarely measure goodness of fit or penalize complexity, instead using subjective criteria such as robustness and plausibility.

Both approaches might benefit from a more integrated framework. The simplicity of clinical models (e.g. PFS) may sometimes obscure some implicit assumptions, particularly when there are multiple outcomes. These can lead to internal bias. There is a choice of endpoints and censoring events, and these may not be independent. K-M curves of single events may be biased when there are competing risks. Observation times may be interval censored because of arbitrary observation times. There may be external bias, for example, in survival time post-progression.

The complexity of structure in decision models may give false sense of precision, so that the model is “precisely wrong” or overfitted. Data may be inadequate to estimate unbiased conditional probabilities for infrequent events. A simpler structure might give better predictions, though with greater uncertainty.

Multistate models such as msm in R can estimate multiple outcomes in a similar integrated framework to a decision model. This may allow greater transparency and consistency, and avoid some of the internal and external biases that can occur when these analyses are carried out separately. The same model can be used for parameter estimation and for prediction. The framework can make more efficient use of IPD, using all the patient history. The model can deal with interval censoring. An integrated package can measure goodness of fit (e.g. likelihood) and complexity (number of parameters) and hence AIC. However, msm is limited by assuming constant baseline hazards.