What we mean by UQ, and problems of interest to us.
What is UQ?
The term “Uncertainty Quantification”(UQ) is rather vague, and could describe much of what is done within Statistics. Nevertheless, we use the term to refer quantifying uncertainty in a (typically) deterministic mathematical model of a physical system: a model built based on an understanding of the underlying physical process, as opposed to a purely empirical or statistical model fitted to observed data. Often, such models are built precisely because it’s not practical to do physical experiments to get the input-output data. We sometimes refer to such models as simulators to distinguish them from other models used within UQ methodology.
Sources of uncertainty in a simulator prediction
- Uncertainty in simulator inputs parameters
To run a simulator, the user may have to specify particular input values, the values of which may be uncertain. For example, a health economic decision model may predict long term costs and health outcomes for a population of patients given a particular drug. simulator inputs may include quantities such as the proportion of patients ‘responding’ to the drug, which would be uncertain. Rather more pernicious is the problem of parameters within the simulator that have been estimated, and then hard-coded and forgotten about. A comprehensive UQ will need to deal with these too.
- Simulator discrepancy
Even if we could known all the ‘true’ values of the simulator inputs/parameters, the simulator output typically will not give the correct value of the corresponding real-world quantity. This may because simplifications in the simulator structure have been necessary, or the underlying physical process may not be well enough understood to model it ‘correctly’. Errors may also be introduced through numerical approximations used within the simulator.
Problems of interest
- Uncertainty and sensitivity analysis of simulator inputs
Representing the simulator by a function \(f\), the uncertainty analysis problem is to quantify uncertainty about the outputs \(Y=f(X)\) given uncertainty about the inputs \(X\). This is sometimes referred to as forward uncertainty propagation. By sensitivity analysis, for a vector input \(X=(X_1,\ldots,X_d)\), we mean the process of investigating how elements of \(X\) contribute (either individually, or in groups) to uncertainty in \(Y\).
- Calibration and history matching
Also described as the inverse problem, here we have observations of the physical process, and the aim is to infer the values of the uncertain simulator inputs; to simulator model input values such that the corresponding simulator outputs match the physical observations as closely as possible. simulator discrepancy plays an important role here, to avoid overconfident parameter inferences and calibrated simulator predictions.
Emulators for computationally expensive simulators
A complication is when the simulator is computationally expensive: it takes too long to run the simulator at all the desired input values in our analyses. In this case, an intermediate step is to build a fast approximation for \(f\), based on the simulator runs that we are able to perform. Such an approximation may be known as a surrogate model or meta-model. We use the term emulator to mean a full probability distribution for \(f\), so that we can both estimate \(f(x_1),\ldots,f(x_n)\) at any desired set of inputs \(x_1,\ldots,x_n\) and quantify joint uncertainty in the predictions. Hence a comprehensive UQ will also account for uncertainty due to a limited number of simulator runs. Building an emulator is a regression problem, with a popular technique being nonparametric regression using Gaussian processes.
Although it can be helpful to first define the field of UQ in terms of deterministic simulators, the same problems and methodoloy apply to stochastic simulators too. Typically, stochasticity arises from an internal Monte Carlo process conducted within the simulator.
These themes are discussed in more detail in the MUCM Toolkit. Some example papers (involving us) are as follows.
Uncertainty and sensitivity analysis
- Bounceur, N., Crucifix, M., & Wilkinson. (2015). Global sensitivity analysis of the climate vegetation system to astronomical forcing: an emulator-based approach. Earth Syst. Dynam. Discuss, 6, 205–224.
- Strong, M., Oakley, J. E., & Brennan., A. (2014). Estimating multi-parameter partial Expected Value of Perfect Information from a probabilistic sensitivity analysis sample: a non-parametric regression approach. Medical Decision Making, 34(3), 311–26.
Calibration and model discrepancy
- Oakley, J. E. and Youngman, B.D. (2016) Calibration of stochastic computer simulators using likelihood emulation. To appear in Technometrics.
- Strong, M., & Oakley, J. E. (2014). When is a model good enough? Deriving the expected value of model improvement via specifying internal model discrepancies. SIAM/ASA Journal On Uncertainty Quantification, 2(1), 106–125.
- Holden, P. B., Edwards, N. R., Garthwaite, P. H., & Wilkinson. (2015). Emulation and interpretation of high-dimensional climate model outputs. Journal Of Applied Statistics, 42, 2038–2055.
- Fricker, T., Oakley, J. E., & Urban, N. M. (2013). Multivariate Gaussian process emulators with nonseparable covariance structures. Technometrics, 55(1), 47–56.