Verification and validation using state of the art measures and modular uncertainty techniques

Abstract

As quantitative validation measures have become available, so has the controversy regarding the construction of such measures. The complexity of the physical processes involved is compounded by uncertainties introduced due to model inputs, experimental errors, and modeling assumptions just to name a few. Also, how these uncertainties are treated is of major importance. In this dissertation, the issues associated with several state of the art quantitative validation metrics are discussed in detail. Basic Verification and Validation (V&V) framework is introduced outlining areas where some agreement has been reached in the engineering community. In addition, carefully constructed examples are used to shed light on differences among the state of the art validation metrics. The results show that the univariate validation metric fails to account for correlation structure due to common systematic error sources in the comparison error results. Also, the confidence interval metric is an inadequate measure of the noise level of the validation exercise. Therefore, the multivariate validation metric should be utilized whenever possible. In addition, end-to-end examples of the V&V effort are provided using the multivariate and univariate validation metrics. Methodology is introduced using Monte Carlo analysis to construct the covariance matrix used in the multivariate validation metric when non-linear sensitivities exist. Also, the examples show how multiple iterations of the validation exercise can lead to a successful validation effort. Finally, modular uncertainty techniques are introduced for the uncertainty analysis of large systems where many data reduction equations or models are used to examine multiple outputs of interest. In addition, the modular uncertainty methodology was shown to be an equivalent method to the traditional propagation of errors approach with a drastic reduction in computational effort. The modular uncertainty technique also has the advantage in that insight is given into the relationship between the uncertainties of the quantities of interest being examined. An extension of the modular uncertainty methodology to cover full scale V&V exercises is also introduced.