An Overview of Verification and Validation Methodology for CFD Simulation of Ship Hydrodynamics

WU Bao-shan

(China Ship Scientific Research Center,Wuxi 214082,China)

1 Introduction

Mendenhall,Childs and Morrison(2003)[1]summarized that CFD plays an essential role in the design and analysis of advanced aerospace vehicles,and has evolved from a research topic to an integral tool in aerospace design.Many aircrafts are designed on the computer and then validated in wind tunnel and flight tests.However,uncertainties in CFD simulation limit the ability to optimize aircraft performance and affect the performance of aerospace products.

In ITTC community,CFD simulation is usually called as virtual towing tank.In uncertainty analysis procedures for CFD is usually used the concepts from experimental uncertainty analysis for considering the errors and uncertainties in both the solution and the data.However,it must be kept in mind that CFD is not of measurement and the methodology of uncertainty analysis in measurement is not applicable to CFD.Considering resistance prediction of a specific ship model,anyone can obtain the same simulation result with the same code if the grid,iteration,turbulence model and input parameters in computation are the same.While,the measurement data are always different among repeated experiment runs even if the ship model test is performed by the same engineers and all the measurement system is kept unchanged.

It is early in 1986 that the first editorial policy statement on the control of numerical accuracy was published in the ASME Journal of Fluids Engineering.Although several standards or guidelines for uncertainty analysis in numerical simulation have been formed,even by now,‘uncertainty analysis in CFD is a controversial subject.…the distinction between Validation(solving the right equations or mathematical correctness)and Verification(solving the equations right or physical correctness)is now widely recognized and accepted,but for example,it is still under debate whether uncertainty quantification and error estimation are the same thing or not’.This statement in preface of the 1st Workshop on CFD Uncertainty Analysis,Lisbon 2004 is still fitting the state of the art.For example,the word‘Simulation’in Fig.1 may be more appropriate than the‘Prediction’.

Verification and validation(V&V)are the two main processes for assessing the credibility of modeling and simulations in CFD.Validation is the process of determining the degree to which a computation model is an accurate representation of the real world.Validation must be preceded by verification.This has led to developing standards and guides to address V&V in numerical modeling.AIAA and ASME published V&V guides.Organizations(say,ERCOFTAC)other than AIAA and ASME are currently in the process of developing V&V guides and standards.On the other hand,the determination of the degree of accuracy of a simulation result at a set point other than validation points is still an unresolved research area(Coleman et al,2009)[3].

This paper is aimed at generally reviewing the methodology of V&V to provide a guide for V&V in practical application of CFD in ship hydrodynamics,other than going deep and in detail into CFD computation itself.

2 Review of general guidelines for V&V in CFD

Although the ASME Journal of Fluids Engineering required stating the numerical accuracy as early as in 1986,no procedure had been defined then.In 1988,the Fluids Engineering Division of ASME formed the Coordinating Group on CFD whose focus was the driving force to develop guidelines,procedures,and methods for verification,validation,and uncertainty estimation.

The earliest-published general guidelines for UA in CFD are possibly the AIAA G-077-1988[4],in which is addressed the process of verifying simulation codes and verifying and validating calculations,including design of validation experiments.It is the synthesis of the pub-lished literature prior to 1998 on V&V in CFD,aiming to provide support to researchers,developers and users with a common basic terminology and methodology that establishes some common meaning which can be used to describe internally consistent processes of V&V.Francesca Iudicello stated in a review that the AIAA guidelines can help experienced CFD users setup V&V procedures for specific applications which can then be used by less experienced users to assess and improve confidence in CFD simulations and predictions.But as matter of fact,the purpose of this guide was to formalize definitions and basic methodology for V&V in CFD and however,it does not present techniques for estimating uncertainty.

Calculation errors are delineated by two definitions:

·Uncertainty:A potential deficiency in any phase or activity of the modeling process that is due to lack of knowledge.

·Error:A recognizable deficiency in any phase or activity of modeling and simulation that is not due to lack of knowledge.

AIAA guide is not intended for certification or accreditation of CFD codes.AIAA focuses its procedures for bounding and controlling calculation errors.Several terms are defined as follows:

·Verification:The process of determining that a model implementation accurately re-presents the developer’s conceptual description of the model and the solution to the model.

·Validation:The process of determining the degree to which a model is an accurate representation of the real world from the perspective of the intended uses of the model.

·Calibration:The process of adjusting numerical or physical modeling parameters in the computational model for the purpose of improving agreement with experimental data.

·Model:A representation of the physical system or process intended to enhance our ability to understand,predict,or control its behavior.

·Modeling:The process of construction or modification of a model.

·Simulation:The exercise or use of a model.

·Prediction:Use of a CFD model to foretell the state of a physical system under conditions for which the CFD model has not been validated.

The definition for verification stresses comparison with the reference standard‘conceptual model’,while for validation,the standard is the‘real world’.Calibration of simulation is a response to the degree of representation of the real world directed towards improvement of agreement.Calibration is commonly conducted before validation activities.The relationship between validation,calibration and prediction is illustrated as in Fig.2.(Oberkampf,2004)[5].

The newly issued standard for uncertainty analysis in CFD is the ASME V&V20-2009[6]on November 30,2009.A 9-members committee for this standard is formed in 2004 and chaired by Coleman.The V&V20 standard was introduced to the 3rd Workshop on CFD Uncertainty Analysis,Lisbon 2008 and adopted as validation procedure.According to Coleman and Steele(2009)[3]and Coleman(2008)[7-8],the V&V20 approach was initially proposed by Coleman and Stern(1997)[9]and originated from ONR Program 1996-2000 in which two RANS codes are used and experiments on models are carried out in three towing tanks in USA(DTMB,IIHR)and Italy(INSEAN).

By now,no methodology is available for prediction uncertainty analysis.Consideration of the accuracy of simulation at points other than the validation points is a matter of engineering judgment specific to each family of problems.

The ASME Standard uses the definitions of verification and validation that are consistent with those in AIAA guideline.

Verification is now commonly divided into two types:code verification and solution verification as defined as

·Code/Software verification:The process of determining that the numerical algorithms are correctly implemented in the computer code and of identifying errors in the software.

·Solution/Calculation verification:The process of determining the solution accuracy of a particular calculation.

Before uncertainty estimation,it is first to verify the code itself.Code verification is to determine the code is free of mistakes and directed towards[5]:

(1)Finding and removing mistakes in the source code;

(2)Finding and removing errors in numerical algorithms;

(3)Improving software using software quality assurance practices.

Solution verification is the process to estimate the numerical uncertainty required for the validation process.Solution verification activities are directed toward[5]:

(1)Assuring the accuracy of input data for the problem of interest;

(2)Estimating the numerical solution error;

(3)Assuring the accuracy of output data for the problem of interest.

The recommended approach for code verification of RANS solvers is the use of the Method of Manufactured Solution(MMS)(Eça et al,2005)[10].The MMS assumes a sufficiently complex solution form so that all the terms in the Partial Differential Equations(PDEs)are exercised.This particular technique is usually more of a developer’s tool,and code verification is commonly assumed to have been completed,especially,for those extensively-used commercial codes,although it must be kept in mind that code verification is not of the exclusive responsibility of code developers.

The validation in the ASME V&V20 is shown schematically in Fig.3.The validation comparison error,E,is defined as[3]

where,S is the simulation solution,D the experimental data,T the true value(unknown)of the reality of interest,δSthe error in the simulation solution andδDthe error in the experiment data.

The errorsδScan be composed of three categories of errors,

where,Sexactis the assumed analytical solution of the PDEs in simulation;δmodelis the error due to modeling assumptions and approximations;δnumis the error due to numerical solution of equations,andδinputis the error in the simulation result due to errors in the simulation input parameters.

The estimation of a range within which the simulation modeling error lies is a primary objective of the validation progress.Combining Eqs.(1)and(2),the modeling error can then be written as

Once S and D are determined,the sign and magnitude of E are known from Eq.(1).However,the signs and magnitudes of the errorsδnum,δinputandδDare unknown.The standard uncertainties corresponding to these errors are unum,uinputand uD.

A standard validation uncertainty is defined as the combination of these uncertainties

For validation of CFD in ship hydrodynamics,the input data and parameters,e.g.,water density and viscosity,are commonly set as the nominal value or assumed precisely known,uinputcan be assumed as null.The Eq.(4a)can then be simplified as

The model errorδmodelwill fall within the following interval

where,k is a coverage factor that is chosen on the basis of the level of required confidence of the interval.In general,k will be in the ranges 2 to 3,approximately k=2 for 95% and k=3 for 99%.

When the validation uncertainty is obtained,one can make the following statements:

The estimation of the validation uncertainty uvalor Uvalis at the core of V&V.The uncertainty of the benchmark data uDin Eq.(4b)is obtained from the corresponding experiment and while,the numerical uncertainty unumis estimated by solution verification.It is commonly accepted that the numerical error has three components:the round-off error;the iterative error and the discrimination error.In problems with smooth solutions,the round-off error becomes negligible with the use of double precision.In principle,the iterative errorδImay be reduced to the level of the round-off error,but,that may be excessively time consuming.Much less demanding convergence criteria than machine accuracy are generally adopted in practical calculations(Eça et al,2010)[11].

Grid refinement studies used in solution verification provide an estimate of the discrimination errorδGand,the most-widely-used estimation method is classical Richardson extrapolation(RE),δG≈δRE.Uncertainty estimates at a level of confidence 95% can then be calculated by Roache’s grid convergence index(GCI)(Roache，1997)[12]that is obtained by multiplying the(generalized)RE error estimate,δRE,by an empirically determined factor of safety,FS

in which,Siis the simulation solution by the ith grid,S0is the estimated exact solution(un-known),hiis a parameter representing the grid cell size,αand p are unknown constants.Therefore,at least three grids are required to determine the three unknowns(S0,αand p).

Generally,these grids must be geometric-ally similar and in the asymptotic range.Meanwhile,the iterative error should be reduced to negligible levels,i.e.,being 2 to 3 orders of magnitude smaller than the discretization error.

Considering a gird triplet,i.e.,there is a uniform refinement ratio between solutions,as

where,the grid size parameter may be

A convergence ratio is defined as

When 0

When more than 3 grids are used to estimate GCI,the least-squares method,which was pioneered by Eça and Hoekstra,is cited in ASME V&V20 as the most robust and tested method available for the prediction of numerical uncertainty as of this date.Three unknowns,S0,αand p,are simultaneously computed using a least squares root approach that minimizes the function

in which,nG>3.Then we obtain the RE error

and the fitting standard deviation uGs,

is regarded as one of the contributions of numerical uncertainty and included in the grid uncertainty UGin the way that is modified in this report

where,the factor of safety,FS,is chosen according to the so-called observed order of accuracy(rate of convergence),p,and maybe depends on the gird refinement ratio,r(usually=),e.g.,Eça et al(2003)[13],shown in Fig.4.

In the case of oscillatory convergence,i.e.,R<0 in Eq.(10),the grid uncertainty may be estimated by bounding the error based on the oscillation maximums SUand minimums SL.

It should be noted that the resulted uncertainty from Eq.(18a)is some kind of limit that may be approximately regarded as at a level of confidence 99%.The corresponding uncertainty at a level of confidence 95% is recommended in this report to be approximated as

As the last result,the validation uncertainty estimate at a level of confidence 95% is then obtained as

3 Practices in ITTC community

As early as in 1999,the 22nd ITTC recommended its interim procedure QM4.9-04-01-01 for uncertainty analysis in CFD simulation and provided an example of RANS code for resistance simulation in QM 4.9-04-01-02.The latest reversion of uncertainty analysis procedure for CFD is recommended by the 25th ITTC(2008)[14].Most part of these ITTC procedures are based on the excellent work of Prof.Fred Stern[15-19]from IIHR.Stern has been making one of the most significant contributions to ITTC in this field.

The ITTC procedure is very detailed for estimating the uncertainty in a simulation result.It is intended for practical use and presented in an easily-implemented way.The latest workshop on CFD-Gothenburg 2010 was held on 8-10 December 2010.It was the sixth of series of workshops since 1980(Larsson et al,2010)[20].Over 30 organizations have taken part in the activities of Gothenburg 2010.

Most of those organizations who had performed V&V and presented uncertainty results complied with the ITTC(2008)approach for uncertainty estimation of CFD simulation.There are several exceptions,e.g.

·VTT used 9 grids to estimate the numerical uncertainty by the least-squares method proposed by Eça and Hoekstra[6].

·SSPA:the proposed method of Eça and Hoekstra[21].

·Southampton/QinetiQ:the ASME V&V20.

Additionally,IIHR used the improved‘correction factor’method proposed by Xing and Stern(2008)[23]on the basis of the ITTC(2008)approach.Tab.1 shows some V&V results given by Gothenburg 2010 Workshop.As comparison,two V&V results by Roache’s GCI method are put in the shadowed area of this table.It is shown that it is not easy to draw conclusion from different computations and different V&V methods.

Tab.1 V&V Results for resistance coef.of KVLCC2

One of distinguishable aspects in the ITTC(2008)procedure is the introduction of the correction factor verification method that was proposed by Stern et al(2001)[16],although there is still some argument on it(Wilson et al,2004)[24].The correction factor can be considered just as one of alternatives for the safety factor(Roache,1997)[12]which is much less complex than the former.However,the least-squares method proposed by Eça and Hoekstra,which is based on the Richardson extrapolation,is not included in the ITTC(2008)approach.

The‘correction factor’is proposed in the ITTC procedure to account for the effects of higher-order terms in truncation and the error is defined as

where FSCis another factor of safety chosen with C.

The introduction of the correction factor was based on the observation that the estimate for the discretization error of Richardson extrapolation has the correct form and that the observed order of accuracy is only poorly estimated when the three grids are in the asymptotic range.As in the original statement of Richardson(1927)for a second-order method,the Richardson extrapolation is theoretically as

where,pestis the estimate of the order of accuracy,and pest=2 for a second-order method.For a practical problem,the grid size is not an infinitesimal,the higher-order effects lead to the observed order of accuracy that is different from the estimate.Define the correction factor C of the Richardson extrapolation as

and then,the corrected uncertainty is conceptually defined as

Additionally,a novel concept introduced in the ITTC(2008)procedure other than AIAA and ASME guides is that the numerical errorδnumis divided into two components

If the input and iterative errors are omitted,the corrected simulation error is represented as

Then,the corrected validation uncertainty is

and the model error can be rewritten as

However,considering Eq.(12)can be rewritten as

We can conclude that introduction of the novel errorwill lead to the same estimate of the numerical uncertainty as that of Eq.(11)when Richardson extrapolation is used.But the model error estimate resulted from Eq.(28)may be different from that of Eq.(5).

4 Proposals for the future of V&V in ITTC

It is shown that the above V&V analysis is mainly focused on verification for estimating the numerical uncertainty.The common practice in validation is to compare different computations to the same specific benchmark data,which is much similar to that of evaluation of the inter-laboratory bias.The author would like to draw attention to another emphasis of the V&V activities,that is,it is of the most importance for code users to make a credible prediction of a new ship design and for clients to be assured that a credible prediction is provided,shown in Fig.5.

The underlying aim of V&V seems somehow to make the CFD simulation accurately represent the reality.But it should be kept in mind,at least for RANS-based codes,the turbulence model is impossible to simulate accurately the reality.This is some like the situation that the model tests is impossible to model the full scale ship performance as the Reynolds number and Froude number can not be the same to full scale at the same time.From practical application and the viewpoint of users,a stable prediction of CFD simulation,i.e.,it can be approximately and reliably corrected if required and applicable,is preferred to a possibly-accurate but quite unsure prediction.

Take the resistance prediction by RANS-based simulation as an example.As the state of the art,CFD simulation of model test is commonly performed.Comparison between simulation results and model tests,or denoted as numerical-experimental correlation in this paper,is much analogous to the model-ship correlation,as shown in Fig.6.Validation of simulation can be treated as that of model-ship correlation.In the numerical correlation analysis,series of ship model data should be used for validation to obtain the desired correlation factor or allowance for prediction,and in addition,one organization should choose a specific code,type of grid,turbulence model and parameters and V&V method for the corresponding specific problem,just like in the model tests that the same procedure and measurement system are used for series tests.It is better to choose the type of grid(including cells geometry)that lead to the monotonic convergence with grid refinement so that the estimates of uncertainty is consistent with each other and a reliable correlation may be obtained.

After the numerical uncertainties have been assessed by verification,the numerical-experimental correlation will mainly determine the deviation of the turbulence model used in CFD simulation from the reality that experimental data represent,shown in Fig.6.

5 Conclusive remarks

The verification and validation in CFD are an area of current research,and there is no universally accepted single method.Just as Coleman said in his post-workshop comments on 3rd Workshop on CFD Uncertainty Analysis,Lisbon,2008,‘it is evident that there is not a‘one-size-fits-all’V&V approach that will be applied in all situations across all areas considered by scientists and engineers’.

As AIAA and ASME guidelines are extensively adopted,there is a consensus on V&V terminology and methodology,however,the estimate of uncertainty in CFD simulation is still under way.For an example,the choice of an appropriate factor of safety in verification is a matter of an ongoing discussion,especially the question in which factor would provide a 95%confidence level for estimate of the numerical uncertainty[33].

V&V is not the final purpose of CFD studies.As numerous experiences and advances in CFD are added to the general knowledge base,an era of virtual towing tanks is coming nearer and nearer.Meanwhile,the CFD users community highly desire the development of specific best-practices for CFD prediction,other than V&V in CFD,that can be broadly applied by users to improve the accuracy and reduce the uncertainty of CFD results and also reduce the time and cost associated with CFD use[1].They are more or less confused in practice because there are many CFD algorithms,solvers,and codes,each with potentially different requirements for best use.It is recommended in this paper that accumulation of application in practical prediction and systematic validation by series of model tests is the best road to develop the specific best-practices guideline for a specific application in numerical towing tanks.

[1]Mendenhall M R,Childs R E,Morrison J H.Best practices for reduction of uncertainty in CFD results[R].(Invited),AIAA,2003.

[2]Thacker B H.ASME standards committee on verification and validation in computational solid mechanics[R].2005.

[3]Coleman H W,Steele W G.Experimentation,validation,and uncertainty analysis for engineers[M].3rd Edition.A John Wiley & Sons,INC,2009.

[4]AIAA.Guide for the verification and validation of computational fluid dynamics simulations[K].AIAA G-077-1998,AIAA,New York,1998.

[5]Oberkampf W L.Verification and validation in computational simulation[C]//2004 Transport Task Force Meeting.Salt Lake City,Utah,2004.

[6]ASME.Standard for verification and validation in computational fluid dynamics and heat transfer[S].V&V20-2009,2009.

[7]Coleman H W.An Overview of ASME V&V 20:Standard for verification and validation in computational fluid dynamics and heat transfer[C]//Presentation at the 3rd Workshop on CFD Uncertainty.Lisbon,Portugal,2008.

[8]Coleman H W.An Overview of ASME V&V 20:Standard for verification and validation in computational fluid dynamics and heat transfer[C]//8th World Congress on Computational Mechanics(WCCM8).5th European Congress on Computational Methods in Applied Sciences and Engineering(ECCOMAS 2008),Venice,Italy,2008.

[9]Coleman H W,Stern F.Uncertainty and CFD code validation[J].Journal of Fluids Engineering,1997,119:635-636.

[10]Eça L,Hoekstra M,Pellertier A H D.A manufactured solution for a two-dimensional steady wall-bounded incompressible turbulent flow[R].EPM-RT-2005-08,2005.

[11]Eça L.Uncertainty quantification for CFD[K].MARIN,2010.

[12]Roache P J.Quantification of uncertainty in computational fluid dynamics[J].Annu.Rev.Fluid.Mech.,1997,29:123-160.

[13]Eça L,Hoekstra M.An example of uncertainty estimation in the calculation of a 2-D turbulent flow[C]//MARNETCFD Final Workshop,March 2003.Haslar,2003.

[14]ITTC.Uncertainty analysis in CFD,uncertainty assessment methodology and procedures[K].ITTC Recommended Procedures and Guidelines,Procedure 7.5-03-01-01,Revision 02,2008.

[15]Coleman H W,Stern F.Uncertainty and CFD code validation[J].Journal of Fluids Engineering,1997,119:635-636.

[16]Stern F,Wilson R V,Coleman H W,Paterson E G.Verification and validation of CFD simulations[R].IIHR Report No.407,1999.

[17]Stern F,Wilson R V,Coleman H,Paterson E.Comprehensive approach to verification and validation of CFD simulations-Part 1:methodology and procedures[J].Journal of Fluids Engineering(ASME),2001,123:793-802.

[18]Stern F,Wilson R,Shao J.Quantitative V&V of CFD simulations and certification of CFD codes[J].Int.J Numer.Meth.Fluids,2006,50:1335-1355.

[19]Wilson R,Shao J,Stern F.Discussion:Criticisms of the‘correction factor’verification method[J].Transactions of the ASME,2004,126.

[20]Larsson L,Stern F,Visonneau M.Proceedings of 6th Workshop on Numerical Ship Hydrodynamics[C].ISBN 1652-9189,Vol.2,Chalmers University of Technology,Gothenburg,Sweden,2010.

[21]Eça L,Hoekstra M.Overall comments on the workshop[C]//3rd Workshop on Uncertainty Analysis.Lisbon,2008.

[22]Eça L.Verification and validation in Computational Fluid Dynamics[Z].Cobem 2009,Engineering for the Future,Brazil,2009.

[23]Xing T,Stern F.Factors of safety for Richardson extrapolation for industrial applications[R].IIHR Technical Report No.466,2008.

[24]Wilson R,Shao J,Stern F.Discussion:Criticisms of the‘Correction Factor’verification method[J].Transactions of the ASME,2004,126.

[25]ITTC.Discussion to the 25th ITTC Resistance Committee by Tao Xing and Fred Stern[C]//Proceedings of the 25th ITTC.Fukuoka,Japan ITTC,2008,3:671-686.

[26]ASME.Guide for verification and validation in computational solid mechanics[K].V&V10-2006,2006.

[27]BIPM.Evaluation of measurement data-Guide to the expression of uncertainty in measurement[K].First Edition.JCGM100:2008.

[28]Faragher J.Probabilistic methods for the quantification of uncertainty and error in Computational Fluid Dynamics simulations[Z].DTSO-TR-1633,Commonwealth of Australia,2004,AR-013-229.

[29]Freitas C J.The issue of numerical uncertainty[C]//Second International Conference on CFD in the Minerals and Process Industries,CSIRO.Melbourne,Australia,1999.

[30]Putko M M,Newman P A,Arthur C,Taylor III A C,Green L L.Approach for uncertainty propagation and robust design in CFD using sensitivity derivatives[C].AIAA 2001-2528.

[31]Roache P J.Recent developments in verification and validation[C]//2nd Workshop on CFD Uncertainty Analysis.Lisbon,2006.

[32]Neel R E,Godfrey A G,Slack D C.Turbulence model validation in GASP version 4[C]//AIAA 2003-3740,33rd AIAA Fluid Dynamics Conference and Exhibit,23-26 June 2003.Orlando,FL,USA.

[33]Franke J,Hellsten A,Schlinzen H,Carissimo B.Quality assurance and improvement of Microscale Meteorological Models[C].COST Action 732,Best Practice Guideline for the CFD Simulation of Flows in the Urban Environment.1 May 2007.

Appendix:List of acronyms

AIAA American Institute of Aeronautics and Astronautics

ASME American Society of Mechanical Engineers

BIPM Bureau International des Poids et Mesures

(International Bureau of Weights and Measures)

CFD Computational Fluid Dynamics

DTMB David Taylor Model Basin

ERCOFTAC European Research Community On Flow,Turbulence And Combustion

GCI Grid Convergence Index

GUM Guide to the Expression of Uncertainty in Measurement

HSVA Hamburgische Schiffbau-VersuchsAnstalt GmbH

(Hamburg Ship Model Basin)

IIHR Iowa Institute of Hydraulic Research

INSEAN Istituto Nazionale per Stud ed Esperienze di Architettura Navale

(Italy Ship Model Basin)

ISO International Organization for Standardization

ITTC International Towing Tank Conference

JCGM Joint Committee for Guides in Metrology

KVLCC Korea Very Large Crude-Oil Carrier

MARIC MArine Design and Research Institute of China

MOERI Marine and Ocean Engineering Research Institute/KORDI,Korea

NTNU Norges Teknisk-Naturvitenskapelige Universitet/Norway

(Norwegian University of Science and Technology)

RANS Reynolds Averaged Navier-Stokes Equations

SSPA Statens Skepps-Provnings Anstalt

(Sweden Ship Model Basin)

V&V Verification and Validation

VTT Valtion Teknillisesta Tutkinmuslaitoksesta

(Technical Research Center of Finland)