In reviewing the results of any model, the reviewer should consider the following points and potential risk areas, asking questions to clarify understanding as much as necessary to insure that outputs are evaluated with appropriate caution, and any caveats or qualifications to the results are duly noted.

How was the model designed? Is it a well-accepted, standard industry model or was it developed in-house? Are there any potential errors in the design of the model itself? The risks inherent in the models include pure faulty design, where the model itself does not properly describe or mirror the reality or problem it is supposed to.
What specific purpose does the model serve? What is it supposed to show? Sometimes mathematical formulas or equations are used for more than one purpose, for an issue or problem that seems to be similar to a second issue/problem, but the linkage might not, in fact, be as close as the modeler believes it to be. Running data through the model for both scenarios might magnify variances or differences between the two, giving less reliable results for the second issue than what the model was originally built for. For example, a model which extrapolates out personal automobile losses from historical trends might not be an exact fit to run commercial auto losses in the same way, if any of the drivers and causes (potential inputs into the model) of commercial auto losses are not exactly the same as for personal auto claims.
What inputs or assumptions were made at the front end of the modeling process? Were they reasonable, clear, well-documented and agreed by the reviewing team? For example, who decides what a "representative sample" would be to run through a model, and what were the parameters of that decision? If the sample was not large enough, or not representative enough of the examined data set, outputs may be skewed.
Companies must also regularly update, refine and enhance each of their models to reflect changing market conditions or shifting company goals, and evolve with the organization's risk landscape. Have changes been made over time? How and by whom? Were the changes authorized? Were they made to correct inaccuracies or faults in prior versions?
Has the model been quality reviewed, or the "math" double-checked? While many models in the insurance world are now highly complex and thus system-run, some modeling is still done either by hand, or via formulas in spreadsheets, and can be ripe for human error (such as cutting and pasting errors, wrong formulas, missing or hidden data fields). Don't rule out software or hardware bugs either, until a system has been well tested and many simulations are done.
Complex models may generate complex or ambiguous results which may require particular expertise to evaluate. Is the output data being properly interpreted? Are the people charged with interpreting results adequately trained or qualified to do so?
Can results be taken out of context? Results in pure numerical form may mean nothing if not put in the appropriate context. For example, are you speeding if you are driving at a speed of 55 mph? Yes, if you are in a school zone with posted limits of 20 mph; no, if you are on the highway with a 65 mph limit. Facts alone are not "good" or "bad." Appropriate context needs to be provided to determine the actual impact of a pure number or mathematical result, and this applies to model outputs as well.

Minimizing Model Risk

Special Reports /

The Risks in ERM Risk Models

Related Stories

Recommended For You