The Model Validator’s Manifesto

Read­ing time: 4 mins

Vic­to­ry Idowu pro­pos­es a mod­el validator’s man­i­festo— prin­ci­ples for mod­el val­i­da­tion that enshrine best prac­tice for actu­ar­i­al mod­els.

The actuary’s role in mod­el­ling is evolv­ing and trans­form­ing. As mod­el com­plex­i­ty has increased, the need for robust val­i­da­tion sim­i­lar­ly increas­es. The mod­ern-day actuary’s con­tri­bu­tion is advanc­ing to include inde­pen­dent over­sight and effec­tive chal­lenge in the process of mod­el­ling.

Mod­el val­i­da­tion has gained pop­u­lar­i­ty due the increased reliance on many types of mod­els through­out organ­i­sa­tions. Reg­u­la­tors man­date the need for robust and reli­able val­i­da­tion of cer­tain mod­els: inter­nal cap­i­tal mod­els, reserv­ing mod­els and pric­ing mod­els with the pri­ma­ry pur­pose of reas­sur­ing soci­etal and investor con­fi­dence. While mod­els will always have inher­ent lim­i­ta­tions, the aim of val­i­da­tion is not to dis­re­gard these lim­i­ta­tions but to pro­vide con­fi­dence in the use of the mod­el giv­en the cur­rent knowl­edge and avail­able infor­ma­tion.

Val­i­da­tion is a process requir­ing sev­er­al steps. Val­ida­tors are expect­ed to know the extent of val­i­da­tion required for each mod­el, often referred to as the val­i­da­tion scope. This often depends on a model’s use: its mate­r­i­al impor­tance in the busi­ness, the fre­quen­cy of mod­el usage and the num­ber of users.

As actu­ar­ies rep­re­sent a data-based pro­fes­sion, clear rules and good val­i­da­tion is need­ed. Some good work has already been done; for exam­ple, the UK Mod­el Risk work­ing group wrote an extend­ed dis­cus­sion on mod­el risk and use in 2015[1].

Oth­er groups have tak­en this idea even fur­ther. The Finan­cial Modeller’s Man­i­festo was pro­duced by Emanuel Der­man and Paul Wilmott. It is a Hip­po­crat­ic Oath for all finan­cial pro­fes­sion­als and it encour­ages more respon­si­bil­i­ty in quan­ti­ta­tive finance and risk man­age­ment as a rejoin­der to the sub­prime mort­gage cri­sis.

In a some­what sim­i­lar humor­ous and wit­ty vein, I present the Mod­el Validator’s Man­i­festo and Hip­po­crat­ic Oath:

The Mod­el Validator’s Hip­po­crat­ic Oath 

  1. I will avoid val­i­dat­ing mod­els that I am a par­ent or emo­tion­al­ly attached too.
  2. I will make sure I have a good enough under­stand­ing of the whole gov­er­nance process before val­i­dat­ing it.
  3. I will use eas­i­ly repeat­able checks in my val­i­da­tion so that can be under­stood by a lay per­son.
  4. I promise to give an hon­est and fair view even if it makes me unpop­u­lar.
  5. I will avoid the use of unnec­es­sary tech­ni­cal lan­guage in prov­ing that my mod­el is right.
  6. I will not be short-sight­ed and dis­card mod­els which will be use­ful in oth­er aspects of the busi­ness.

The ideas dri­ving this Hip­po­crat­ic Oath is described in greater detail below:

  1. Inde­pen­dent Over­sight - I will avoid val­i­dat­ing mod­els that I am a par­ent or emo­tion­al­ly attached too.

It is a fact that mod­ellers eas­i­ly fall in love with their mod­els. This is also men­tioned in The Finan­cial Mod­ellers’ Man­i­festo. The lack of inde­pen­den­cy from the mod­el is one of the pri­ma­ry caus­es for mod­el fail­ures. In order for a val­ida­tor to care­ful­ly eval­u­ate the bor­ders and caveats of a mod­el, they should not be involved in its cre­ation.

  1. Mod­el Gov­er­nance - I will make sure I have a good enough under­stand­ing of the whole gov­er­nance process before val­i­dat­ing it.

There is no pur­pose of a func­tion­ing mod­el that has no-one to be account­able for it! Good gov­er­nance involves named indi­vid­u­als who have respon­si­bil­i­ty for the over­sight and devel­op­ment of the mod­el.

  1. Out­comes Analy­sis - I will use eas­i­ly repeat­able checks in my val­i­da­tion so that can be under­stood by a lay per­son.

Sim­ple checks are prefer­able to com­plex algo­rithms in the val­i­da­tion process. The process­es in val­i­da­tion should be kept as sim­ple as pos­si­ble and well explained so future team mem­bers will be able to repro­duce and under­stand the out­comes.

  1. Con­cep­tu­al Sound­ness - I promise to give an hon­est and fair view even if it makes me unpop­u­lar.

A good organ­i­sa­tion­al cul­ture will enable effec­tive chal­lenge between val­ida­tors and mod­el cre­ators. The val­ida­tor should not be inti­mat­ed by the pos­si­bil­i­ty of reject­ing the con­cep­tu­al sound­ness of an under­ly­ing mod­el due to pos­si­bil­i­ty of “falling out” with the mod­el cre­ators.

  1. Ter­mi­nol­o­gy Con­sid­er­a­tions - I will avoid the use of unnec­es­sary tech­ni­cal lan­guage in prov­ing that my mod­el is right.

In addi­tion to an out­comes analy­sis, it is use­ful for the val­ida­tor to pro­duce a brief report ver­i­fy­ing the valid­i­ty of their checks. Unnec­es­sary jar­gon and tech­ni­cal lan­guage which do not add val­ue to the val­i­da­tion process should be avoid­ed. Again, the read­abil­i­ty of the gov­er­nance doc­u­men­ta­tion should also be addressed with the same cri­te­ria.

  1. Expec­ta­tion Man­age­ment - I will not be short-sight­ed and dis­card mod­els which will be use­ful in oth­er aspects of the busi­ness.

The val­ida­tor should not be ruth­less with their empow­ered posi­tion of objec­tiv­i­ty. It is expect­ed that high­ly com­plex and mate­r­i­al mod­els use more labour hours in their cre­ation and should have more time spent on their val­i­da­tion.

Accord­ing­ly, the val­ida­tor should be crit­i­cal and deep in the val­i­da­tion of a mod­el ensur­ing they place lim­i­ta­tions in con­text of the vest­ed inter­est of stake­hold­ers.

As polit­i­cal and finan­cial uncer­tain­ty grows, the call on actu­ar­ies to val­i­date mod­els will like­ly increase. Val­ida­tors are at the fore­front of change in risk man­age­ment with addi­tion­al empha­sis being placed on sound gov­er­nance process­es.

Sim­i­lar­ly, in acad­e­mia many fields see active research into bet­ter quan­tifi­ca­tion of uncer­tain­ty. My research adds to this by look­ing at mod­els and the risks that they pose. Bet­ter mod­el man­age­ment will reduce fail­ures and sys­temic shocks. Hav­ing val­ida­tors and risk actu­ar­ies adhere to the prin­ci­ples of this Hip­po­crat­ic Oath will help achieve this. Mod­els, like actu­ar­ies, should be under­stood and not under­es­ti­mat­ed.

[1] Mod­el Risk – Dar­ing to open up the black box

CPD Actuaries Institute Members can claim two CPD points for every hour of reading articles on Actuaries Digital.

About the author

Victory Idowu

Victory Idowu is a PhD Student specialising in actuarial research at the London School of Economics and Political Science.

Comment on the article (Be kind)

Likes:1
Comments:0
Print

No Comments

Also this month