Stress testing and COVID 19

Marketfox Columnist Daniel Grioli examines whether stress testing helped in navigating the fallout of the coronavirus pandemic and whether there is a better way of dealing with such events.

How many superannuation funds modelled the risk of a global pandemic on their portfolios? My guess is that many didn’t. Institutions typically model scenarios:

• That have already happened (e.g. Lehman default, European sovereign debt crisis).
• Where there is a broad consensus that a future scenario is likely (e.g. climate change).
• Represent the institution’s ex-ante expectations of the “worst case”.

Global pandemics have occurred before – the Spanish Flu and the Bubonic Plague being notable examples. But we are removed by generations or centuries from these events. As devastating as these pandemics were, their consequences had long faded from our collective consciousness.

That’s not to say that superannuation funds were totally unaware of the risk. For example, they all needed a pandemic plan to meet their ongoing regulatory requirements. The Australian Prudential and Regulatory Authority (APRA) had issued a pandemic planning guide (CPG 233) to assist institutions in preparing for the risks to business continuity posed by a pandemic (supporting compliance with Prudential Standards CPS 232 and SPS 232).

It’s one thing to have a plan for your staff to work from home. It’s another thing to understand how a pandemic will affect your investment portfolio and consequently your members’ future retirement outcomes.

Most commonly used risk assessment tools require ex-ante identification of the risk to be managed or avoided. These tools are useless if you don’t identify the risk first.

I fear that most funds failed to stress test their investment portfolios against a pandemic. Even if they had, the stress tests wouldn’t have helped. What prompts me to make such a bold claim? The limitations inherent in commonly used stress testing methodologies.

APRA Prudential Practice Guide SPG 530 Investment Governance (PDF) describes two stress testing approaches:

1. Sensitivity analysis – The impact of a change in a single variable (e.g. inflation) holding other variables constant.
2. Scenario analysis – simulate the simultaneous impact of a change (i.e. a pandemic) across multiple variables.

Sensitivity analysis wouldn’t have helped. COVID 19 had a significant impact on government policy, the law, fiscal and monetary policy, business activity and financial markets. These impacts were experienced simultaneously. This leaves scenario analysis as the default risk management tool for stress testing risks such as a global pandemic.

Getting scenario analysis right requires successfully navigating the following challenges:

• Event – Have you identified the stress event?
• Data or Forecast – Is historical data available? Is there sufficient sample size and is it relevant?
• Variables – Can you identify and measure the variables affected by the stress event?
• Assumptions- What are your assumptions for these variables?
• Distributions – How are these variables distributed?
• Correlations – What is the correlation between these variables?
• Data or Forecast – Is historical data available? Is there sufficient sample size and is it relevant? If not, how will we forecast the variables, their distributions and their correlations?

We’ve already considered the first challenge. Scenario testing only helps if you’re faced with a scenario that you’ve chosen to consider. This is a subjective judgement.

It’s tempting to ignore risk scenarios where the opportunity cost of mitigation is deemed to be too high. For example, the only way to deal with certain risks is to hold cash. This creates a drag on performance when other asset classes are performing well.

A superannuation fund might also overlook risks that it is unlikely to be held accountable for. These risks could be passed off as “bad luck” because they affect all funds, or because they were difficult to foresee.

The unspoken rule of scenario analysis is that you always choose a scenario that you know you can pass. Show me superannuation fund that will present APRA with a scenario that it can’t survive?

Once the stress event has been specified, the next step is to search for historical analogues. This is only possible if the risk, or a similar risk, has occurred in the past. Examining similar risk periods can give modelers clues on how to specify the stress scenario.

That said, the sample size of similar risks may be very small. Also, the historical analogues may have occurred a long time ago. This means that data from the period may be unavailable or unreliable.

Even if the data is of sufficient quality, the world, the economy and financial markets may have changed in many ways. For example, comparing the current COVID 19 pandemic to the Spanish Flu would be an apple to oranges comparison across multiple dimensions.

The lack of historical data leaves forecasting (guessing) as the only alternative.

Deciding the key variables that matter is also a subjective judgement. For example, a superannuation fund may have prepared a pandemic scenario, but did that scenario include:

• A reduction in contributions
• Early withdrawal of member funds
• Write downs of illiquid defensive assets such as property and infrastructure
• Currency hedging losses

On the flip side, did the stress scenario include positive variables that could offset or even outweigh the negatives? For example, extraordinary levels of fiscal and monetary stimulus resulting in an unprecedented expansion of the money supply?

A scenario modeller would then need to make assumptions about each of these variables. A savvy modeller would realise that a single point estimate is highly unreliable. They would need to specify a distribution for each variable.

The next challenge is deciding how each of the variables fit together. In other words, figuring out correlation and causation. Causation can’t be established with any certainty. Correlations can be measured, but they are rarely stable.

In summary, the probability that a risk modeller will successfully navigate all of these challenging decisions is vanishingly small. APRA Deputy Chair Helen Rowell summed up the challenges of stress testing using a scenario-based approach in her speech to the AIST 2020 Trustee Forum:

Like most of the world, APRA was aware of the emergence of a highly contagious novel coronavirus but we didn’t imagine that Australia was on the verge of a health and economic crisis that would see the domestic economy contract 7 per cent in a quarter, and entire states locked down for months. I also don’t think any of us imagined a scenario where investment losses topped $200 billion in a single quarter, or where the superannuation industry paid out more than $33 billion in early release payments to members in less than six months under expanded eligibility rules designed to support those materially impacted by the crisis. 

How can superannuation funds manage risk if stress testing with sensitivity and scenario analysis don’t work? They can do so by ”breaking” their fund.

Breaking your fund involves reverse engineering the conditions necessary to force your fund into a regulatory breach, capital inadequacy or insolvency (i.e., being unable to pay member benefits when due).

You can then compare the broken fund scenario to a stress event in real-time. Risk is managed in real-time by taking evasive action as the real-life stress event moves closer to what you know will break the fund.

This avoids having to make ANY ex-ante assumptions. It doesn’t rely on your subjective judgement to identify and your ability foresee risks.  And it prepares you for EVERY situation that could break your fund.

__________

[i3] Insights is the official educational bulletin of the Investment Innovation Institute [i3]. It covers major trends and innovations in institutional investing, providing independent and thought-provoking content about pension funds, insurance companies and sovereign wealth funds across the globe.