Market Reports

The Troubled Industry of Risk Assessment

Jonathan Howitt
March 2012

Inherently Flawed from the Bottom up
I do not know where or how the methodology of inherent and residual risk assessment first arose. I assume it gained credence in the audit world since internal auditors were asked to justify their recommendations by quantifying the value of their control recommendations. Obviously then, the larger the inherent risk the greater the value of the control, and since no auditor would ever certify a process as riskless, there would always be some level of residual risk. The concept of inherent risk suited support management too: ‘the risk of my IT system failing could cost the firm billions’ fits well with budgets. It also suited the vendor market: lots of users using toolsets with simple maths displayed on tidy heat map reports showing the value of the control environment. Who wouldn’t like it?

Not me I’m afraid. It’s spurious maths and it’s the wrong approach. Let’s use a simple analogy to demonstrate why. Imagine if you decide to learn to drive a racing car. You should already know it’s inherently risky. If the racing car had no controls, for instance no brakes or steering, you wouldn’t drive it would you? So it’s a fairly worthless discussion asking what the inherent risk is because we all know the answer is death. In the same way, no business can operate without controls, so we have to assume the controls are in place when assessing risk. But we can still consider single or multiple failures: for example, what would happen in your racing car if the steering fails at 120mph on a sharp turn? And what if the brakes fail as well? And how often might that happen in 10,000 laps? We can ask the same questions in the business context to understand tail risk: for instance, what if your payment reconciliation process fails during a period of high volumes? And what if your IT system fails at the same time? Based on past experience how likely is that in any 10 year period?

Alright, let’s assume we’ve actually had a useful risk discussion with business management and we’ve avoided all the flawed pseudoscience and pitfalls around trying to quantify inherent and residual risk and compute the value of the control environment across hundreds of individual processes. Let’s assume we have collected a series of likelihood and impact figures at consistent confidence levels for each significant activity. Now, how do we aggregate all these numbers? Unfortunately, our bottom up assessments simply won’t add up, nor do they lend themselves to easy modelling either. Here’s why: even if they have been implemented with rigorous consistency and objectivity, the complexity of the many to many relationships and dependencies between processes, risks and controls is too great. Thousands of detailed risks and controls will diversify massively across the process universe and the outputs will be highly sensitive to our assumptions on loss distributions, dependencies and correlations. But let’s imagine we have a sophisticated modelling tool and we can somehow cope with all this. The outputs will still be very hard to work through and explain to senior management who may not be statistically minded, let alone feedback the results in a meaningful way to each business area that contributed.

That certainly doesn’t mean I don’t think we should be doing bottom up risk assessment, quite the contrary – no sensible manager would ever want to be blind on the risks in his processes – but we need to be pragmatic about their qualitative and frankly often subjective nature, especially if they’re conducted on an inherent and residual basis, where there’s typically little or no sophistication in the quantitative methodologies deployed. Let’s use bottom up risk assessment for management purposes and extract the risk issues and remediation lessons, but not get too bogged down on control benefit calculations or capital allocations.

Self-Assessment and the Fifth Amendment
I think it was in 1993 that I was required, as a business controller, to complete a new FDIC internal assessment on behalf of the business. It made sense from the regulatory perspective: if you didn’t have the resources to audit something yourself, ask firms to complete their own self-assessments, which establish a moral hazard for them and provide you with somewhere to start if you have to review them directly later. Of course many of us in large firms have regularly had to produce self-assessments for personal performance appraisals, it’s a well tried and tested management technique. The only issue is: nobody will knowingly incriminate themselves. That’s not to say there isn’t value in a frank discussion on risk strengths and weaknesses with a business, but they don’t want their dirty laundry hung out in public. In fact quite the reverse: it becomes a ‘make me look good’ exercise if it’s going north. Declaring weaknesses will only be worthwhile if it evidences a need for more resource, or posits a ‘get out of jail free’ should a problem subsequently occur. Without independence, the risk assessment process can too easily be gamed.

What elements are we assessing? The original thinking was to focus on purely causal components of risk – people, systems, organisation and external factors, often because there was too little objective data to assess these elements any other way. Most of the available data was process-related and couldn’t take into account qualitative factors, management opinion, or attribute a value on planned remediation.

Self-assessments are difficult to keep fresh. When first introduced, quite senior business management might be directly involved, but the following year it gets delegated until eventually a contractor, trainee or even intern is simply managing an update process, often within an automated toolset. At this point the exercise is almost entirely without value, in fact it makes rather a mockery of what the assessment process should be trying to achieve. It is an abdication of risk management responsibility, or at the very least, outsourcing to much less qualified people to do it.

Furthermore, if you have an internal audit department, and they are adequately resourced, self-assessment is often a substantial duplication of what audit are supposed to be doing independently. There is nothing more depressing for the risk department than being the reconciliation agent between business self-assessments and internal audit’s risk assessment. Given the problems of gaming outlined above, in my experience the internal audit assessment is usually much more valuable. Self-assessment in my view is largely debunked. If all risk assessment is independent, it is then simply a matter of which elements are performed by the risk department and which are conducted by internal audit. Each firm has to work this boundary out for itself, but intuitively risk will own the risk framework and risk quantification including risk indicator metrics, and audit will assess the design and effectiveness internal controls and conduct some measure of substantive testing.

Sarbanes-Oxley and the Industry of Compliance
I was once asked at a conference in New York how my firm was integrating its operational risk and Sarbanes-Oxley programmes. I was the last to answer on a panel of 4, the other participants all senior risk officers at major US banks. Each had given a drawn out response but I simply said that my firm was not US listed and did not have to comply with Sarbanes-Oxley. I immediately received a somewhat unexpected rapturous applause.

Operational risk professionals and certainly the vendor market had initially welcomed the budgets that came with the new Sarbanes-Oxley rules. The enthusiasm was short-lived. Up to that point, operational risk, although perhaps thinly resourced, had enjoyed a relative period of invention: regulators were happy to ‘let a thousand flowers bloom’ in development of the discipline. The general impetus of the profession had been to collect data and give objectivity to the assessment process. #implementing operational risk management was about achieving a commercial benefit from understanding and managing risk exposures better – it was not about regulatory compliance for its own sake.

Unfortunately the new budgets that arrived with Sarbanes-Oxley brought with it a different focus and an industry of compliance for operational risk that it is still struggling to shake free from. Despite my comments in New York, It isn’t just a US problem: many major European and Asian banks are either US listed or encouraged by their auditors or self-serving consulting firms to at least implement ‘Sox lite’. Risk professionals may protest that this is Finance work, but that doesn’t fly if the CRO or Head of Risk reports to the CFO. Anyway, why duplicate the risk and internal audit assessment process if it can just be extended to work for Sarbanes-Oxley as well? So operational risk headcounts have spawned, but no one in the business is seeing the commercial benefit of this new industry, because its focus has become too compliance driven.

We have talked about the flawed process of self-assessment, but Sarbanes-Oxley is self-assessment on steroids. A colleague once shared an article early on when Sarbanes-Oxley was first being introduced, that it was at heart a don’t ask, don’t tell regime. The example given was the father who tells his teenage daughter to behave when she goes to a party – when she appears in the morning looking bedraggled, not having slept much and quite hung over, he asks her if she behaved. ‘Yes Daddy’ she replies, because she cannot give any other response. Like so many outputs of self-assessment, this makes a mockery of risk management, and sadly reflects a giant leap backwards in the development of the discipline.

Start Again from the Top Down
I know many banks which have beaten their chests over their op risk implementation because they hired armies of self-assessment teams and gleaned little or no commercial benefit from it, if anything the opposite: they wasted a lot of business time and focus, lost in the undergrowth, unable to see the forest for the trees. None of them dares to call a halt though: what CRO wants to give the impression that they aren’t serious about all risks, however small? And if they stopped, wouldn’t there be pressure to start again after a loss? But their calling card is poor, and in my view has put risk in the space of audit and seriously damaged the brand. Op risk has lost so much credibility over the industry of control self-assessment.

Instead, CROs and their Op Risk Heads should have been religiously focussed on the top down perspective, collecting objective data to support their scenarios, and drilling into the detail as and when needed. That way they would have been in a position to keep their department lean, high quality, motivated and commercially-oriented. Unfortunately, few have had the courage or shown the leadership to consistently follow this path. Too many op risk functions became the collection points for disparate risk assessment cottage industries rather than the strategic, business-minded and advisory functions they should have been.

So my message to the CRO who is disillusioned with their op risk implementation is that it’s not too late to start again and refocus the activity. Assess the big risks and hold capital for them, but manage the rest of the risk implementation as an expected loss management programme. Of course you need to map your processes and understand the risk and control dependencies for major activities, but once that’s done, it’s fairly static information. Risk then needs to be dynamic and data-driven to be useful for the business. So invest in collecting risk information and synthesising it, and keep looking for new angles on data – don’t stop reinventing your KRI reports and loss analysis and firm up the link with remediation. And when your assessments need updating, don’t do this in a vacuum or bury it in an IT toolset. Engage the business in a positive and interactive way – use risk workshops, brainstorm the issues and demonstrate the valuable contribution of the risk function as objective facilitator and trusted adviser, a catalyst not a cost.