Pitfalls in the risk matrix

download this column in pdf

	I like this column
	989 persons like this column

In the past years risk management has grown in importance within the profession of asset management. In the Netherlands the topic has been put firmly on the map by the national safety board led by Pieter van Vollenhove. A formal education for risk management has been established as well. Within risk management, risk assessment is a vital step. A risk is a two dimensional entity (risk equals probability times consequences) and establishing a risk level thus requires estimating both the probability and impact. The risk level has to be judged against a risk tolerance criterion. A common tool for doing that is the risk matrix. This is a table with a description of potential consequences and their classification on one side, and the likelihood of occurrence on the other side. In the table itself is the risk level that is associated with the combinations of likelihood and consequence. This is often displayed by color (green, yellow, red) and name (low, medium, high). As the risk matrix is a handy tool, virtually all risk managers will have one.

Despite the common use of risk matrices there is surprisingly little standardization of risk matrices in use[1], and a framework for making one is also lacking. To add to the mystery, even the international standard for risk management (ISO31000) does not formulate requirements on this topic. Therefore most risk managers will have to develop their matrix themselves. On one hand that is a good thing. As we mentioned in an earlier column, the risk matrix captures the values system of the organization, which is unique for every organization. Besides, it is the quest for the right matrix that provides insight in the subtleties of what is really important. But on the other hand, are the value systems that different? Organizations operate in the same society and by external factors (shareholders, markets, customers) they will be straitjacketed to some extent. Furthermore, in the areas of safety and the environment there will be legal requirements and one would expect (or hope?) that the risk attitude with regard to those values will not be very different to what society thinks is acceptable. As a robust procedure for establishing a risk matrix is lacking, it would not be surprising if errors are made. There is a number of pitfalls to step into regarding the risk matrix. These can be distinguished in design flaws and operational errors.

Design flaws

A common design flaw is that the matrix has an inadequate structure. The risks an organization face scan have a spread of multiple orders of magnitude, both in terms of likelihood and consequence). Think for example about a printing error (small cost, several times per day) and of a fire (large damage, remote probability). Furthermore, there is (at least initially) a significant uncertainty in the estimates for likelihood and consequence. The only scale capable of dealing with a large spread and relative uncertainty is the logarithmic scale. A linear scale cannot discriminate between risks in the lowest categories of likelihood and impact, whereas they can still differ orders of magnitude. A linear scale is constant in absolute uncertainty, whereas a logarithmic scale is constant in relative uncertainty. Part of the inadequate structure is the dimension of the matrix (number of categories and risk levels). If everything is categorized into three risk levels (low, medium high) it does not make sense to judge the likelihood on a scale with 9 levels.

The second design flow is a misalignment of the impact descriptions. That is the case if the decision maker is not indifferent between impacts on different values within the same impact category. Indifferent means that no answer can be given to the question what impact is preferred, e.g. a serious safety incident or serious reputational damage. If there still is a preference, the values are not properly aligned. Ideally this indifference is also the willingness to pay, at least with regard to the alignment between the financial and non-financial values. In plain English that means that one is willing to spend the corresponding amount in the financial column to prevent the impact from happening. This helps in developing mitigations, as mitigations that would cost more could be abandoned early and mitigations costing less could be quickly employed. Improper alignment in the end means that a high risk has not the same meaning for all values.

Below is a risk matrix complying with the basic design requirements.

Operational errors

The risk matrix is a tool for evaluating risks. Risks are events with a likelihood and an impact. A risk matrix is not fit for evaluating the system performance, as that is not an event but the combined impact of a large number of events. If the system performance is judged against the risk matrix, the outcome is most likely unacceptable. For example, the total costs of running an organization are virtually always in the top category. If that occurs every year (and OPEX has the tendency to do so) there is something wrong.

Furthermore, the variation of system performances is typically measured in percentages whereas risks are estimated in an order of magnitude. Risk matrices therefore hardly will discriminate between performances a factor of two apart. But from a management perspective a change by a factor of two is a huge difference. If the performance evaluation is forced into the risk matrix this will result in a serious deformation and imbalance of the matrix which will make the use for which the risk matrix was intended in the first place almost impossible.

Another operational error is that the evaluation by means of the matrix can be on level off. By dividing  the continuum of likelihood and impact into discrete categories a small change in the boundary values between the categories can result a different verdict. A high risk (e.g. 1,5 M impact about once every 10 years) would become medium if the boundary between serious and catastrophic was 2M instead of 1M. Fortunately the shift will be no more than one category. However, using the risk matrix beyond this resolution can result in errors. This is almost certainly the case when the risk matrix is used for making decisions on mitigations, which happens to be a common practice. For decision making it is better to use the expected value (expressed on a continuous scale) instead of the risk level (a discrete scale). Within the same risk level the expected value spreads over 2 orders of magnitude (in the above matrix) which can make a difference in the cost effectiveness of the mitigation. Risk levels are only useful in determining what risks to analyze for mitigation development.

Advocating standardization

Summarizing one could state that in the design and use so many errors can be made that a risk matrix is an accident waiting to happen. In that light, some standardization might not be a bad idea. 

<< back to overview