Risk Tip 16 – Let us start at the very end
One of the areas that organisations find difficult is determining the effectiveness of controls, something I have written about previously on how we measure effectiveness.
The most significant challenge I have noted is the development of performance measures for controls in order for effectiveness to be measured.
The methodology I have developed to assist organisations to develop performance measures, counterintuitively, begins at the end with the control end state.
For the example in this blog I have chosen a water based joy ride at an amusement park.
Let’s explore this by using a control for a joy ride to demonstrate the methodology for a training and competency framework.
Step 1 – Control End State
Step 1 of the process is to define the Control End State. The Control End State is a narrative that starts with the statement “if this control is effective …..”. In the case of the training and competency framework for the joy ride it might look something like this:
| Control | Joy ride training and competency framework |
|---|---|
| 1 | Has been trained by a qualified trainer, utilising a training manual with all training serials clearly articulated, and the training manual accurately reflects the current operating procedures for the ride |
| 2 | Has had their competency assessed by a qualified independent assessor against a competency checklist that accurately reflects the current operating procedures for the ride |
| 3 | Has their achievement of competency recorded in the centralised training record maintained by HR |
| 4 | Competency has been periodically assessed, through observation, by a qualified independent assessor in accordance with the competency review schedule in the Ride Training and Competency Policy |
Once we have established the Control End State, this then allows us to develop performance measures.
Step 2 – Develop Performance Measures
Based on the Control End State, we can now develop performance measures. In the case of the joy ride, the performance measures for the ride’s training and competency framework would be:
| % of personnel operating the joy ride who have received the training specified in RG 02-1-04 – Joy Ride Training/Assessment of Competency |
| % of personnel operating the joy ride that have been trained by a certified trainer as defined in TRG 02-2 – Qualification Standards for Trainers and Competence Assessors |
| % of training serials on the joy ride training checklist provided at Attachment 1 of TRG 02-1-04 –Training/Assessment of Competency that have been signed by the trainer and the operator |
| % of training checklists submitted to HR in accordance with TRG 02-1 – Training and Assessment of Competence for Ride Operators |
| % of personnel operating the joy ride that have been certified as competent in accordance with the requirements specified in TRG 02-1-04 –Training/Assessment of Competency |
| % of personnel operating the joy ride that have been assessed as competent by a qualified assessor as defined in TRG 02-2 – Qualification Standards for Trainers and Competence Assessors |
| % of competency serials on the joy ride competency checklist provided at Attachment 2 of TRG 02-1-04 – Joy Ride Training/Assessment of Competency that have been signed by the trainer |
| % of competency assessment forms provided at Attachment 2 of TRG 02-1-04 –Training/Assessment of Competency that have been signed by the operator |
| % of competency checklists submitted to HR in accordance with TRG 02-1 – Training and Assessment of Competence for Ride Operators |
| % of personnel operating the joy ride that have been retrained and had their competency assessed after any change to operating procedures as specified in TRG 02-3 – Ongoing Assessment of Competency to Operate Rides |
| % of personnel operating the Joy Ride that have had their competency periodically assessed in accordance with Attachment 1 of TRG 02-3 – Ongoing Assessment of Competency to Operate Rides |
The review of performance against these measures will be done by a combination of Control Owners, a second line assurance function and, if required, internal and/or external audit.
We now have the performance measures, but how do we determine the effectiveness of a control?
Step 3 – Define KPIs and Effectiveness Parameters
The methodology I use to define whether a control is effective or not is through the development of effectiveness parameters. These give us an understanding, based on our assurance activities, as to whether the control is being adhered to. At this stage we are not assessing the design of the control – that is done through the assessment of incidents (explained below).
The following is an example for several of the controls listed above:
In this case, there is no “wriggle room” with this control: anything less than 100% is not acceptable.
Once again, in this case, anything less than 100% is ineffective.
Provided the training has been completed and it has been done by a qualified instructor, submitting the checklist to HR is not as critical, provided it has been completed and is accessible within the Operations Department. To that end, there is some leeway in the measure of effectiveness for this control.
The key thing to point out at this stage is that we are only assessing conformance against the control, but what we have not done is assess the design of the control for effectiveness. This is much harder to do and requires that the organisation maintains a centralised Incident Register with the capacity to identify the factors that contributed to the incident.
We will look at that process in the next newsletter, but for now, why not try developing some end-states for the controls in your environment and see what emerges in terms of performance measures.
Much of this analysis and methodology has been shaped by the tragic events that came of the Thunder Rides Rapid event that occurred at Dreamworld. I’ll be releasing a book on this matter very soon and I hope, the observations and analysis provide readers of risk management a better understanding of how important risk management is from policies and procedures, through to frameworks, performance measures and control measures, just to name a few.
