By Kent Haden Founder and CEO of Resilience Software
In my last post I discussed how inefficient and possibly ineffective I believe it is to use performance counts to measure competency. Instead I suggested that we use data from teacher-observed competence to measure learner success. In this post I explain in more detail what I mean by Observed Competence and why I believe it is a more effective way to measure competence.
As people learn, they gain both knowledge and skills (and attitudes). A person’s knowledge can usually be assessed through tests and exams but skills are far more difficult to assess. Skills assessment, especially in medical education, is very challenging because it involves people helping people in a domain that is very complex and sensitive.
Training programs, learners, and the public all want to ensure that learners get the depth and breadth of hands-on experience they need to be ready for the next stage in their career and be skillful practitioners. There is wide variation in the amount of repetition needed to learn a new skill ‘well enough’ (I know from experience, I took up martial arts in my 50’s). The variation depends on an individuals previous experience, confidence, and other factors. With this wide variation and not wanting to use counts, how can we be sure people are competent?
I believe the only practical way to tell if someone is truly competent is for an experienced, competent, practitioner to carefully observe the learner performing the skill in the real world and then document the level of competence witnessed. This is what I am calling Observed Competence.
While I believe that observed competence is a better method to ensure competency and should be used instead of counts, the approach also has challenges and isn’t a quick fix. There are some questions and obstacles to consider:
- Even the best teachers rarely want to take the time to document. How do we get the needed observations?
- The level of granularity changes as someone progresses through a long training program. At the start they may only be assessed on small steps and later on they need to be assessed on providing complex services independently.
- What is ‘well enough’? Is it a 3 on a 5-point scale or something more complex?
- Do we rate learners considering their level of training or use level expected at graduation as the target at all levels of training? In other words, do we rate relative to expectations for their level of training or expect people to score very poorly at the start of their training?
- How do (and should) we track levels of mastery beyond ‘well enough’?
- How many different observers need to document that the learner is competent? How many times do each of them need to confirm it and what if there are not enough competent observers to provide those assessments if the skill is rarely practiced?
- How recent does the observation have to be to? With memory decay curves and the need for spaced repetition, it is important to consider this, especially when someone is in practice.
- How do we ensure the learner can handle even the most challenging settings?
- How do we ensure the learner has enough experience to handle the complexity in cases they will face in practice?
As you can see I’ve spent a lot of time thinking about this topic. Measuring competence more effectively is at the heart of why we developed the T-Res Assessment System to make it practical to document self and learner assessments of individual skills. Although I am writing about learner assessment before graduation, there is no logical reason that Observed Competency could not be used to ensure practitioners are maintaining their skills after graduation.
Although T-Res does not yet provide a solution to all of the questions and issues listed above, my experience working with hundreds of training programs and building out the functionality of our assessment & evaluation software has provided me with some ideas about how to help educators transition to this new method.
I have put some thought into designing the few key variables that should to be available in the ideal system to support observed competence. Those variables are closely tied to our plan for expanding T-Res and I will discuss in future blog post.
The post Assessing Competency, Beyond Counts appeared first on .