Review and opinion piece on the CASPer test publication titled “Extending the interview to all medical school candidates--Computer-Based Multiple Sample Evaluation of Noncognitive Skills (CMSENS)”.
Note: The following represent the opinions of the authors and additional independent research is required to validate or refute the findings in the original CASPer test publication.
Reviewed CASPer test publication: Extending the interview to all medical school candidates--Computer-Based Multiple Sample Evaluation of Noncognitive Skills (CMSENS). Acad Med. 2009 Oct;84(10 Suppl):S9-12. doi: 10.1097/ACM.0b013e3181b3705a.
“To avoid criticism say nothing, do nothing, be nothing.” – Aristotle
Summary of the Overall Findings
1. This study (nor any other study to date) have proven that CASPer is valid predictor of future behavior. At best, the test has shown a mild correlation with future tests. In essence this test is merely a predictor of future tests. For a test to be valid it must be able to measure the constructs it is designed to measure. For example, a ruler is a good tool for measuring distance but it's not good for measuring the temperature of a room. Similarly, CASPer has been "validated" to measure future test performances such as performance on the objective structured clinical examinations (OSCE) or other examinations. This means that by design such tests are only valid measures of future tests and are not necessarily able to predict future on-the-job behavior. The best predictor of future performance is intrinsic motivation because motivation directs behavior.
2. The original article on CMSENS (later renamed to CASPer), is a small-scale, pilot study that includes one case site: McMaster University. The positive results showing correlations between CASPer and MMI are at best, predictive for future studies that are larger and geographically diverse.
3. The authors have attempted to generalize the findings to various populations such as domestic and international medical graduates, and undergraduate medical school applicants in the Netherlands, without adequate scientific evidence and sufficient sample size.
5. Statistical issues – the small sample size in the publication suffers from “the law of small numbers” and “a bias of confidence over doubt”. Read more...