By Stephen Brosnan
Recently, I attended the 23rd International Conference on Science, Technology and Innovation indicators. The focus of the conference was to stimulate discussion on comprehensive approaches and indicators in the study of science, technology and innovation. A key debate at the conference was the use of quantitative and qualitative approaches to research evaluations. Typically, there are two approaches in the evaluation of scientific quality: peer review vs metrics. Peer review refers to a process of research assessment based on the use of expert opinion and judgment while metrics refer to a quantifiable measure used to track and assess the status of a specific process.
The use of metrics as a tool for evidence-based decision making has gained traction amongst sports teams, most famously by Billy Beane of the Oakland A’s. The ‘Moneyball’ strategy has been implemented by many sports teams with varying rates of success. Daryl Morey, General Manager of the Houston Rockets, has utilised metrics to assemble the team with the best regular season record in the NBA while Bill Gerrard has provided data analysis for both AZ Alkmaar in soccer and Saracens in rugby. However, similar to the academic debate, there has been significant push back towards the overreliance on metrics to guide decision making. Charles Barkley, an 11-time NBA All-Star, argues that “analytics is crap” because the NBA is all about talent, that Morey is “one of those idiots”, and that proponents of analytics are "a bunch of guys who ain't never played the game [and] they never got the girls in high school”.
A key issue in the debate is that metrics tend to ‘count what is easily measurable rather than measuring what counts’. The argument is that metrics fail to capture important tacit elements of performance such as hard work, heart and determination. Furthermore, there is an argument that once a measure becomes known it is no longer a useful measure due to ‘gaming’ of the system. This occurred last year when OKC point guard Russell Westbrook was accused of padding his stats to achieve a season average triple-double. In the final regular season game, he needed 16 rebounds to average a triple-double for the second-straight year. Westbrook wound up grabbing 20, a career-high.
Another limitation associated with using metrics to guide evaluations and decision-making is that choosing the right metrics is not easy and the choice of metrics can lead to contrasting recommendations. This issue is evident every time there is a debate surrounding the greatest NBA player of all time. Michael Jordan’s fans point to his 100% record in NBA finals along with his six final MVP awards as reason he is the GOAT. The choice of these metrics paints the narrative that Jordan was the ultimate competitor with a win at all costs mentality and always showed up on the big occasion. However, Karim Abdul Jabbar’s supporters point to the fact he is the NBA leading scorer of all time with the most unstoppable shot ever, the Skyhook while Lebron James fans point to the fact that he is the only player to be top 10 all time in scoring and assists, thus painting the narrative of the perfect teammate that makes everyone better.
Recently, Jerry Muller published a book entitled “Tyranny of the Metrics” which highlights the dangers associated with the quantification of performance metrics in education, medicine, business and finance, government, the police and military. I think it is clear from this piece that sport may also be added to the list. Metrics may be used (and abused) in sport to convey particular narratives and recommendations and should be used as a complement - not a substitute for- judgements based on experience and expertise.
Recently, I attended the 23rd International Conference on Science, Technology and Innovation indicators. The focus of the conference was to stimulate discussion on comprehensive approaches and indicators in the study of science, technology and innovation. A key debate at the conference was the use of quantitative and qualitative approaches to research evaluations. Typically, there are two approaches in the evaluation of scientific quality: peer review vs metrics. Peer review refers to a process of research assessment based on the use of expert opinion and judgment while metrics refer to a quantifiable measure used to track and assess the status of a specific process.
The use of metrics as a tool for evidence-based decision making has gained traction amongst sports teams, most famously by Billy Beane of the Oakland A’s. The ‘Moneyball’ strategy has been implemented by many sports teams with varying rates of success. Daryl Morey, General Manager of the Houston Rockets, has utilised metrics to assemble the team with the best regular season record in the NBA while Bill Gerrard has provided data analysis for both AZ Alkmaar in soccer and Saracens in rugby. However, similar to the academic debate, there has been significant push back towards the overreliance on metrics to guide decision making. Charles Barkley, an 11-time NBA All-Star, argues that “analytics is crap” because the NBA is all about talent, that Morey is “one of those idiots”, and that proponents of analytics are "a bunch of guys who ain't never played the game [and] they never got the girls in high school”.
A key issue in the debate is that metrics tend to ‘count what is easily measurable rather than measuring what counts’. The argument is that metrics fail to capture important tacit elements of performance such as hard work, heart and determination. Furthermore, there is an argument that once a measure becomes known it is no longer a useful measure due to ‘gaming’ of the system. This occurred last year when OKC point guard Russell Westbrook was accused of padding his stats to achieve a season average triple-double. In the final regular season game, he needed 16 rebounds to average a triple-double for the second-straight year. Westbrook wound up grabbing 20, a career-high.
Another limitation associated with using metrics to guide evaluations and decision-making is that choosing the right metrics is not easy and the choice of metrics can lead to contrasting recommendations. This issue is evident every time there is a debate surrounding the greatest NBA player of all time. Michael Jordan’s fans point to his 100% record in NBA finals along with his six final MVP awards as reason he is the GOAT. The choice of these metrics paints the narrative that Jordan was the ultimate competitor with a win at all costs mentality and always showed up on the big occasion. However, Karim Abdul Jabbar’s supporters point to the fact he is the NBA leading scorer of all time with the most unstoppable shot ever, the Skyhook while Lebron James fans point to the fact that he is the only player to be top 10 all time in scoring and assists, thus painting the narrative of the perfect teammate that makes everyone better.
Recently, Jerry Muller published a book entitled “Tyranny of the Metrics” which highlights the dangers associated with the quantification of performance metrics in education, medicine, business and finance, government, the police and military. I think it is clear from this piece that sport may also be added to the list. Metrics may be used (and abused) in sport to convey particular narratives and recommendations and should be used as a complement - not a substitute for- judgements based on experience and expertise.
RSS Feed
