Digital Transform MedEd
Webinar: Lies, damned lies, and surveys: Designing better surveys for education and research
Led by: Professor Anthony Artino
The George Washington University School of Medicine and Health Sciences, Washington, DC, USA
Held on 29th September 2021
Webinar: Lies, damned lies, and surveys: Designing better surveys for education and research
Led by: Professor Anthony Artino
The George Washington University School of Medicine and Health Sciences, Washington, DC, USA
Held on 29th September 2021
Lies, damned lies, and surveys: Designing better surveys for education and research
Anthony R. Artino, Jr., PhD
Professor and Associate Dean for Evaluation and Educational Research
The George Washington University School of Medicine and Health Sciences
Washington, DC, USA
Mark Twain famously expressed his disdain for statistics when he said, “There are three kinds of lies: lies, damned lies, and statistics.” Beyond referencing his own difficulties with math, Twain was making the point that statistics can have persuasive power, even when used inappropriately. In short, statistics often are used to bolster weak arguments.
The same can be said of surveys—results from a poorly designed and/or poorly executed survey can have considerable persuasive power. Flawed survey results can send educators off searching for fixes to non-existent problems, negatively influence policy decisions, and fill the scientific literature with unsubstantiated knowledge claims that take years to correct, if they are ever corrected.
In this presentation, Professor Artino discusses the challenge of designing and implementing high-quality, self-administered surveys. Using empirical data and a series of examples from health care, education, and politics, Professor Artino describes how survey results can be used to shape science and policy, often inappropriately. In addition, Professor Artino provides insight into the cognitive processes that guide how respondents make sense of survey questions, and demonstrates what happens when these cognitive processes are ignored by educators and researchers. Finally, several design principles are proposed, which are meant to improve surveys and introduce a seven-step process that front loads the survey-design process by focusing heavily on item development and pretesting. Taken together, the goal of this talk is to demonstrate that Mark Twain’s lament, when applied to surveys, is often correct—but it certainly does not have to be.
Learning Objectives:
By the end of this session, participants will be able to:
Anthony R. Artino, Jr., PhD
Professor and Associate Dean for Evaluation and Educational Research
The George Washington University School of Medicine and Health Sciences
Washington, DC, USA
Mark Twain famously expressed his disdain for statistics when he said, “There are three kinds of lies: lies, damned lies, and statistics.” Beyond referencing his own difficulties with math, Twain was making the point that statistics can have persuasive power, even when used inappropriately. In short, statistics often are used to bolster weak arguments.
The same can be said of surveys—results from a poorly designed and/or poorly executed survey can have considerable persuasive power. Flawed survey results can send educators off searching for fixes to non-existent problems, negatively influence policy decisions, and fill the scientific literature with unsubstantiated knowledge claims that take years to correct, if they are ever corrected.
In this presentation, Professor Artino discusses the challenge of designing and implementing high-quality, self-administered surveys. Using empirical data and a series of examples from health care, education, and politics, Professor Artino describes how survey results can be used to shape science and policy, often inappropriately. In addition, Professor Artino provides insight into the cognitive processes that guide how respondents make sense of survey questions, and demonstrates what happens when these cognitive processes are ignored by educators and researchers. Finally, several design principles are proposed, which are meant to improve surveys and introduce a seven-step process that front loads the survey-design process by focusing heavily on item development and pretesting. Taken together, the goal of this talk is to demonstrate that Mark Twain’s lament, when applied to surveys, is often correct—but it certainly does not have to be.
Learning Objectives:
By the end of this session, participants will be able to:
- Recognize the elements of a survey;
- Describe how cognitive processes and motivation guide the way people understand and respond to survey questions;
- Identify poorly written survey items and other design pitfalls;
- State several design principles; and
- Identify the importance of conducting expert reviews and cognitive interviews.
Anthony R. Artino, Jr., PhD, is a tenured professor and Associate Dean for Evaluation and Educational Research at the George Washington (GW) University School of Medicine and Health Sciences in Washington, DC. Prior to his arrival at GW, Dr. Artino served 23 years in the U.S. Navy, retiring as a Captain in 2020. While on active duty, he was the Deputy Director and co-founder of the Center for Health Professions Education at the Uniformed Services University of the Health Sciences.
In his current leadership role at GW, he conducts research, teaches graduate courses, mentors students and faculty, and provides administrative direction in evaluation and educational research. As a researcher, he has been an investigator on more than a dozen research grants funded by diverse organizations, including the Association of American Medical Colleges, the Congressionally Directed Medical Research Programs, and the National Board of Medical Examiners.
His most highly cited works are a blend of research and education articles on topics ranging from analyzing and interpreting quantitative data, understanding academic motivation and self-regulation in medicine, and developing surveys for educational research. Dr. Artino serves on the editorial review boards of several peer-reviewed academic journals, is a member of the National Academies of Sciences, Engineering, and Medicine’s Global Forum on Innovation in Health Professional Education, and is a fellow of the Association for Medical Education in Europe. He publishes widely in medical and health professions education and has given numerous invited talks and research presentations around the globe.
He earned his PhD in educational psychology from the University of Connecticut and also holds master’s degrees in physiology and education.
In his current leadership role at GW, he conducts research, teaches graduate courses, mentors students and faculty, and provides administrative direction in evaluation and educational research. As a researcher, he has been an investigator on more than a dozen research grants funded by diverse organizations, including the Association of American Medical Colleges, the Congressionally Directed Medical Research Programs, and the National Board of Medical Examiners.
His most highly cited works are a blend of research and education articles on topics ranging from analyzing and interpreting quantitative data, understanding academic motivation and self-regulation in medicine, and developing surveys for educational research. Dr. Artino serves on the editorial review boards of several peer-reviewed academic journals, is a member of the National Academies of Sciences, Engineering, and Medicine’s Global Forum on Innovation in Health Professional Education, and is a fellow of the Association for Medical Education in Europe. He publishes widely in medical and health professions education and has given numerous invited talks and research presentations around the globe.
He earned his PhD in educational psychology from the University of Connecticut and also holds master’s degrees in physiology and education.