The credibility revolution has led to a greater focus on the quality of research designs when assessing what inferences to draw from published research. Surprisingly however there are few common standards for either characterizing designs or assessing their quality. A book, paper, software package, and a series of blog posts take up the question.
Documents
2023
PUP
Research Design in the Social Sciences: Declaration, Diagnosis, and Redesign
Assessing the properties of research designs before implementing them can be tricky for even the most seasoned researchers. This book provides a powerful frameworkâModel, Inquiry, Data Strategy, and Answer Strategy, or MIDAâfor describing any empirical research design in the social sciences. MIDA enables you to characterize the key analytic features of observational and experimental designs, qualitative and quantitative designs, and descriptive and causal designs. An accompanying algorithm lets you declare designs in the MIDA framework, diagnose properties such as bias and precision, and redesign features like sampling, assignment, measurement, and estimation procedures. Research Design in the Social Sciences is an essential tool kit for the entire life of a research project, from planning and realization of design to the integration of your results into the scientific literature.
@book{dd_2023,author={Blair, Graeme and Coppock, Alexander and Humphreys, Macartan},title={Research Design in the Social Sciences: Declaration, Diagnosis, and Redesign},year={2023},publisher={Princeton University Press},url={https://press.princeton.edu/books/paperback/9780691199573/research-design-in-the-social-sciences},status={peer},proj={design},keywords={methods},proj.1={design}}
Researchers need to select high-quality research designs and communicate those designs clearly to readers. Both tasks are difficult. We provide a framework for formally declaring the analytically relevant features of a research design in a demonstrably complete manner, with applications to qualitative, quantitative, and mixed methods research. The approach to design declaration we describe requires defining a model of the world (M), an inquiry (I), a data strategy (D), and an answer strategy (A). Declaration of these features in code provides sufficient information for researchers and readers to use Monte Carlo techniques to diagnose properties such as power, bias, accuracy of qualitative causal inferences, and other diagnosands. Ex ante declarations can be used to improve designs and facilitate preregistration, analysis, and reconciliation of intended and actual analyses. Ex post declarations are useful for describing, sharing, reanalyzing, and critiquing existing designs. We provide open-source software, DeclareDesign, to implement the proposed approach.
@article{dd_2019,author={Blair, Graeme and Cooper, Jasper and Coppock, Alexander and Humphreys, Macartan},title={Declaring and diagnosing research designs},journal={American Political Science Review},year={2019},number={3},pages={838--859},volume={113},publisher={Cambridge University Press},doi={https://doi.org/10.1017/S0003055419000194},url={https://www.cambridge.org/core/journals/american-political-science-review/article/declaring-and-diagnosing-research-designs/3CB0C0BB0810AEF8FF65446B3E2E4926},status={peer},proj={design},keywords={methods},proj.1={design}}