The Aggregation Challenge

2025

1. The Aggregation Challenge

   Macartan Humphreys, and Alexandra Scacco

   2025

   We have multiple projects innovating on aspects of what we call the ’aggregation challenge.’ The aggregation challenge is the challenge of combining findings from multiple studies to foster cumulative learning. We think of it as one of the biggest challenges facing social scientists and that addressing it will be key for strengthening the relevance and reliability of social science findings. IPI has contributed to the metaketa initiative housed at EGAP. Metaketas are coordinated randomized trials across multiple sites with harmonized measures and analysis strategies. In addition, we are leading the meta-analysis for a multicountry collection of studies coordinated by APRIES to assess the prevalence of human trafficking prevalence estimation. Moreover, our Correlates of Corona project examines socioeconomic predictors of Covid mortality.

2023

1. SSRN

   Gathering, Evaluating, and Aggregating Social Scientific Models

   Golden Miriam A., Slough Tara, Zhai Haoyu, and 29 more authors

   The Social Science Research Network, 2023

   Abs Bib HTML PDF

   On what basis can we claim a scholarly community understands a phenomenon? Social scientists generally propagate many rival explanations for what they study. How best to discriminate between or aggregate them introduces myriad questions because we lack standard tools that synthesize discrete explanations. In this paper, we assemble and test a set of approaches to the selection and aggregation of predictive statistical models representing different social scientific explanations for a single outcome: original crowd-sourced predictive models of COVID-19 mortality. We evaluate social scientists’ ability to select or discriminate between these models using an expert forecast elicitation exercise. We provide a framework for aggregating discrete explanations, including using an ensemble algorithm (model stacking). Although the best models outperform benchmark machine learning models, experts are generally unable to identify models’ predictive accuracy. Findings support the use of algorithmic approaches for the aggregation of social scientific explanations over human judgement or ad-hoc processes.

   @article{golden_covid_2023,
     author = {A., Golden Miriam and Tara, Slough and Haoyu, Zhai and Alexandra, Scacco and Macartan, Humphreys and Eva, Vivalt and Alberto, Diaz-Cayeros and Yi, Dionne Kim and Sampada, KC and Eugenia, Nazrullaeva and M., Aronow P. and Jan-Tino, Brethouwe and Anne, Buijsrogge and John, Burnett and Stephanie, DeMora and Ramón, Enríquez José and Robbert, Fokkink and Chengyu, Fu and Nicholas, Haas and Virginia, Hayes Sarah and Hanno, Hilbig and R., Hobbs William and Dan, Honig and Matthew, Kavanagh and A., Lindelauf Roy H. and Nina, McMurry and L., Merolla Jennifer and Amanda, Robinson and S., Solís Arce Julio and ten Thij Marijn and Felicity, Türkmen Fulya and Stephen, Utych},
     title = {Gathering, Evaluating, and Aggregating Social Scientific Models},
     journal = {The Social Science Research Network},
     year = {2023},
     publisher = {Elsevier Inc.},
     doi = {https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4570855},
     status = {peer},
     proj = {covid, aggregation},
     keywords = {methods},
     proj.1 = {covid}
   }

2. CUP

   Integrated Inferences: Causal Models for Qualitative and Mixed-Method Research

   Macartan Humphreys, and Alan M Jacobs

   2023

   Abs Bib HTML PDF Code

   Integrated Inferences develops a framework for using causal models and Bayesian updating for qualitative and mixed-methods research. By making, updating, and querying causal models, researchers are able to integrate information from different data sources while connecting theory and empirics in a far more systematic and transparent manner than standard qualitative and quantitative approaches allow. This book provides an introduction to fundamental principles of causal inference and Bayesian updating and shows how these tools can be used to implement and justify inferences using within-case (process tracing) evidence, correlational patterns across many cases, or a mix of the two. The authors also demonstrate how causal models can guide research design, informing choices about which cases, observations, and mixes of methods will be most useful for addressing any given question.

   @book{humphreys_jacobs_2023,
     author = {Humphreys, Macartan and Jacobs, Alan M},
     title = {Integrated Inferences: Causal Models for Qualitative and Mixed-Method Research},
     year = {2023},
     publisher = {Cambridge University Press},
     url = {https://www.cambridge.org/core/books/integrated-inferences/45B07964AD4718A74CDE3E35A31F26FA},
     status = {peer},
     proj = {aggregation},
     keywords = {methods},
     proj.1 = {aggregation}
   }

2020

1. WD

   The aggregation challenge

   Macartan Humphreys, and Alexandra Scacco

   World Development, 2020

   Abs Bib PDF

   Banerjee, Duflo, and Kremer have had an enormous impact on scholarship on the political economy of development. But as RCTs have become more central in this field, political scientists have struggled to draw implications from proliferating micro-level studies for longstanding macro­ level problems. We describe these challenges and point to recent innovations to help address them.

   @article{aggregation_2020,
     author = {Humphreys, Macartan and Scacco, Alexandra},
     title = {The aggregation challenge},
     journal = {World Development},
     year = {2020},
     pages = {104806},
     volume = {127},
     publisher = {Elsevier},
     status = {peer},
     proj = {aggregation},
     keywords = {methods, development},
     proj.1 = {aggregation}
   }

2. Chapter

   Field experiments, theory, and external validity

   Anna Wilke, and Macartan Humphreys

   SAGE Handbook of Research Methods in Political Science and International Relations, 2020

   Abs Bib HTML PDF Code

   Critics of field experiments lament a turn away from theory and criticize findings for weak external validity. In this chapter, we outline strategies to address these challenges. Highlighting the connection between these twin critiques, we discuss how structural approaches can both help design experiments that maximize the researcher’s ability to learn about theories and enable researchers to judge to what extent the results of one experiment can travel to other settings. We illustrate with a simulated analysis of a bargaining problem to show how theory can help make external claims with respect to both populations and treatments and how combining random assignment and theory can both sharpen learning and alert researchers to over-dependence on theory.

   @article{wilke_2020,
     author = {Wilke, Anna and Humphreys, Macartan},
     title = {Field experiments, theory, and external validity},
     journal = {SAGE Handbook of Research Methods in Political Science and International Relations},
     year = {2020},
     pages = {1007--35},
     publisher = {SAGE London},
     status = {peer},
     proj = {aggregation},
     keywords = {theory, methods},
     proj.1 = {aggregation}
   }

2019

1. CUP

   Information, accountability, and cumulative learning: Lessons from Metaketa I

   Thad Dunning, Guy Grossman, Macartan Humphreys, and 3 more authors

   2019

   Abs Bib PDF

   Throughout the world, voters lack access to information about politicians, government performance, and public services. Efforts to remedy these informational deficits are numerous. Yet do informational campaigns influence voter behavior and increase democratic accountability? Through the first project of the Metaketa Initiative, sponsored by the Evidence in Governance and Politics (EGAP) research network, this book aims to address this substantive question and at the same time introduce a new model for cumulative learning that increases coordination among otherwise independent researcher teams. It presents the overall results (using meta-analysis) from six independently conducted but coordinated field experimental studies, the results from each individual study, and the findings from a related evaluation of whether practitioners utilize this information as expected. It also discusses lessons learned from EGAP’s efforts to coordinate field experiments, increase replication of theoretically important studies across contexts, and increase the external validity of field experimental research.

   @book{metaketa_book_2019,
     author = {Dunning, Thad and Grossman, Guy and Humphreys, Macartan and Hyde, Susan D and McIntosh, Craig and Nellis, Gareth},
     title = {Information, accountability, and cumulative learning: Lessons from Metaketa I},
     year = {2019},
     publisher = {Cambridge University Press},
     status = {peer},
     proj = {aggregation},
     keywords = {experimental, development, vertical linkages},
     proj.1 = {aggregation}
   }