Policy learning with rare outcomes

Abstract: In settings with heterogeneous policy impacts, observational data can be used to learn optimal policy assignment rules under unconfoundedness. One recent approach (Athey and Wager (2021)) searches over shallow decision trees, using estimates of double-robust (DR) scores obtained via machine learning, to maximise the expected value of a particular treatment rule. In health applications, policy impacts are commonly evaluated for rare outcomes, where it may be challenging to obtain estimates that fully capture treatment effect heterogeneity. Little is known about how ML algorithms perform in this setting. This paper, therefore, contrasts various methods for obtaining DR-scores, using simulated data with varying degrees of overlap quality and treatment effect heterogeneity, and reports their performance in terms of errors in estimated average and heterogeneous treatment effects, and the value of learned policy rules for rare outcomes. The methods are also applied to a case study with the goal of reducing infant mortality through improved targeting of subsidised health insurance in Indonesia. Although the methods perform similarly in settings with normal outcome prevalence, there are clear winners when outcomes are rare. We find that Bayesian additive regression trees (BART) perform particularly well in situations where the sample size is relatively low compared to the number of covariates. We also compare simulation and case study decision-tree results to a plug-in policy in which everyone with an estimated benefit receives treatment. In cases with rare outcomes and complex treatment effect heterogeneity, this policy class may be preferable - but requires a trade-off between ease of interpretability/implementation and the average benefits.

Join Zoom Meeting
https://york-ac-uk.zoom.us/j/98144782919?pwd=ZE1JNC9iV1NMTkRLT1ZoT0YrOC85Zz09

Meeting ID: 981 4478 2919
Passcode: 170470