Automated decisions increasingly mediate civic life. Governments use algorithms to screen immigrants and allocate social services. Corporations rely on software to help make decisions in vital areas like hiring, credit, and political discourse. Trends like these have made algorithms objects of concern beyond the confines of computer science.

Advocates, policymakers, and technologists have begun demanding that these automated decisions be explained, justified, and audited. There is a growing desire to “open the black box” of complex algorithms and hold the institutions using them accountable. But across the globe, civil society faces a range of challenges as they pursue these goals.

This paper considers how the public can effectively scrutinize, understand, and govern automated decisions, based on an extensive review of computer and social science literature as well as dozens of semi-structured interviews and conversations with global digital rights advocates, regulators, technologists, and industry representatives. We also surveyed a broad array of real-world attempts to scrutinize automated systems, documenting the purpose of each inquiry, its methods, and its findings.

To encourage clear discussion, we offer a conceptual framework to describe how different elements of automated systems work together to produce a result. In addition to its source code, knowledge of a system’s existence, purpose, impact, constitution, policies, inputs and outputs, and training data can be helpful to both scrutinize and govern these systems. This framework highlights how nontechnical insights about an automated system can be just as important, and often more important, than its technical, tangible artifacts.

We catalog a variety of both analog and technical approaches to scrutiny that we encountered during our research, and analyze each: journalism, qualitative research, legal process, black box testing, examination of training data, and code review. We also discuss emerging methods to design systems for accountability, including applying fairness metrics, designing for interpretability, cryptographically ensuring procedural regularity, and producing audit trails.

We consider these methods alongside common legal and regulatory approaches. Based on this appraisal, we highlight promising areas for progress and opportunities for further research, policy discussion, and advocacy.