Responsible data management

J. Stoyanovich, S. Abiteboul, B. Howe, H. V. Jagadish, and S. Schelter
Communications of the ACM, Vol 65, Issue 6 (2022) read online


Resume format, Linkedin urls and other unexpected influences on AI personality prediction in hiring: Results of an audit

A. K. Rhea, K. Markey, L. D’Arinzo, H. Schellmann, M. Sloane, P. Squires, and J. Stoyanovich
Proceedings of the Fifth AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society (AIES), 2022, forthcoming


An external stability audit framework to test the validity of personality prediction in AI hiring

A. K. Rhea, K. Markey, L. D’Arinzo, H. Schellmann, M. Sloane, P. Squires, F. Arif Khan, and J. Stoyanovich
Data Mining and Knowledge Discovery, Special Issue on Bias and Fairness in AI, 2022, forthcoming


Fairness in Ranking, Part I: Score-based Ranking

Meike Zehlike, Ke Yang & Julia Stoyanovich
ACM Computer Surveys (2022) read online


Fairness in Ranking, II: Learning-to-Rank and Recommender Systems

Meike Zehlike, Ke Yang & Julia Stoyanovich
ACM Computer Surveys (2022) read online


It’s Just Not That Simple: An Empirical Study of the Accuracy-Explainability Trade-off in Machine Learning for Public Policy

Andrew Bell, Ian Solano-Kamaiko, Oded Nov & Julia Stoyanovich
FAccT ’22: 2022 ACM Conference on Fairness, Accountability, and Transparency (2022) read online


German AI Start-Ups and “AI Ethics”: Using A Social Practice Lens for Assessing and Implementing Socio-Technical Innovation

Mona Sloane & Janina Zakrzewski
FAccT ’22: 2022 ACM Conference on Fairness, Accountability, and Transparency (2022) read online


Teaching Responsible Data Science

Julia Stoyanovich
DataEd ’22: 1st International Workshop on Data Systems Education (2022) read online


Towards substantive conceptions of algorithmic fairness: Normative guidance from equal opportunity doctrines

Falaah Arif Khan, Eleni Manis, Julia Stoyanovich
CoRR, vol. abs/2207.02912, (2022) read online


Spending Privacy Budget Fairly and Wisely

Lucas Rosenblatt, Joshua Allen, Julia Stoyanovich
CoRR, vol. abs/2204.12903, (2022) read online


Think About the Stakeholders First! Towards an Algorithmic Transparency Playbook for Regulatory Compliance

Andrew Bell, Oded Nov, Julia Stoyanovich
CoRR, vol. abs/2207.01482, (2022) read online


Covid-19 Brings Data Equity Challenges to the Fore

H.V. Jagadish, Julia Stoyanovich & Bill Howe
ACM Digital Government Research and Practice (2021) read online

Disaggregated Interventions to Reduce Inequality 

Lucius Bynum, Joshua R. Loftus & Julia Stoyanovich
EAAMO’21, ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (2021) read online

Lightweight Inspection of Data Preprocessing In Native Machine Learning Pipelines

Stefan Grafberger, Julia Stoyanovich & Sebastian Schelter
CIDR’21, 11th Conference on Innovative Data Systems Research (2021) read online

Causal Intersectionality and Fair Ranking 

Ke Yang, Joshua R. Loftus & Julia Stoyanovich
FORC’21, 2nd Symposium on Foundations of Responsible Computing (2021) read online

Comparing Apples and Oranges: Fairness and Diversity in Ranking 

Julia Stoyanovich
ICDT’21, 24th International Conference on Database Theory (2021) read online    watch the talk


The Imperative of Interpretable Machines 

Julia Stoyanovich, Jay Van Bavel & Tessa V. West
Nature Machine Intelligence (2020) read online

Responsible Data Management 

Julia Stoyanovich, Bill Howe & H.V. Jagadish
Proceedings of VLDB Endowment (2020) read online

Taming Technical Bias in Machine Learning Pipelines

Sebastian Schelter & Julia Stoyanovich
IEEE Data Engineering Bulletin (2020) read online

FairPrep: Promoting Data to a First-Class Citizen in Studies on Fairness-Enhancing Interventions

Sebastian Schelter, Yuxuan He, Jatin Khilnani & Julia Stoyanovich
EDBT’20, 23rd International Conference on Extending Database Technology (2020) read online

Balanced Ranking with Diversity Constraints 

Ke Yang, Vasilis Gkatzelis & Julia Stoyanovich
Proceedings of IJCAI (2019) read online

Designing Fair Ranking Schemes 

Abolfazl Asudeh, H. V. Jagadish, Julia Stoyanovich & Gautam Das
Proceedings of ACM SIGMOD (2019) read online

Mithra Ranking: A System for Responsible Ranking Design 

Yifan Guan, Abolfazl Asudeh, Pranav Mayuram, H. V. Jagadish, Julia Stoyanovich, Gerome Miklau & Gautam Das
Proceedings of ACM SIGMOD (2019) read online

Transparency, Fairness, Data Protection, Neutrality: Data Management in the Face of New Regulation 

Serge Abiteboul & Julia Stoyanovich
ACM Journal of Data and Information Quality (2019) read online


Nutritional Labels for Data and Models 

Julia Stoyanovich & Bill Howe
IEEE Data Engineering Bulletin 42(3): 13-23 (2019) read online

Towards Responsible Data-Driven Decisions Making in Score-Based Systems

Abolfazl Asudeh, H. V. Jagadish & Julia Stoyanovich
IEEE Data Engineering Bulletin 42(3): 76-87 (2019) read online

The Responsibility Challenge for Data 

H. V. Jagadish, Francesco Bonchi, Tina Eliassi-Rad, Lise Getoor, Krishna P. Gummadi & Julia Stoyanovich
Proceedings of ACM SIGMOD (2019) read online


TransFAT: Translating Fairness, Accountability, and Transparency into Data Science Practice 

Julia Stoyanovich
International Workshop on Processing Information Ethically (2019) read online

AI Procurement Primer

Mona Sloane, Rumman Chowdhury, John C. Havens,  Tomo Lazovich & Luis C. Rincon Alba (2021)


Public Engagement Showreel Int 1894 

Julia Stoyanovich, Steven Kuyan, Meghan McDermott, Maria Grillo & Mona Sloane (2020)



We Are AI

We Are AI is a 5-week learning circle course that provides an introduction to the basics of AI and the social and ethical dimensions of the use of AI in modern life.


Responsible Data Science Courses at the NYU Center for Data Science

Responsible Data Science is a technical course for graduate and undergraduate students at NYU that tackles the issues of ethics, legal compliance, data quality, algorithmic fairness and diversity, transparency of data and algorithms, privacy, and data protection.


AI Ethics: Global Perspectives

This course is designed to raise awareness of the societal impacts of technology and to give individuals and institutions the tools to pursue responsible AI use.

Data Responsibly Series 

Vol. 1: Mirror, Mirror

Falaah Arif Khan and Julia Stoyanovich


Vol. 1: Miroir, mon beau Miroir (French Edition)

Falaah Arif Khan and Julia Stoyanovich

Translated by Serge Abiteboul, Pascal Guitton, Victor Vianu, and Eve Francois Trement


Vol. 1: Espejito, Espejito (Spanish Edition)

Falaah Arif Khan and Julia Stoyanovich

Translated by Charles Schroeder and Ana Elisa Mendez

Vol. 2: Fairness and Friends

Falaah Arif Khan, Eleni Manis and Julia Stoyanovich

We Are AI Series

We are AI Issue 5 cover image with a tree

Vol. 5: We are AI


Somos IA #5: Somos IA (Spanish Edition)


Julia Stoyanovich and Falaah Arif Khan


Ranking Facts

ranking facts tile image

Ranking Facts is a nutritional label for rankings.  The tool allows you to upload data, design a ranking methodology, and understand the fairness, diversity, and stability of the result.  Use it through a web-based UI, or incorporate it into your python code.


Go to GitHub

Fair Prep

Fair Prep Tile Image

FairPrep is a design and evaluation framework for fairness-enhancing interventions that treats data as a first-class citizen in complex data science pipelines.


Go to GitHub 

Data Synthesizer

Data Synthesizer project tile

The Data Synthesizer is a tool that generates privacy-preserving synthetic data, using the framework of differential privacy.  Synthetic data can facilitate collaborations between data scientists and owners of sensitive data, and it can support algorithmic transparency and disclosure.  Use it through a web-based UI, or incorporate it into your python code.


Go to GitHub 


MLinspect project tile

MLInspect is a data distribution debugger.  Use it to tame technical bias in your complex data science pipelines that use common libraries like pandas and scikit-learn. 


Go to GitHub