Auditing AI in Hiring

Algorithmic personality tests are in broad use, but do they work? We seek to answer this question by interrogating the validity of algorithmic personality tests that claim to estimate a job seeker’s personality based on their resume or social media profile. We developed a methodology for auditing the stability of predictions made by these tests and conducted an external audit of two commercial systems, Humantic AI and Crystal, over a dataset of job applicant profiles collected through an IRB-approved study.  
Learn More  Right Arrow
We Are AI is a free 5-week course that introduces the basics of AI, and discusses the social and ethical implications of the use of AI in modern life.  Developed in collaboration with P2PU and the Queens Public Library, the course will be offered to members of the public starting in January 2022.   
Learn More  Right Arrow


The 2021 NYU Public Interest Technology (PIT) Convention and Career Fair was a virtual hands-on event for professionals, researchers and students interested in creating a better world through technology.
Learn More  Right Arrow

NYC Hiring Bill

NYU R/AI ran a rapid series of virtual workshops and public engagement events to raise awareness of automated decision systems used in hiring, and to solicit concerns, questions and feedback from New Yorkers as the city considers the adoption of a bill that pertains to the use of automated hiring tools.  This bill became Local Law 144 of 2021.
We collaborated with Queens Public Library  to conduct three well-attended public workshops; with Schneps Media to hold a public forum; and successfully supported sourcing expert testimonies for the City’s Council’s hearing on the bill.
Learn More  Right Arrow

AI Procurement Primer

Existing public procurement processes and standards for AI are in urgent need of innovation to address potential risks and harms to citizens. We wrote a primer based on our research and on input from leading experts to propose  pathways forward.
Learn More  Right Arrow

Responsible AI Comics

The Data, Responsibly and We Are AI comic series cut through the technical jargon to make the discourse around critical technologies more accessible to the public. We Are AI can be read on its own, or it can be paired with our free 5-week course for the general public. Data, Responsibly comics are used as supplementary reading for responsible data science courses for current and future data scientists.
Learn More  Right Arrow

AI Ethics Global Perspectives Course

This course is designed to raise awareness of the societal impacts of technology and to give individuals and institutions the tools to pursue responsible AI use. Intended for current and future data scientists, policymakers, and business leaders, this course contains modules on topics related to artificial intelligence. Each module consists of a video lecture accompanied by additional resources such as podcasts, videos, and readings.  This course is the result of a partnership between The GovLab, Institute for Ethics in AI at the Technical University of Munich, Global AI Ethics Consortium, and NYU R/AI.
Learn More   Right Arrow

Ranking Facts

Ranking Facts is a nutritional label for rankings.  The tool allows you to upload data, design a ranking methodology, and understand the fairness, diversity, and stability of the result.  Use it through a web-based UI, or incorporate it into your python code.
Learn More  Right Arrow

Data Synthesizer

The Data Synthesizer is a tool that generates privacy-preserving synthetic data, using the framework of differential privacy.  Synthetic data can facilitate collaborations between data scientists and owners of sensitive data, and it can support algorithmic transparency and disclosure.  Use it through a web-based UI, or incorporate it into your python code.
Learn More  Right Arrow


MLInspect is a data distribution debugger.  Use it to tame technical bias in your complex data science pipelines that use common libraries like pandas and scikit-learn.  
Learn More  Right Arrow


FairPrep is a design and evaluation framework for fairness-enhancing interventions that treats data as a first-class citizen in complex data science pipelines.  
Learn More  Right Arrow


Mirror is a data generator that produces a synthetic dataset based on a directed acyclic graph (DAG) that captures the relationships between the attributes. Mirror can be used to produce “dirty” data, mirroring bias that may occur in real datasets, and it can be used to test the performance of your classifier or ranker.
Learn More  Right Arrow