Below, you will find an extensive compilation of our academic publications and software artifacts, providing a window into our scholarly contributions. Each entry includes an external link to the full text of the paper for a comprehensive understanding.
In addition, you can explore our ongoing technology policy and education projects, complete with detailed descriptions and direct links to the project repositories. We hope that this collection of resources will serve as a valuable tool for researchers, students, and the broader community to engage with our work.
Data-centric AI and responsible data management
Incorporating ethics and legal compliance into data-driven algorithmic systems has been attracting significant attention from the computing research community, most notably under the umbrella of fair and interpretable machine learning. While important, much of this work has been limited in scope to the last mile of data analysis and has disregarded both the system’s design, development, and use life cycle (What are we automating and why? Is the system working as intended? Are there any unforeseen consequences post-deployment?) and the data life cycle (Where did the data come from? How long is it valid and appropriate?). Our work on data-centric responsible AI and on responsible data management is based on the observation that the decisions we make during data collection and preparation profoundly impact the robustness, fairness, and interpretability of the systems we build.
2023
Query Refinement for Diversity Constraint Satisfaction
Jinyang Li, Yuval Moskovitch, Julia Stoyanovich, and H. V. Jagadish
@article{DBLP:journals/pvldb/LiMSJ23,author={Li, Jinyang and Moskovitch, Yuval and Stoyanovich, Julia and Jagadish, H. V.},title={Query Refinement for Diversity Constraint Satisfaction},journal={Proc. {VLDB} Endow.},volume={17},number={2},pages={106--118},year={2023},timestamp={Mon, 27 Nov 2023 13:13:34 +0100},biburl={https://dblp.org/rec/journals/pvldb/LiMSJ23.bib},bibsource={dblp computer science bibliography, https://dblp.org},keywords={journal,fairness,data},}
ERICA: Query Refinement for Diversity Constraint Satisfaction
Jinyang Li, Alon Silberstein, Yuval Moskovitch, Julia Stoyanovich, and H. V. Jagadish
@article{DBLP:journals/pvldb/LiSMSJ23,author={Li, Jinyang and Silberstein, Alon and Moskovitch, Yuval and Stoyanovich, Julia and Jagadish, H. V.},title={{ERICA:} Query Refinement for Diversity Constraint Satisfaction},journal={Proc. {VLDB} Endow.},volume={16},number={12},pages={4070--4073},year={2023},doi={10.14778/3611540.3611623},timestamp={Mon, 23 Oct 2023 16:16:16 +0200},biburl={https://dblp.org/rec/journals/pvldb/LiSMSJ23.bib},bibsource={dblp computer science bibliography, https://dblp.org},keywords={journal,fairness,data},}
Subset Modelling: A Domain Partitioning Strategy for Data-efficient Machine-Learning
Vı́tor Ribeiro, Eduardo H. M. Pena, Raphael Freitas Saldanha, Reza Akbarinia, Patrick Valduriez, Falaah Arif Khan, Julia Stoyanovich, and Fábio Porto
In Proceedings of the 38th Brazilian Symposium on Databases, SBBD 2023, Belo Horizonte, MG, Brazil, September 25-29, 2023 2023
@inproceedings{DBLP:conf/sbbd/RibeiroPSAVKS023,author={Ribeiro, V{\'{\i}}tor and Pena, Eduardo H. M. and de Freitas Saldanha, Raphael and Akbarinia, Reza and Valduriez, Patrick and {Arif Khan}, Falaah and Stoyanovich, Julia and Porto, F{\'{a}}bio},title={Subset Modelling: {A} Domain Partitioning Strategy for Data-efficient
Machine-Learning},booktitle={Proceedings of the 38th Brazilian Symposium on Databases, {SBBD} 2023,
Belo Horizonte, MG, Brazil, September 25-29, 2023},pages={318--323},publisher={{SBC}},year={2023},timestamp={Wed, 25 Oct 2023 16:14:33 +0200},biburl={https://dblp.org/rec/conf/sbbd/RibeiroPSAVKS023.bib},bibsource={dblp computer science bibliography, https://dblp.org},keywords={conference,data},}
Automated Data Cleaning Can Hurt Fairness in Machine Learning-based Decision Making
Shubha Guha, Falaah Arif Khan, Julia Stoyanovich, and Sebastian Schelter
In Proceedings of the 39th International Conference on Data Engineering, ICDE 2023
@inproceedings{guha_23,author={Guha, Shubha and Khan, Falaah Arif and Stoyanovich, Julia and Schelter, Sebastian},year={2023},title={Automated Data Cleaning Can Hurt Fairness in Machine Learning-based Decision Making},keywords={conference,data,fair},booktitle={Proceedings of the 39th International Conference on Data Engineering, {ICDE}},}
@article{jdiq23,author={Jagadish, H.V. and Stoyanovich, Julia and Howe, Bill},title={The Many Facets of Data Equity},journal={ACM Journal of Data and Information Quality},year={2023},keywords={journal,data,fair},volume={14},issue={4},pages={1--21},doi={https://doi.org/10.1145/3533425},}
Personal Data for Personal Use: Vision or Reality?
Xin Luna Dong, Bo Li, Julia Stoyanovich, Anthony Kum Hoe Tung, Gerhard Weikum, Alon Y. Halevy, and Wang-Chiew Tan
In Companion of the 2023 International Conference on Management of Data, SIGMOD/PODS 2023, Seattle, WA, USA, June 18-23, 2023 2023
@inproceedings{DBLP:conf/sigmod/DongLSTWHT23,author={Dong, Xin Luna and Li, Bo and Stoyanovich, Julia and Tung, Anthony Kum Hoe and Weikum, Gerhard and Halevy, Alon Y. and Tan, Wang{-}Chiew},title={Personal Data for Personal Use: Vision or Reality?},booktitle={Companion of the 2023 International Conference on Management of Data,
{SIGMOD/PODS} 2023, Seattle, WA, USA, June 18-23, 2023},pages={263--264},publisher={{ACM}},year={2023},doi={10.1145/3555041.3589378},keywords={panel,data,privacy},addendum={panelist, by invitation},}
2022
Spending Privacy Budget Fairly and Wisely
Lucas Rosenblatt, Joshua Allen, and Julia Stoyanovich
Theory and Practice of Differential Privacy (@ICML) 2022
@article{rosenblatt2022spending,title={Spending Privacy Budget Fairly and Wisely},author={Rosenblatt, Lucas and Allen, Joshua and Stoyanovich, Julia},journal={Theory and Practice of Differential Privacy (@ICML)},year={2022},keywords={differential privacy, fairness, databases},}
Responsible Data Management
Julia Stoyanovich, Serge Abiteboul, Bill Howe, H. V. Jagadish, and Sebastian Schelter
@article{DBLP:journals/cacm/StoyanovichAHJS22,author={Stoyanovich, Julia and Abiteboul, Serge and Howe, Bill and Jagadish, H. V. and Schelter, Sebastian},title={Responsible Data Management},journal={Communications of the {ACM}},volume={65},number={6},pages={64--74},year={2022},doi={10.1145/3488717},timestamp={Wed, 25 May 2022 10:40:29 +0200},bibsource={dblp computer science bibliography, https://dblp.org},keywords={data},author+an={1=self},}
Developing data capability with non-profit organisations using participatory methods
Anthony McCosker, Xiaofang Yao, Kath Albury, Alexia Maddox, Jane Farmer, and Julia Stoyanovich
@article{doi:10.1177/20539517221099882,author={McCosker, Anthony and Yao, Xiaofang and Albury, Kath and Maddox, Alexia and Farmer, Jane and Stoyanovich, Julia},title={Developing data capability with non-profit organisations using participatory methods},journal={Big Data \& Society},volume={9},number={1},pages={20539517221099882},year={2022},doi={10.1177/20539517221099882},eprint={https://doi.org/10.1177/20539517221099882},keywords={journal,data,policy,governance},}
@article{DBLP:journals/pvldb/StoyanovichHJ20,title={Responsible Data Management},author={Stoyanovich, Julia and Howe, Bill and Jagadish, HV},journal={Proceedings of the VLDB Endowment},volume={13},number={12},year={2020},keywords={fairness,data,journal},doi={10.14778/3415478.3415570},}
FairPrep: Promoting Data to a First-Class Citizen in Studies on Fairness-Enhancing Interventions
Sebastian Schelter, Yuxuan He, Jatin Khilnani, and Julia Stoyanovich
In Proceedings of the 23nd International Conference on Extending Database Technology, EDBT 2020
@inproceedings{DBLP:conf/edbt/SchelterHKS20,author={Schelter, Sebastian and He, Yuxuan and Khilnani, Jatin and Stoyanovich, Julia},title={{F}air{P}rep: Promoting Data to a First-Class Citizen in Studies on Fairness-Enhancing
Interventions},booktitle={Proceedings of the 23nd International Conference on Extending Database
Technology, {EDBT}},pages={395--398},year={2020},doi={10.5441/002/edbt.2020.41},timestamp={Wed, 25 Mar 2020 15:46:06 +0100},bibsource={dblp computer science bibliography, https://dblp.org},keywords={conference,data,fair},}
Fairness-Aware Instrumentation of Preprocessing Pipelines for Machine Learning
Ke Yang, Biao Huang, Julia Stoyanovich, and Sebastian Schelter
In Proceedings of the Workshop on Human-In-the-Loop Data Analytics, HILDA at SIGMOD 2020
@inproceedings{DBLP:hilda20,author={Yang, Ke and Huang, Biao and Stoyanovich, Julia and Schelter, Sebastian},title={Fairness-Aware Instrumentation of Preprocessing~Pipelines for Machine Learning},booktitle={Proceedings of the Workshop on Human-In-the-Loop Data Analytics, {HILDA} at {SIGMOD}},year={2020},doi={https://doi.org/10.1145/3398730.3399194},keywords={workshop,data,fairness},}
Taming Technical Bias in Machine Learning Pipelines
@article{debull21,author={Schelter, Sebastian and Stoyanovich, Julia},title={Taming Technical Bias in Machine Learning Pipelines},journal={{IEEE} Data Eng. Bull.},year={2020},volume={43},number={4},keywords={journal,data,fairness},author+an={2=self},}
2019
Transparency, Fairness, Data Protection, Neutrality: Data Management Challenges in the Face of New Regulation
@article{DBLP:journals/jdiq/AbiteboulS19,author={Abiteboul, Serge and Stoyanovich, Julia},title={Transparency, Fairness, Data Protection, Neutrality: Data Management
Challenges in the Face of New Regulation},journal={ACM Journal of Data and Information Quality},volume={11},number={3},pages={15:1--15:9},year={2019},doi={10.1145/3310231},timestamp={Tue, 20 Aug 2019 08:40:18 +0200},bibsource={dblp computer science bibliography, https://dblp.org},keywords={journal,data,fairness,policy,governance},}
MithraLabel: Flexible Dataset Nutritional Labels for Responsible Data Science
Chenkai Sun, Abolfazl Asudeh, H. V. Jagadish, Bill Howe, and Julia Stoyanovich
In Proceedings of the 28th International Conference on Information and Knowledge Management, CIKM 2019
@inproceedings{DBLP:conf/cikm/SunAJHS19,author={Sun, Chenkai and Asudeh, Abolfazl and Jagadish, H. V. and Howe, Bill and Stoyanovich, Julia},title={MithraLabel: Flexible Dataset Nutritional Labels for Responsible Data
Science},booktitle={Proceedings of the 28th International Conference on Information
and Knowledge Management, {CIKM}},pages={2893--2896},publisher={{ACM}},year={2019},doi={10.1145/3357384.3357853},timestamp={Mon, 04 Nov 2019 11:09:32 +0100},bibsource={dblp computer science bibliography, https://dblp.org},keywords={conference,data,explainability},}
The Responsibility Challenge for Data
H. V. Jagadish, Francesco Bonchi, Tina Eliassi-Rad, Lise Getoor, Krishna P. Gummadi, and Julia Stoyanovich
In Proceedings of the 2019 International Conference on the Management of Data, SIGMOD 2019
@inproceedings{DBLP:conf/sigmod/JagadishBEGGS19,author={Jagadish, H. V. and Bonchi, Francesco and Eliassi{-}Rad, Tina and Getoor, Lise and Gummadi, Krishna P. and Stoyanovich, Julia},title={The Responsibility Challenge for Data},booktitle={Proceedings of the 2019 International Conference on the Management of
Data, {SIGMOD}},pages={412--414},publisher={{ACM}},year={2019},doi={10.1145/3299869.3314327},timestamp={Sat, 22 Jun 2019 17:10:04 +0200},bibsource={dblp computer science bibliography, https://dblp.org},keywords={panel,data},author+an={6=self},addendum={plenary panel co-organizer and presenter},}
TransFAT: Translating Fairness, Accountability, and Transparency into Data Science Practice
Julia Stoyanovich
In Proceedings of the 1st International Workshop on Processing Information Ethically, PIE at CAiSE 2019
@inproceedings{DBLP:conf/caise/Stoyanovich19,author={Stoyanovich, Julia},title={TransFAT: Translating Fairness, Accountability, and Transparency into
Data Science Practice},booktitle={Proceedings of the 1st International Workshop on Processing Information
Ethically, {PIE} at {CAiSE}},series={{CEUR} Workshop Proceedings},volume={2417},publisher={CEUR-WS.org},year={2019},timestamp={Wed, 12 Feb 2020 16:44:33 +0100},bibsource={dblp computer science bibliography, https://dblp.org},keywords={invited,data},author+an={1=self}}
Explainability
There is a variety of terms associated with this topic: transparency, interpretability, explainability, intelligibility. But let’s not get too tangled up in terminology. The main point is that we need to allow people to understand the data, the operation, and the decisions or predictions of an AI system, and to also understand why these decisions or predictions are made. This understanding is critical because it allows people to exercise agency and take control over their interactions with AI systems. And so, no matter what terminology we use, the overarching idea behind transparency & friends is to expose the “knobs of responsibility” to people, as a means to support the responsible design, development, use, and oversight of AI systems.
2023
Introducing contextual transparency for automated decision systems
Mona Sloane, Ian Solano-Kamaiko, Jun Yuan, Aritra Dasgupta, and Julia Stoyanovich
@article{NMI_2023,author={Sloane, Mona and Solano-Kamaiko, Ian and Yuan, Jun and Dasgupta, Aritra and Stoyanovich, Julia},title={Introducing contextual transparency for automated decision systems},journal={Nature Machine Intelligence},volume={5},year={2023},pages={187-–195},doi={https://doi.org/10.1038/s42256-023-00623-7},keywords={journal,explainability,policy,hiring},}
Think About the Stakeholders First! Towards an Algorithmic Transparency Playbook for Regulatory Compliance
@article{bell_nov_stoyanovich_2023,author={Bell, Andrew and Nov, Oded and Stoyanovich, Julia},title={Think About the Stakeholders First! {T}owards an Algorithmic Transparency
Playbook for Regulatory Compliance},volume={5},journal={Data \& Policy},publisher={Cambridge University Press},year={2023},keywords={journal,policy,explainability,education,playbook,governance},}
2022
Rankers, Rankees, & Rankings: Peeking into the Pandora’s Box from a Socio-Technical Perspective
@article{DBLP:journals/corr/abs-2211-02932,author={Yuan, Jun and Stoyanovich, Julia and Dasgupta, Aritra},title={Rankers, Rankees, {\&} Rankings: Peeking into the Pandora's Box
from a Socio-Technical Perspective},journal={CoRR},volume={abs/2211.02932},year={2022},url={https://doi.org/10.48550/arXiv.2211.02932},doi={10.48550/arXiv.2211.02932},keywords={working,ranking,explainability},eprinttype={arXiv},eprint={2211.02932},timestamp={Wed, 09 Nov 2022 17:33:26 +0100},biburl={https://dblp.org/rec/journals/corr/abs-2211-02932.bib},bibsource={dblp computer science bibliography, https://dblp.org},}
It’s Just Not That Simple: An Empirical Study of the Accuracy-Explainability Trade-off in Machine Learning for Public Policy
Andrew Bell, Ian Solano-Kamaiko, Oded Nov, and Julia Stoyanovich
In Proceedings of the 5th Annual ACM Conference on Fairness, Accountability, and Transparency, FAccT 2022
@inproceedings{DBLP:conf/fat/BellSNS22,author={Bell, Andrew and Solano{-}Kamaiko, Ian and Nov, Oded and Stoyanovich, Julia},title={It's Just Not That Simple: An Empirical Study of the Accuracy-Explainability
Trade-off in Machine Learning for Public Policy},booktitle={Proceedings of the 5th Annual {ACM} Conference on Fairness, Accountability, and
Transparency, {FAccT}},pages={248--266},publisher={{ACM}},year={2022},doi={10.1145/3531146.3533090},timestamp={Wed, 22 Jun 2022 10:20:30 +0200},bibsource={dblp computer science bibliography, https://dblp.org},keywords={explainability, transparency, policy,governance},}
2020
The Imperative of Interpretable Machines
Julia Stoyanovich, Jay J. Van Bavel, and Tessa V. West
@article{NMI_2020,author={Stoyanovich, Julia and {Van Bavel}, Jay J. and West, Tessa V.},title={The Imperative of Interpretable Machines},journal={Nature Machine Intelligence},volume={2},year={2020},pages={197--199},doi={https://doi.org/10.1038/s42256-020-0171-8},keywords={journal,explainability},}
@article{DBLP:journals/debu/StoyanovichH19,author={Stoyanovich, Julia and Howe, Bill},title={Nutritional Labels for Data and Models},journal={{IEEE} Data Eng. Bull.},volume={42},number={3},pages={13--23},year={2019},timestamp={Tue, 10 Mar 2020 16:23:50 +0100},bibsource={dblp computer science bibliography, https://dblp.org},keywords={journal,explainability},}
MithraLabel: Flexible Dataset Nutritional Labels for Responsible Data Science
Chenkai Sun, Abolfazl Asudeh, H. V. Jagadish, Bill Howe, and Julia Stoyanovich
In Proceedings of the 28th International Conference on Information and Knowledge Management, CIKM 2019
@inproceedings{DBLP:conf/cikm/SunAJHS19,author={Sun, Chenkai and Asudeh, Abolfazl and Jagadish, H. V. and Howe, Bill and Stoyanovich, Julia},title={MithraLabel: Flexible Dataset Nutritional Labels for Responsible Data
Science},booktitle={Proceedings of the 28th International Conference on Information
and Knowledge Management, {CIKM}},pages={2893--2896},publisher={{ACM}},year={2019},doi={10.1145/3357384.3357853},timestamp={Mon, 04 Nov 2019 11:09:32 +0100},bibsource={dblp computer science bibliography, https://dblp.org},keywords={conference,data,explainability},}
Fairness
Algorithmic fairness is a central topic in responsible AI. Fairness is a complex concept, and we treat it through a socio-legal-technical lens, in ways that are domain- and context-specific. We always start with a statement of the normative criteria (what are we trying to accomplish and why?), and propose technical solutions only as appropriate, in ways that align with the normative criteria. Highlights of our work include fairness in ranking, connections between equality of opportunity from political philosophy and algoroithmic fairness, and investigating the trade-offs between fairness and other normative dimensions of responsible AI.
2023
Setting the Right Expectations: Algorithmic Recourse Over Time
João Fonseca, Andrew Bell, Carlo Abrate, Francesco Bonchi, and Julia Stoyanovich
In Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, EAAMO 2023, Boston, MA, USA, 30 October 2023 - 1 November 2023 2023
@inproceedings{DBLP:conf/eaamo/FonsecaBABS23,author={Fonseca, Jo{\~{a}}o and Bell, Andrew and Abrate, Carlo and Bonchi, Francesco and Stoyanovich, Julia},title={Setting the Right Expectations: Algorithmic Recourse Over Time},booktitle={Proceedings of the 3rd {ACM} Conference on Equity and Access in Algorithms,
Mechanisms, and Optimization, {EAAMO} 2023, Boston, MA, USA, 30 October
2023 - 1 November 2023},pages={29:1--29:11},publisher={{ACM}},year={2023},url={https://doi.org/10.1145/3617694.3623251},doi={10.1145/3617694.3623251},timestamp={Thu, 09 Nov 2023 21:12:58 +0100},biburl={https://dblp.org/rec/conf/eaamo/FonsecaBABS23.bib},bibsource={dblp computer science bibliography, https://dblp.org},keywords={conference, fairness},}
Query Refinement for Diversity Constraint Satisfaction
Jinyang Li, Yuval Moskovitch, Julia Stoyanovich, and H. V. Jagadish
@article{DBLP:journals/pvldb/LiMSJ23,author={Li, Jinyang and Moskovitch, Yuval and Stoyanovich, Julia and Jagadish, H. V.},title={Query Refinement for Diversity Constraint Satisfaction},journal={Proc. {VLDB} Endow.},volume={17},number={2},pages={106--118},year={2023},timestamp={Mon, 27 Nov 2023 13:13:34 +0100},biburl={https://dblp.org/rec/journals/pvldb/LiMSJ23.bib},bibsource={dblp computer science bibliography, https://dblp.org},keywords={journal,fairness,data},}
ERICA: Query Refinement for Diversity Constraint Satisfaction
Jinyang Li, Alon Silberstein, Yuval Moskovitch, Julia Stoyanovich, and H. V. Jagadish
@article{DBLP:journals/pvldb/LiSMSJ23,author={Li, Jinyang and Silberstein, Alon and Moskovitch, Yuval and Stoyanovich, Julia and Jagadish, H. V.},title={{ERICA:} Query Refinement for Diversity Constraint Satisfaction},journal={Proc. {VLDB} Endow.},volume={16},number={12},pages={4070--4073},year={2023},doi={10.14778/3611540.3611623},timestamp={Mon, 23 Oct 2023 16:16:16 +0200},biburl={https://dblp.org/rec/journals/pvldb/LiSMSJ23.bib},bibsource={dblp computer science bibliography, https://dblp.org},keywords={journal,fairness,data},}
The Possibility of Fairness: Revisiting the Impossibility Theorem in Practice
Andrew Bell, Lucius Bynum, Nazarii Drushchak, Tetiana Herasymova, Lucas Rosenblatt, and Julia Stoyanovich
Proceedings of the Conference on Fairness, Accountability, and Transparency (ACM FAccT) 2023
@article{bell2023possibility,title={The Possibility of Fairness: Revisiting the Impossibility Theorem in Practice},author={Bell, Andrew and Bynum, Lucius and Drushchak, Nazarii and Herasymova, Tetiana and Rosenblatt, Lucas and Stoyanovich, Julia},journal={Proceedings of the Conference on Fairness, Accountability, and Transparency (ACM FAccT)},year={2023},keywords={fairness, theory, impossibility}}
Counterfactual Fairness Is Basically Demographic Parity
Lucas Rosenblatt, and R Teal Witter
Proceedings of the AAAI Conference on Artificial Intelligence 2023
@article{rosenblatt2022counterfactual,title={Counterfactual Fairness Is Basically Demographic Parity},author={Rosenblatt, Lucas and Witter, R Teal},journal={Proceedings of the AAAI Conference on Artificial Intelligence},year={2023},keywords={fairness, counterfactuals, causal},}
The Unbearable Weight of Massive Privilege: Revisiting Bias-Variance Trade-Offs in the Context of Fair Prediction
@article{DBLP:journals/corr/abs-2302-08704,author={Khan, Falaah Arif and Stoyanovich, Julia},title={The Unbearable Weight of Massive Privilege: Revisiting Bias-Variance
Trade-Offs in the Context of Fair Prediction},journal={CoRR},volume={abs/2302.08704},year={2023},doi={10.48550/arXiv.2302.08704},eprinttype={arXiv},eprint={2302.08704},keywords={working,fairness,stability},}
On Fairness and Stability: Is Estimator Variance a Friend or a Foe?
Falaah Arif Khan, Denys Herasymuk, and Julia Stoyanovich
@article{DBLP:journals/corr/abs-2302-04525,author={Khan, Falaah Arif and Herasymuk, Denys and Stoyanovich, Julia},title={On Fairness and Stability: Is Estimator Variance a Friend or a Foe?},journal={CoRR},volume={abs/2302.04525},year={2023},doi={10.48550/arXiv.2302.04525},eprinttype={arXiv},eprint={2302.04525},keywords={working,fairness,stability},}
Fairness in Ranking: From Values to Technical Choices and Back
Julia Stoyanovich, Meike Zehlike, and Ke Yang
In Companion of the 2023 International Conference on Management of Data, SIGMOD/PODS 2023, Seattle, WA, USA, June 18-23, 2023 2023
@inproceedings{DBLP:conf/sigmod/StoyanovichZY23,author={Stoyanovich, Julia and Zehlike, Meike and Yang, Ke},editor={Das, Sudipto and Pandis, Ippokratis and Candan, K. Sel{\c{c}}uk and Amer{-}Yahia, Sihem},title={Fairness in Ranking: From Values to Technical Choices and Back},booktitle={Companion of the 2023 International Conference on Management of Data,
{SIGMOD/PODS} 2023, Seattle, WA, USA, June 18-23, 2023},pages={7--12},publisher={{ACM}},year={2023},doi={10.1145/3555041.3589405},keywords={panel,fairness,ranking},addendum={peer-reviewed tutorial},}
Counterfactuals for the Future
Lucius E. J. Bynum, Joshua R. Loftus, and Julia Stoyanovich
In Proceedings of the AAAI Conference on Artificial Intelligence 2023
@inproceedings{bynum2023counterfactuals,author={Bynum, Lucius E. J. and Loftus, Joshua R. and Stoyanovich, Julia},title={Counterfactuals for the Future},url={https://ojs.aaai.org/index.php/AAAI/article/view/26655},doi={10.1609/aaai.v37i12.26655},number={12},booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},year={2023},pages={14144-14152},keywords={journal,fairness,causal,counterfactuals},}
2022
Spending Privacy Budget Fairly and Wisely
Lucas Rosenblatt, Joshua Allen, and Julia Stoyanovich
Theory and Practice of Differential Privacy (@ICML) 2022
@article{rosenblatt2022spending,title={Spending Privacy Budget Fairly and Wisely},author={Rosenblatt, Lucas and Allen, Joshua and Stoyanovich, Julia},journal={Theory and Practice of Differential Privacy (@ICML)},year={2022},keywords={differential privacy, fairness, databases},}
Critical Perspectives: A Benchmark Revealing Pitfalls in PerspectiveAPI
Lucas Rosenblatt, Lorena Piedras, and Julia Wilkins
In Proceedings of the Second Workshop on NLP for Positive Impact (NLP4PI) 2022
@inproceedings{rosenblatt2022critical,title={Critical Perspectives: A Benchmark Revealing Pitfalls in PerspectiveAPI},author={Rosenblatt, Lucas and Piedras, Lorena and Wilkins, Julia},booktitle={Proceedings of the Second Workshop on NLP for Positive Impact (NLP4PI)},pages={15--24},year={2022},keywords={fairness, nlp, benchmark},}
Spending Privacy Budget Fairly and Wisely
Lucas Rosenblatt, Joshua Allen, and Julia Stoyanovich
@inproceedings{DBLP:conf/eaamo/ArifKhanMS22,author={{Arif Khan}, Falaah and Manis, Eleni and Stoyanovich, Julia},title={Towards Substantive Conceptions of Algorithmic Fairness: Normative
Guidance from Equal Opportunity Doctrines},booktitle={Equity and Access in Algorithms, Mechanisms, and Optimization, {EAAMO}
2022, Arlington, VA, USA, October 6-9, 2022},pages={18:1--18:10},publisher={{ACM}},year={2022},url={https://doi.org/10.1145/3551624.3555303},doi={10.1145/3551624.3555303},keywords={conference, fairness},}
2021
The Many Facets of Data Equity
H. V. Jagadish, Julia Stoyanovich, and Bill Howe
In Proceedings of the Workshops of the EDBT/ICDT 2021 Joint Conference, Nicosia, Cyprus, March 23, 2021 2021
@inproceedings{DBLP:conf/edbt/JagadishSH21,author={Jagadish, H. V. and Stoyanovich, Julia and Howe, Bill},editor={Costa, Constantinos and Pitoura, Evaggelia},title={The Many Facets of Data Equity},booktitle={Proceedings of the Workshops of the {EDBT/ICDT} 2021 Joint Conference,
Nicosia, Cyprus, March 23, 2021},series={{CEUR} Workshop Proceedings},volume={2841},publisher={CEUR-WS.org},year={2021},keywords={workshop,data,fairness},}
COVID-19 Brings Data Equity Challenges to the Fore
@article{DBLP:journals/dgov/JagadishSH21,author={Jagadish, H. V. and Stoyanovich, Julia and Howe, Bill},title={{COVID-19} Brings Data Equity Challenges to the Fore},journal={Digit. Gov. Res. Pract.},volume={2},number={2},pages={24:1--24:7},year={2021},doi={10.1145/3440889},keywords={journal,data,fairness},}
Comparing Apples and Oranges: Fairness and Diversity in Ranking (Invited Talk)
Julia Stoyanovich
In 24th International Conference on Database Theory, ICDT 2021, March 23-26, 2021, Nicosia, Cyprus 2021
@inproceedings{DBLP:conf/icdt/Stoyanovich21,author={Stoyanovich, Julia},editor={Yi, Ke and Wei, Zhewei},title={Comparing Apples and Oranges: Fairness and Diversity in Ranking (Invited
Talk)},booktitle={24th International Conference on Database Theory, {ICDT} 2021, March
23-26, 2021, Nicosia, Cyprus},series={LIPIcs},volume={186},pages={2:1--2:1},publisher={Schloss Dagstuhl - Leibniz-Zentrum f{\"{u}}r Informatik},year={2021},doi={10.4230/LIPIcs.ICDT.2021.2},bibsource={dblp computer science bibliography, https://dblp.org},keywords={invited,fairness,ranking},author+an={1=self},}
@article{DBLP:journals/pvldb/StoyanovichHJ20,title={Responsible Data Management},author={Stoyanovich, Julia and Howe, Bill and Jagadish, HV},journal={Proceedings of the VLDB Endowment},volume={13},number={12},year={2020},keywords={fairness,data,journal},doi={10.14778/3415478.3415570},}
Fairness-Aware Instrumentation of Preprocessing Pipelines for Machine Learning
Ke Yang, Biao Huang, Julia Stoyanovich, and Sebastian Schelter
In Proceedings of the Workshop on Human-In-the-Loop Data Analytics, HILDA at SIGMOD 2020
@inproceedings{DBLP:hilda20,author={Yang, Ke and Huang, Biao and Stoyanovich, Julia and Schelter, Sebastian},title={Fairness-Aware Instrumentation of Preprocessing~Pipelines for Machine Learning},booktitle={Proceedings of the Workshop on Human-In-the-Loop Data Analytics, {HILDA} at {SIGMOD}},year={2020},doi={https://doi.org/10.1145/3398730.3399194},keywords={workshop,data,fairness},}
Taming Technical Bias in Machine Learning Pipelines
@article{debull21,author={Schelter, Sebastian and Stoyanovich, Julia},title={Taming Technical Bias in Machine Learning Pipelines},journal={{IEEE} Data Eng. Bull.},year={2020},volume={43},number={4},keywords={journal,data,fairness},author+an={2=self},}
2019
Transparency, Fairness, Data Protection, Neutrality: Data Management Challenges in the Face of New Regulation
@article{DBLP:journals/jdiq/AbiteboulS19,author={Abiteboul, Serge and Stoyanovich, Julia},title={Transparency, Fairness, Data Protection, Neutrality: Data Management
Challenges in the Face of New Regulation},journal={ACM Journal of Data and Information Quality},volume={11},number={3},pages={15:1--15:9},year={2019},doi={10.1145/3310231},timestamp={Tue, 20 Aug 2019 08:40:18 +0200},bibsource={dblp computer science bibliography, https://dblp.org},keywords={journal,data,fairness,policy,governance},}
MithraRanking: A System for Responsible Ranking Design
Yifan Guan, Abolfazl Asudeh, Pranav Mayuram, H. V. Jagadish, Julia Stoyanovich, Gerome Miklau, and Gautam Das
In Proceedings of the 2019 International Conference on the Management of Data, SIGMOD 2019
@inproceedings{DBLP:conf/sigmod/GuanAMJSM019,author={Guan, Yifan and Asudeh, Abolfazl and Mayuram, Pranav and Jagadish, H. V. and Stoyanovich, Julia and Miklau, Gerome and Das, Gautam},title={{M}ithra{R}anking: {A} System for Responsible Ranking Design},booktitle={Proceedings of the 2019 International Conference on the Management of
Data, {SIGMOD}},pages={1913--1916},publisher={{ACM}},year={2019},doi={10.1145/3299869.3320244},timestamp={Sat, 22 Jun 2019 17:10:04 +0200},bibsource={dblp computer science bibliography, https://dblp.org},keywords={conference,fairness,ranking},addendum={demonstration}}
2017
Measuring Fairness in Ranked Outputs
Ke Yang, and Julia Stoyanovich
In Proceedings of the 29th International Conference on Scientific and Statistical Database Management, SSDBM 2017
@inproceedings{DBLP:conf/ssdbm/YangS17,author={Yang, Ke and Stoyanovich, Julia},title={Measuring Fairness in Ranked Outputs},booktitle={Proceedings of the 29th International Conference on Scientific and
Statistical Database Management, {SSDBM}},pages={22:1--22:6},publisher={{ACM}},year={2017},doi={10.1145/3085504.3085526},timestamp={Tue, 06 Nov 2018 16:57:31 +0100},bibsource={dblp computer science bibliography, https://dblp.org},keywords={fairness, ranking, data management},author+an={1=self;2=self}}
Policy
We engage in technology policy work in the US and internationally. These practical engagements are based on our research, highlighted below.
2023
Introducing contextual transparency for automated decision systems
Mona Sloane, Ian Solano-Kamaiko, Jun Yuan, Aritra Dasgupta, and Julia Stoyanovich
@article{NMI_2023,author={Sloane, Mona and Solano-Kamaiko, Ian and Yuan, Jun and Dasgupta, Aritra and Stoyanovich, Julia},title={Introducing contextual transparency for automated decision systems},journal={Nature Machine Intelligence},volume={5},year={2023},pages={187-–195},doi={https://doi.org/10.1038/s42256-023-00623-7},keywords={journal,explainability,policy,hiring},}
The Algorithmic Transparency Playbook: A Stakeholder-first Approach to Creating Transparency for Your Organization’s Algorithms
Andrew Bell, Oded Nov, and Julia Stoyanovich
In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, CHI EA 2023, Hamburg, Germany, April 23-28, 2023 2023
@inproceedings{DBLP:conf/chi/BellNS23,author={Bell, Andrew and Nov, Oded and Stoyanovich, Julia},title={The Algorithmic Transparency Playbook: {A} Stakeholder-first Approach
to Creating Transparency for Your Organization's Algorithms},booktitle={Extended Abstracts of the 2023 {CHI} Conference on Human Factors in
Computing Systems, {CHI} {EA} 2023, Hamburg, Germany, April 23-28,
2023},pages={554:1--554:4},publisher={{ACM}},year={2023},site={https://r-ai.co/transparency-playbook},doi={10.1145/3544549.3574169},keywords={panel,policy,explanability,education,playbook,governance},addendum={peer-reviewed course},author+an={1=self;3=self}}
Think About the Stakeholders First! Towards an Algorithmic Transparency Playbook for Regulatory Compliance
@article{bell_nov_stoyanovich_2023,author={Bell, Andrew and Nov, Oded and Stoyanovich, Julia},title={Think About the Stakeholders First! {T}owards an Algorithmic Transparency
Playbook for Regulatory Compliance},volume={5},journal={Data \& Policy},publisher={Cambridge University Press},year={2023},keywords={journal,policy,explainability,education,playbook,governance},}
2022
It’s Just Not That Simple: An Empirical Study of the Accuracy-Explainability Trade-off in Machine Learning for Public Policy
Andrew Bell, Ian Solano-Kamaiko, Oded Nov, and Julia Stoyanovich
In Proceedings of the 5th Annual ACM Conference on Fairness, Accountability, and Transparency, FAccT 2022
@inproceedings{DBLP:conf/fat/BellSNS22,author={Bell, Andrew and Solano{-}Kamaiko, Ian and Nov, Oded and Stoyanovich, Julia},title={It's Just Not That Simple: An Empirical Study of the Accuracy-Explainability
Trade-off in Machine Learning for Public Policy},booktitle={Proceedings of the 5th Annual {ACM} Conference on Fairness, Accountability, and
Transparency, {FAccT}},pages={248--266},publisher={{ACM}},year={2022},doi={10.1145/3531146.3533090},timestamp={Wed, 22 Jun 2022 10:20:30 +0200},bibsource={dblp computer science bibliography, https://dblp.org},keywords={explainability, transparency, policy,governance},}
Developing data capability with non-profit organisations using participatory methods
Anthony McCosker, Xiaofang Yao, Kath Albury, Alexia Maddox, Jane Farmer, and Julia Stoyanovich
@article{doi:10.1177/20539517221099882,author={McCosker, Anthony and Yao, Xiaofang and Albury, Kath and Maddox, Alexia and Farmer, Jane and Stoyanovich, Julia},title={Developing data capability with non-profit organisations using participatory methods},journal={Big Data \& Society},volume={9},number={1},pages={20539517221099882},year={2022},doi={10.1177/20539517221099882},eprint={https://doi.org/10.1177/20539517221099882},keywords={journal,data,policy,governance},}
2019
Transparency, Fairness, Data Protection, Neutrality: Data Management Challenges in the Face of New Regulation
@article{DBLP:journals/jdiq/AbiteboulS19,author={Abiteboul, Serge and Stoyanovich, Julia},title={Transparency, Fairness, Data Protection, Neutrality: Data Management
Challenges in the Face of New Regulation},journal={ACM Journal of Data and Information Quality},volume={11},number={3},pages={15:1--15:9},year={2019},doi={10.1145/3310231},timestamp={Tue, 20 Aug 2019 08:40:18 +0200},bibsource={dblp computer science bibliography, https://dblp.org},keywords={journal,data,fairness,policy,governance},}
2018
Follow the Data! Algorithmic Transparency Starts with Data Transparency
@article{Follow,title={Follow the Data! {A}lgorithmic Transparency Starts with Data Transparency},author={Stoyanovich, Julia and Howe, Bill},journal={The Ethical Machine},year={2018},month=nov,day={27},publisher={Shorenstein Center on Media, Politics, and Public Policy, Harvard Kennedy School},keywords={public,data,explainability,policy},author+an={1=self}}
2017
Privacy
Our work in privacy includes peer-reviewed papers, as well as tools and benchmarks. Take a look at the Data Synthesizer tool that we've been using to teach differentially private data synthesis. And check out our latest DP synethetic data benchmarking package, SynRD, posing the question: "Can a DP synthesizer produce private (tabular) data that preserves scientific findings?" In other words, do DP synthesizers satisfy epistemic parity?
2023
Personal Data for Personal Use: Vision or Reality?
Xin Luna Dong, Bo Li, Julia Stoyanovich, Anthony Kum Hoe Tung, Gerhard Weikum, Alon Y. Halevy, and Wang-Chiew Tan
In Companion of the 2023 International Conference on Management of Data, SIGMOD/PODS 2023, Seattle, WA, USA, June 18-23, 2023 2023
@inproceedings{DBLP:conf/sigmod/DongLSTWHT23,author={Dong, Xin Luna and Li, Bo and Stoyanovich, Julia and Tung, Anthony Kum Hoe and Weikum, Gerhard and Halevy, Alon Y. and Tan, Wang{-}Chiew},title={Personal Data for Personal Use: Vision or Reality?},booktitle={Companion of the 2023 International Conference on Management of Data,
{SIGMOD/PODS} 2023, Seattle, WA, USA, June 18-23, 2023},pages={263--264},publisher={{ACM}},year={2023},doi={10.1145/3555041.3589378},keywords={panel,data,privacy},addendum={panelist, by invitation},}
Epistemic Parity: Reproducibility as an Evaluation Metric for Differential Privacy
Lucas Rosenblatt, Bernease Herman, Anastasia Holovenko, Wonkwon Lee, Joshua R. Loftus, Elizabeth Mckinnie, Taras Rumezhak, Andrii Stadnik, Bill Howe, and Julia Stoyanovich
@article{DBLP:journals/pvldb/RosenblattHHLLM23,author={Rosenblatt, Lucas and Herman, Bernease and Holovenko, Anastasia and Lee, Wonkwon and Loftus, Joshua R. and Mckinnie, Elizabeth and Rumezhak, Taras and Stadnik, Andrii and Howe, Bill and Stoyanovich, Julia},title={Epistemic Parity: Reproducibility as an Evaluation Metric for Differential
Privacy},journal={Proc. {VLDB} Endow.},volume={16},number={11},pages={3178--3191},year={2023},doi={10.14778/3611479.3611517},timestamp={Mon, 23 Oct 2023 16:16:16 +0200},biburl={https://dblp.org/rec/journals/pvldb/RosenblattHHLLM23.bib},keywords={journal,privacy},author+an={1=self;3=self;4=self;6=self;7=self;8=self;10=self}}
2022
Spending Privacy Budget Fairly and Wisely
Lucas Rosenblatt, Joshua Allen, and Julia Stoyanovich
Theory and Practice of Differential Privacy (@ICML) 2022
@article{rosenblatt2022spending,title={Spending Privacy Budget Fairly and Wisely},author={Rosenblatt, Lucas and Allen, Joshua and Stoyanovich, Julia},journal={Theory and Practice of Differential Privacy (@ICML)},year={2022},keywords={differential privacy, fairness, databases},}
Spending Privacy Budget Fairly and Wisely
Lucas Rosenblatt, Joshua Allen, and Julia Stoyanovich
@article{rosenblatt2020differentially,title={Differentially private synthetic data: Applied evaluations and enhancements},author={Rosenblatt, Lucas and Liu, Xiaoyan and Pouyanfar, Samira and de Leon, Eduardo and Desai, Anuj and Allen, Joshua},journal={arXiv preprint arXiv:2011.05537},year={2020},keywords={differential privacy, benchmark},}
@inproceedings{DBLP:conf/vldb/RodriguezSPSH19,author={Rodriguez, Luke and Salimi, Babak and Ping, Haoyue and Stoyanovich, Julia and Howe, Bill},editor={Oliveira, Jonice and de Farias, Claudio M. and Pacitti, Esther and Fortino, Giancarlo},title={MobilityMirror: Bias-Adjusted Transportation Datasets},booktitle={Big Social Data and Urban Computing - First Workshop, BiDU@VLDB 2018,
Rio de Janeiro, Brazil, August 31, 2018, Revised Selected Papers},series={Communications in Computer and Information Science},volume={926},pages={18--39},publisher={Springer},year={2018},url={https://doi.org/10.1007/978-3-030-11238-7_2},doi={10.1007/978-3-030-11238-7_2},keywords={privacy},timestamp={Tue, 29 Jan 2019 14:00:34 +0100},biburl={https://dblp.org/rec/conf/vldb/RodriguezSPSH18.bib},bibsource={dblp computer science bibliography, https://dblp.org}}
@inproceedings{DBLP:conf/ssdbm/PingSH17,author={Ping, Haoyue and Stoyanovich, Julia and Howe, Bill},title={{DataSynthesizer}: Privacy-Preserving Synthetic Datasets},booktitle={Proceedings of the 29th International Conference on Scientific and
Statistical Database Management, {SSDBM}},pages={42:1--42:5},publisher={{ACM}},year={2017},doi={10.1145/3085504.3091117},timestamp={Tue, 06 Nov 2018 16:57:31 +0100},bibsource={dblp computer science bibliography, https://dblp.org},keywords={conference,privacy},author+an={1=self;3=self}}
Ranking
One kind of algorithm that is at once especially obscure, powerful, and common is the ranking algorithm. Algorithms rank individuals to determine credit worthiness, desirability for college admissions and employment, and compatibility as dating partners. They encode ideas of what counts as the best schools, neighborhoods, and technologies. Despite their importance, we actually can know very little about why one person was ranked higher than another in a dating app, or why one school has a better rank than that one. This is true even if we have access to the ranking algorithm, for example, if we have complete knowledge about the factors used by the ranker and their relative weights, as is the case for US News ranking of colleges. We have been working on several aspects of responsible ranking design and use, including fairness, transparency and interpretability, and stability.
2023
Fairness in Ranking: From Values to Technical Choices and Back
Julia Stoyanovich, Meike Zehlike, and Ke Yang
In Companion of the 2023 International Conference on Management of Data, SIGMOD/PODS 2023, Seattle, WA, USA, June 18-23, 2023 2023
@inproceedings{DBLP:conf/sigmod/StoyanovichZY23,author={Stoyanovich, Julia and Zehlike, Meike and Yang, Ke},editor={Das, Sudipto and Pandis, Ippokratis and Candan, K. Sel{\c{c}}uk and Amer{-}Yahia, Sihem},title={Fairness in Ranking: From Values to Technical Choices and Back},booktitle={Companion of the 2023 International Conference on Management of Data,
{SIGMOD/PODS} 2023, Seattle, WA, USA, June 18-23, 2023},pages={7--12},publisher={{ACM}},year={2023},doi={10.1145/3555041.3589405},keywords={panel,fairness,ranking},addendum={peer-reviewed tutorial},}
@article{10.1145/3533379,author={Zehlike, Meike and Yang, Ke and Stoyanovich, Julia},title={Fairness in Ranking, Part I: Score-Based Ranking},journal={{ACM} Computing Surveys},volume={55},number={6},pages={118:1--118:36},year={2023},doi={10.1145/3533379},keywords={journal,fair,ranking},author+an={2=self;3=self},}
Fairness in Ranking, Part II: Learning-to-Rank and Recommender Systems
@article{10.1145/3533380,author={Zehlike, Meike and Yang, Ke and Stoyanovich, Julia},title={Fairness in Ranking, Part II: Learning-to-Rank and Recommender Systems},journal={{ACM} Computing Surveys},volume={55},number={6},pages={117:1--117:41},year={2023},doi={10.1145/3533380},keywords={journal,fair,ranking},}
2022
Rankers, Rankees, & Rankings: Peeking into the Pandora’s Box from a Socio-Technical Perspective
@article{DBLP:journals/corr/abs-2211-02932,author={Yuan, Jun and Stoyanovich, Julia and Dasgupta, Aritra},title={Rankers, Rankees, {\&} Rankings: Peeking into the Pandora's Box
from a Socio-Technical Perspective},journal={CoRR},volume={abs/2211.02932},year={2022},url={https://doi.org/10.48550/arXiv.2211.02932},doi={10.48550/arXiv.2211.02932},keywords={working,ranking,explainability},eprinttype={arXiv},eprint={2211.02932},timestamp={Wed, 09 Nov 2022 17:33:26 +0100},biburl={https://dblp.org/rec/journals/corr/abs-2211-02932.bib},bibsource={dblp computer science bibliography, https://dblp.org},}
2021
Causal Intersectionality and Fair Ranking
Ke Yang, Joshua Loftus, and Julia Stoyanovich
In Symposium on the Foundations of Responsible Computing FORC 2021
@inproceedings{forc21,author={Yang, Ke and Loftus, Joshua and Stoyanovich, Julia},title={Causal Intersectionality and Fair Ranking},booktitle={Symposium on the Foundations of Responsible Computing {FORC}},year={2021},doi={10.4230/LIPIcs.FORC.2021.7},keywords={conference,fair,ranking},author+an={1=self;3=self}}
Comparing Apples and Oranges: Fairness and Diversity in Ranking (Invited Talk)
Julia Stoyanovich
In 24th International Conference on Database Theory, ICDT 2021, March 23-26, 2021, Nicosia, Cyprus 2021
@inproceedings{DBLP:conf/icdt/Stoyanovich21,author={Stoyanovich, Julia},editor={Yi, Ke and Wei, Zhewei},title={Comparing Apples and Oranges: Fairness and Diversity in Ranking (Invited
Talk)},booktitle={24th International Conference on Database Theory, {ICDT} 2021, March
23-26, 2021, Nicosia, Cyprus},series={LIPIcs},volume={186},pages={2:1--2:1},publisher={Schloss Dagstuhl - Leibniz-Zentrum f{\"{u}}r Informatik},year={2021},doi={10.4230/LIPIcs.ICDT.2021.2},bibsource={dblp computer science bibliography, https://dblp.org},keywords={invited,fairness,ranking},author+an={1=self},}
2019
Towards Responsible Data-driven Decision Making in Score-Based Systems
Abolfazl Asudeh, H. V. Jagadish, and Julia Stoyanovich
@article{DBLP:journals/debu/AsudehJS19,author={Asudeh, Abolfazl and Jagadish, H. V. and Stoyanovich, Julia},title={Towards Responsible Data-driven Decision Making in Score-Based Systems},journal={{IEEE} Data Eng. Bull.},volume={42},number={3},pages={76--87},year={2019},timestamp={Tue, 10 Mar 2020 16:23:50 +0100},bibsource={dblp computer science bibliography, https://dblp.org},keywords={journal,fair,ranking},author+an={3=self}}
Balanced Ranking with Diversity Constraints
Ke Yang, Vasilis Gkatzelis, and Julia Stoyanovich
In Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI 2019
@inproceedings{DBLP:conf/ijcai/YangGS19,author={Yang, Ke and Gkatzelis, Vasilis and Stoyanovich, Julia},title={Balanced Ranking with Diversity Constraints},booktitle={Proceedings of the 28th International Joint Conference on
Artificial Intelligence, {IJCAI}},pages={6035--6042},publisher={ijcai.org},year={2019},doi={10.24963/ijcai.2019/836},timestamp={Tue, 20 Aug 2019 16:18:18 +0200},bibsource={dblp computer science bibliography, https://dblp.org},keywords={conference,fair,ranking},author+an={1=self;3=self}}
Designing Fair Ranking Schemes
Abolfazl Asudeh, H. V. Jagadish, Julia Stoyanovich, and Gautam Das
In Proceedings of the 2019 International Conference on the Management of Data, SIGMOD 2019
@inproceedings{DBLP:conf/sigmod/AsudehJS019,author={Asudeh, Abolfazl and Jagadish, H. V. and Stoyanovich, Julia and Das, Gautam},title={Designing Fair Ranking Schemes},booktitle={Proceedings of the 2019 International Conference on the Management of
Data, {SIGMOD}},pages={1259--1276},publisher={{ACM}},year={2019},doi={10.1145/3299869.3300079},timestamp={Sat, 22 Jun 2019 17:10:04 +0200},bibsource={dblp computer science bibliography, https://dblp.org},keywords={conference,fair,ranking},author+an={3=self}}
MithraRanking: A System for Responsible Ranking Design
Yifan Guan, Abolfazl Asudeh, Pranav Mayuram, H. V. Jagadish, Julia Stoyanovich, Gerome Miklau, and Gautam Das
In Proceedings of the 2019 International Conference on the Management of Data, SIGMOD 2019
@inproceedings{DBLP:conf/sigmod/GuanAMJSM019,author={Guan, Yifan and Asudeh, Abolfazl and Mayuram, Pranav and Jagadish, H. V. and Stoyanovich, Julia and Miklau, Gerome and Das, Gautam},title={{M}ithra{R}anking: {A} System for Responsible Ranking Design},booktitle={Proceedings of the 2019 International Conference on the Management of
Data, {SIGMOD}},pages={1913--1916},publisher={{ACM}},year={2019},doi={10.1145/3299869.3320244},timestamp={Sat, 22 Jun 2019 17:10:04 +0200},bibsource={dblp computer science bibliography, https://dblp.org},keywords={conference,fairness,ranking},addendum={demonstration}}
2018
On Obtaining Stable Rankings
Abolfazl Asudeh, H. V. Jagadish, Gerome Miklau, and Julia Stoyanovich
@article{DBLP:journals/pvldb/AsudehJMS18,author={Asudeh, Abolfazl and Jagadish, H. V. and Miklau, Gerome and Stoyanovich, Julia},title={On Obtaining Stable Rankings},journal={PVLDB},volume={12},number={3},pages={237--250},year={2018},doi={10.14778/3291264.3291269},timestamp={Sat, 25 Apr 2020 13:58:47 +0200},bibsource={dblp computer science bibliography, https://dblp.org},keywords={journal,stable,ranking},author+an={4=self}}
Online Set Selection with Fairness and Diversity Constraints
Julia Stoyanovich, Ke Yang, and H. V. Jagadish
In Proceedings of the 21th International Conference on Extending Database Technology, EDBT 2018
@inproceedings{DBLP:conf/edbt/StoyanovichYJ18,author={Stoyanovich, Julia and Yang, Ke and Jagadish, H. V.},title={Online Set Selection with Fairness and Diversity Constraints},booktitle={Proceedings of the 21th International Conference on Extending Database
Technology, {EDBT}},pages={241--252},publisher={OpenProceedings.org},year={2018},doi={10.5441/002/edbt.2018.22},timestamp={Fri, 16 Mar 2018 23:12:22 +0100},bibsource={dblp computer science bibliography, https://dblp.org},keywords={conference,fair,ranking},author+an={1=self;2=self}}
A Nutritional Label for Rankings
Ke Yang, Julia Stoyanovich, Abolfazl Asudeh, Bill Howe, H. V. Jagadish, and Gerome Miklau
In Proceedings of the 2018 International Conference on the Management of Data, SIGMOD 2018
@inproceedings{DBLP:conf/sigmod/YangSAHJM18,author={Yang, Ke and Stoyanovich, Julia and Asudeh, Abolfazl and Howe, Bill and Jagadish, H. V. and Miklau, Gerome},title={A Nutritional Label for Rankings},booktitle={Proceedings of the 2018 International Conference on the Management of
Data, {SIGMOD}},publisher={{ACM}},year={2018},doi={10.1145/3183713.3193568},timestamp={Wed, 21 Nov 2018 12:44:08 +0100},bibsource={dblp computer science bibliography, https://dblp.org},keywords={conference,ranking,explanability},}
Refining the Concept of a Nutritional Label for Data and Models
Julia Stoyanovich, and Bill Howe
Freedom to Tinker, Center for Information Technology Policy, Princeton University May 2018
@article{citp_2,author={Stoyanovich, Julia and Howe, Bill},title={Refining the Concept of a Nutritional Label for Data and Models},journal={Freedom to Tinker, Center for Information Technology Policy, Princeton University},year={2018},month=may,day={3},keywords={public,ranking,explainability},}
2017
Measuring Fairness in Ranked Outputs
Ke Yang, and Julia Stoyanovich
In Proceedings of the 29th International Conference on Scientific and Statistical Database Management, SSDBM May 2017
@inproceedings{DBLP:conf/ssdbm/YangS17,author={Yang, Ke and Stoyanovich, Julia},title={Measuring Fairness in Ranked Outputs},booktitle={Proceedings of the 29th International Conference on Scientific and
Statistical Database Management, {SSDBM}},pages={22:1--22:6},publisher={{ACM}},year={2017},doi={10.1145/3085504.3085526},timestamp={Tue, 06 Nov 2018 16:57:31 +0100},bibsource={dblp computer science bibliography, https://dblp.org},keywords={fairness, ranking, data management},author+an={1=self;2=self}}
2016
Revealing Algorithmic Rankers
Julia Stoyanovich, and Ellen P. Goodman
Freedom to Tinker, Center for Information Technology Policy, Princeton University Aug 2016
@article{citp_1,author={Stoyanovich, Julia and Goodman, Ellen P.},title={Revealing Algorithmic Rankers},journal={Freedom to Tinker, Center for Information Technology Policy, Princeton University},year={2016},month=aug,day={5},keywords={public,ranking,explainability},author+an={1=self}}
2015
Analyzing Crowd Rankings
Julia Stoyanovich, Marie Jacob, and Xuemei Gong
In Proceedings of the 18th International Workshop on Web and Databases, Melbourne, VIC, Australia, May 31, 2015 Aug 2015
@inproceedings{DBLP:conf/webdb/StoyanovichJG16,author={Stoyanovich, Julia and Jacob, Marie and Gong, Xuemei},editor={Stoyanovich, Julia and Suchanek, Fabian M.},title={Analyzing Crowd Rankings},booktitle={Proceedings of the 18th International Workshop on Web and Databases,
Melbourne, VIC, Australia, May 31, 2015},pages={41--47},keywords={ranking},publisher={{ACM}},year={2015},url={https://doi.org/10.1145/2767109.2767110},doi={10.1145/2767109.2767110},timestamp={Tue, 06 Nov 2018 16:58:14 +0100},biburl={https://dblp.org/rec/conf/webdb/StoyanovichJG15.bib},bibsource={dblp computer science bibliography, https://dblp.org}}
Responsible AI Education
We cannot understand the impact – and especially the risks – of AI systems without active and thoughtful participation of everyone in society, either directly or through their trusted representatives. To think otherwise is to go against our democratic values. To enable broad participation, we have been developing responsible AI curricula and methodologies for different stakeholders: university students, working practitioners, and the public at large. In this section, you will find our publication on responsible AI education. Take a look at the education area of the site to access our courses and other open-source materials we have developed.
2023
Responsible AI literacy: A stakeholder-first approach
@article{DominguezStoyanovich23,author={Dominguez, Daniel and Stoyanovich, Julia},title={Responsible AI literacy: A stakeholder-first approach},journal={Big Data and Society},year={2023},keywords={journal,education,rds},doi={https://doi.org/10.1177/20539517231219958},}
All Aboard! Making AI Education Accessible
Falaah Arif Khan, Lucius Bynum, Amy Hurst, Lucas Rosenblatt, Meghana Shanbhogue, Mona Sloane, and Julia Stoyanovich
Center for Responsible AI, New York University 2023
@article{AllAboard,author={{Arif Khan}, Falaah and Bynum, Lucius and Hurst, Amy and Rosenblatt, Lucas and Shanbhogue, Meghana and Sloane, Mona and Stoyanovich, Julia},title={{All Aboard! Making AI Education Accessible}},journal={Center for Responsible AI, New York University},keywords={panel,education,weareai},year={2023},}
The Algorithmic Transparency Playbook: A Stakeholder-first Approach to Creating Transparency for Your Organization’s Algorithms
Andrew Bell, Oded Nov, and Julia Stoyanovich
In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, CHI EA 2023, Hamburg, Germany, April 23-28, 2023 2023
@inproceedings{DBLP:conf/chi/BellNS23,author={Bell, Andrew and Nov, Oded and Stoyanovich, Julia},title={The Algorithmic Transparency Playbook: {A} Stakeholder-first Approach
to Creating Transparency for Your Organization's Algorithms},booktitle={Extended Abstracts of the 2023 {CHI} Conference on Human Factors in
Computing Systems, {CHI} {EA} 2023, Hamburg, Germany, April 23-28,
2023},pages={554:1--554:4},publisher={{ACM}},year={2023},site={https://r-ai.co/transparency-playbook},doi={10.1145/3544549.3574169},keywords={panel,policy,explanability,education,playbook,governance},addendum={peer-reviewed course},author+an={1=self;3=self}}
Think About the Stakeholders First! Towards an Algorithmic Transparency Playbook for Regulatory Compliance
@article{bell_nov_stoyanovich_2023,author={Bell, Andrew and Nov, Oded and Stoyanovich, Julia},title={Think About the Stakeholders First! {T}owards an Algorithmic Transparency
Playbook for Regulatory Compliance},volume={5},journal={Data \& Policy},publisher={Cambridge University Press},year={2023},keywords={journal,policy,explainability,education,playbook,governance},}
2022
An Interactive Introduction to Causal Inference
Lucius E.J. Bynum, Falaah Arif Khan, Oleksandra Konopatska, Joshua R. Loftus, and Julia Stoyanovich
VISxAI: Workshop on Visualization for AI Explainability 2022
@article{bynum2022interactive,author={Bynum, Lucius E.J. and Khan, Falaah Arif and Konopatska, Oleksandra and Loftus, Joshua R. and Stoyanovich, Julia},title={An Interactive Introduction to Causal Inference},journal={VISxAI: Workshop on Visualization for AI Explainability},year={2022},site={https://r-ai.co/ci-playground},publisher={IEEE},keywords={workshop,education,playground},}
Teaching Responsible Data Science
Julia Stoyanovich
In Proceedings of the 1st ACM SIGMOD International Workshop on Data Systems Education: Bridging Education Practice with Education Research, DataEd@SIGMOD 2022, 17 June 2022, Philadelphia, PA, USA 2022
@inproceedings{DBLP:conf/dataed/Stoyanovich22,author={Stoyanovich, Julia},editor={Aivaloglou, Efthimia and Fletcher, George and Miedema, Daphne},title={Teaching Responsible Data Science},booktitle={Proceedings of the 1st {ACM} {SIGMOD} International Workshop on Data
Systems Education: Bridging Education Practice with Education Research,
DataEd@SIGMOD 2022, 17 June 2022, Philadelphia, PA, {USA}},pages={4--9},publisher={{ACM}},year={2022},doi={10.1145/3531072.3535318},timestamp={Wed, 29 Jun 2022 12:08:29 +0200},bibsource={dblp computer science bibliography, https://dblp.org},keywords={invited,education},}
Teaching Responsible Data Science
Armanda Lewis, and Julia Stoyanovich
International Journal of Artificial Intelligence in Education (IJAIED) 2022
@article{LewisStoyanovich21,author={Lewis, Armanda and Stoyanovich, Julia},title={Teaching Responsible Data Science},journal={International Journal of Artificial Intelligence in Education (IJAIED)},volume={32},number={3},pages={783--807},year={2022},keywords={journal,education,rds},addendum={Special Issue: The FATE of AI in Education: Fairness, Accountability, Transparency, and Ethics},doi={https://doi.org/10.1007/s40593-021-00241-7},site={https://dataresponsibly.github.io/rds/},}
@article{weareaicomic_vol1,author={Stoyanovich, Julia and Khan, Falaah Arif},title={What is AI?},journal={We are AI Comic Series},volume={1},year={2021},keywords={public,education,comics,english},}
@article{weareaicomic_vol1_sp,author={Stoyanovich, Julia and Khan, Falaah Arif},title={What is AI? (Spanish Edition)},journal={We are AI Comic Series},volume={1},year={2021},keywords={public,education,comics,spanish},}
@article{weareaicomic_vol1_gr,author={Stoyanovich, Julia and Khan, Falaah Arif},title={What is AI? (Greek Edition)},journal={We are AI Comic Series},volume={1},year={2021},keywords={public,education,comics,other},}
@article{weareaicomic_vol2,author={Stoyanovich, Julia and Khan, Falaah Arif},title={Learning from data},journal={We are AI Comic Series},volume={2},year={2021},keywords={public,education,comics,english},}
@article{weareaicomic_vol2_sp,author={Stoyanovich, Julia and Khan, Falaah Arif},title={Learning from data (Spanish Edition)},journal={We are AI Comic Series},volume={2},year={2021},keywords={public,education,comics,spanish},}
Who lives, who dies, who decides?
Julia Stoyanovich, Mona Sloane, and Falaah Arif Khan
@article{weareaicomic_vol3,author={Stoyanovich, Julia and Sloane, Mona and Khan, Falaah Arif},title={Who lives, who dies, who decides?},journal={We are AI Comic Series},volume={3},year={2021},keywords={public,education,comics,english},}
Who lives, who dies, who decides? (Spanish Edition)
Julia Stoyanovich, Mona Sloane, and Falaah Arif Khan
@article{weareaicomic_vol3_sp,author={Stoyanovich, Julia and Sloane, Mona and Khan, Falaah Arif},title={Who lives, who dies, who decides? (Spanish Edition)},journal={We are AI Comic Series},volume={3},year={2021},keywords={public,education,comics,spanish},}
@article{weareaicomic_vol4,author={Stoyanovich, Julia and Khan, Falaah Arif},title={All about that bias},journal={We are AI Comic Series},volume={4},year={2021},keywords={public,education,comics,english}}
@article{weareaicomic_vol4_sp,author={Stoyanovich, Julia and Khan, Falaah Arif},title={All about that bias (Spanish Edition},journal={We are AI Comic Series},volume={4},year={2021},keywords={public,education,comics,spanish}}
@article{weareaicomic_vol5,author={Stoyanovich, Julia and Khan, Falaah Arif},title={We are AI},journal={We are AI Comic Series},volume={5},year={2021},keywords={public,education,comics,english},}
@article{weareaicomic_vol5_sp,author={Stoyanovich, Julia and Khan, Falaah Arif},title={We are AI},journal={We are AI Comic Series},volume={5},year={2021},keywords={public,education,comics,english},}
Fairness and Friends
Falaah Arif Khan, Eleni Manis, and Julia Stoyanovich
@article{rdscomic_Vol2,author={Khan, Falaah Arif and Manis, Eleni and Stoyanovich, Julia},title={Fairness and Friends},journal={Data, Responsibly Comic Series},volume={2},year={2021},keywords={public,edu,comics,english},}
We are AI: Taking Control of Technology
Julia Stoyanovich, and Eric Corbett
Center for Responsible AI, New York University 2021
@article{rdscomic_vol1,author={Khan, Falaah Arif and Stoyanovich, Julia},title={Mirror, Mirror},journal={Data, Responsibly Comic Series},volume={1},year={2020},keywords={public,edu,comics,english},}
The Algorithmic Transparency Playbook: A Stakeholder-first Approach to Creating Transparency for Your Organization’s AlgorithmsIn Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, CHI EA 2023, Hamburg, Germany, April 23-28, 2023 2023
An Interactive Introduction to Causal InferenceVISxAI: Workshop on Visualization for AI Explainability 2022
Teaching Responsible Data ScienceInternational Journal of Artificial Intelligence in Education (IJAIED) 2022
We are AI: Taking Control of TechnologyCenter for Responsible AI, New York University 2021
