Recommender systems are used in a many different domains, from recommending recipes to song playlists. They increasingly act as "gatekeepers" to what users are exposed to online. Therefore it is important that we, as designers of these systems, think about the norms and values appropriate to the domain when designing and evaluating these recommender systems, such that they have a positive impact on society. For example, should news recommender systems ensure that different viewpoints are presented to its users? Or, how should a travel recommender system consider sustainability aspects?

Submission Guidelines

All papers must be original and not simultaneously submitted to another journal or conference. We accept contributions in the form of

• Full papers (7-12 pages excl. references) are expected to report on novel and original technical contributions with strong normative foundations.
• Short papers (4-6 pages excl. references) that discuss exciting new ideas or theories that may not have gone through elaborate experimental validation. 
• Extended abstracts (3 pages excl. references) that discuss important challenges faced when applying normative thinking to the design and evaluation of recommender systems. We especially invite industry participants and participants working in public services to share how normative thinking is currently being applied within their organizations and the challenges they may face.

Authors should submit manuscripts that follow the CEURART style and will be reviewed following a double-blind process. Every paper will be reviewed by at least two reviewers, one with a background in computer science and one with a background in the humanities or social sciences. By having at least one reviewer from each domain, we aim to ensure that papers combine a strong technical contribution with strong normative thinking. 

Only papers using the CEURART template will be considered.

List of Topics

• Normative/Value-Sensitive Algorithm Design
  • How can different norms and values be operationalized?
  • How can we design algorithms that optimize for these norms and values?
  • How can we balance these multiple objectives and multiple stakeholders?
• Metrics & Evaluation Methods
  • How should norms and values be measured? 
  • What data representations are rquired to measure a norm or value ?
  • How do metrics for norms and values behave in different domains? Do they generalize? 
  • How should we design experiments that measure norms and values? 
• Datasets
  • How can norms and values be operationalized in available public datasets?
  • What is the influence of the data representation on the metric? 
  • New datasets that contain annotations and metadata that can be used to compute metrics of norms and values.
• Case Studies
  • What norms and values are of interest in practice?
  • What issues do you face when operationalizing norms and values? 
  • Empirical studies of how recommender systems behave with regards to norms and values. 
• Philosophical & Conceptual Work
  • Which norms and values are of interest in a specific domain and why?
  • How should these norms and values be balanced?

Committees

Program Committee

• Dietmar Jannach, University of Klagenfurt
• Toine Bogers, IT University of Copenhagen
• Antske Fokkens, VU Amsterdam
• Myrthe Reuver, VU Amsterdam
• Jes Frellsen, Technical University of Denmark
• Yashar Deldjoo, Polytechnic University of Bari
• Hanna Hauptmann, Utrecht University
• Andres Ferraro, Pandora
• Meike Zehlike, Zalando
• Olivier Jeunen, ShareChat
• Savvina Daniil, CWI
• Lucien Heitz, University of Zurich
• Nick Diakopoulos, Northwestern University
• Kasper Lindskow, Ekstra Bladet
• Judith Möller, Hans Bredow Institut
• Rupert Kiddle, University of Amsterdam
• Damian Trilling, University of Amsterdam
• Laura Jansen, Wageningen University
• Annelien Smets, imec-SMIT VUB
• Sophie Morosoli, University of Amsterdam
• Hannes Cools, University of Amsterdam
• Valeria Resendez, University of Amsterdam
• Tomas Dodds Rojas, Leiden University
• Erik Knudsen, University of Bergen
• Mathias Felipe, University of Amsterdam
• Natali Helberger, University of Amsterdam
• Marijn Sax, University of Amsterdam
• Max van Drunen, University of Amsterdam
• Laurens Naudts, University of Amsterdam
• Naomi Appelman, University of Amsterdam

Organizing committee

• Sanne Vrijenhoek, University of Amsterdam
• Lien Michiels, University of Antwerp
• Johannes Kruse, Ekstra Bladet
• Alain Starke, University of Amsterdam
• Nava Tintarev, Maastricht University

Publication

NORMalize 2023 proceedings will be published as CEUR Workshop Proceedings for easy access and reference.

Contact

All questions about submissions should be emailed to normalizeworkshop@gmail.com.