CFP

ntcir19proposal: NTCIR-19 Task Proposal

National Institute of Informatics (NII)

Tokyo, Japan, December 14-17, 2026

Conference website	https://research.nii.ac.jp/ntcir/index-en.html
Submission link	https://easychair.org/conferences/?conf=ntcir19proposal
Task proposal submission deadline	March 31, 2025

NTCIR (NII Testbeds and Community for Information Access Research) is a series of evaluation conferences that mainly focus on information access with East Asian languages and English. The first NTCIR conference (NTCIR-1) took place in August/September 1999; the latest NTCIR-18 conference will be held in June 2025. Research teams from all over the world participate in one or more NTCIR tasks to advance the state of the art and to learn from one another's experiences.

It is time to call for task proposals for the next NTCIR (NTCIR-19), which will start in September 2025 and conclude in December 2026. The NTCIR Program Committee will review task proposals, and organizers of accepted tasks will have a chance to present their proposed tasks at the NTCIR-18 Conference held in NII, Tokyo, Japan, from June 10 to 13, 2025.

Submission link: https://easychair.org/conferences/?conf=ntcir19proposal

Submission Guidelines

We invite new task proposals within the expansive field of information access. Organizing an evaluation task entails pinpointing significant research challenges, strategically addressing them through collaboration with fellow researchers (including co-organizers and participants), developing the requisite evaluation framework to propel advancements in the state of the art, and generating a meaningful impact on both the research community and future developments. Prospective applicants are urged to underscore the real-world applicability of their proposed tasks by utilizing authentic data, focusing on practical tasks, and solving tangible problems. Additionally, they should confront challenges in evaluating information access technology, such as the extensive number of assessments needed for evaluation, ensuring privacy while using proprietary data, and conducting live tests with actual users.

In the era of large language models (LLMs), these models are anticipated to influence daily human activities significantly. Nonetheless, the content produced by LLMs often exhibits issues, such as hallucinations. NTCIR-19 encourages tasks that focus on quality evaluation of content generated by LLMs continued from NTCIR-18 as well as information access exploiting LLMs including generative information retrieval (IR), IR using generative queries, conversational search using generated utterances, evaluation using LLM (relevance judgements or language annotation using LLM), RAG, etc.

IMPORTANT DATES

March 31, 2025: Task Proposal Submission Due (Anywhere on Earth)
May 15, 2025: Acceptance Notification of Task Proposals
June 10-13, 2025: NTCIR-18 Conference (Organizers of accepted tasks have a chance to present their proposed tasks)

NTCIR-19 TENTATIVE SCHEDULE

January 2026: Dataset release*
January-June 2026: Dry run*
March-July 2026: Formal run*
August 1, 2026: Evaluation results return
August 1, 2026: Task overview release (draft)
September 1, 2026: Submission due of participant papers (draft)
November 1, 2026: Camera-ready participant paper due
December 2026: NTCIR-19 Conference at NII, Tokyo, Japan

(* indicates that the schedule can be different for different tasks)

PROPOSAL TYPES

We will accept two types of task proposals:

Proposal of a Core task: This is for fostering research on a particular information access problem by providing researchers with a common ground for evaluation. New test collections and evaluation methods may be developed by collaborating with task organizers (proposers) and task participants. At NTCIR-18, the core tasks are AEOLLM, FairWeb-2, FinArg-2, Lifelog-6, MedNLP-CHAT, RadNLP, and Transfer-2. Details can be found at http://research.nii.ac.jp/ntcir/NTCIR-18/tasks.html.
Proposal of a Pilot task: This is recommended for organizers who propose to focus on a novel information access problem, and there are uncertainties either in task design or organization. It may focus on a sub-problem of an information access problem and attract a smaller group of participating teams than core tasks. However, it may become a core challenging task in the next round of NTCIR. At NTCIR-18, the pilot tasks are HIDDEN-RAD, SUSHI, and U4. Details can be found at http://research.nii.ac.jp/ntcir/NTCIR-18/tasks.html.

Organizers are expected to run their tasks mainly with their own funding and to make the task as self-sustaining as possible. A part of the fund can be supported by NTCIR, which is called "seed funding." It is usually used for limited purposes, such as hiring relevant assessors. The amount of seed funding allocated to each task varies depending on requirements and the total number of accepted tasks. Typical cases would be around 1M JPY for a core task and around 0.5M JPY for a pilot task (note that the amount is subject to change).

Please submit your task proposal as a PDF file via EasyChair by March 31, 2025 (Anywhere on Earth). https://easychair.org/conferences/?conf=ntcir19proposal

TASK PROPOSAL FORMAT

The proposal should not exceed four pages in A4 single-column format. The first three pages should contain the main part and appendix, and the last page should only describe the data to be used in the task. Please describe the data in as much detail as possible so that we can help your data release process after the proposal is accepted. In the past NTCIRs, it took much time to create memorandums for data release, which sometimes slowed down the task organization.

Main part

Task name and short name
Task type (core or pilot)
Abstract
Motivation
Methodology
Expected results

Appendix

Names and contact information of the organizers
Prospective participants
Data to be used and/or constructed
Budget planning
Schedule
Other notes

Data (to be used in your task) - Details

(Please describe the details of the data, which should include the source of the data, methods to collect the data, range of the data, etc.)

License (Please make sure that you have a license to distribute the data, and details of the license should be provided. If you do not have permission to release the data yet, please describe your plan to get the permission.)
Distribution (Please describe how you plan to distribute the data to participants. There are three choices: distributed by the data provider, by organizers, and by NII.)
Legal / Ethical issues

(If the data can cause legal or ethical problems, please describe how you propose to address them. For example, some medical data may need approval from an ethical committee. e.g., some Web data may need filtering to exclude discriminative messages.)

If you want NII to distribute your data to task participants on your behalf, please send an email to ntc-admin@nii.ac.jp before your task proposal submission, attaching the task proposal.

NTCIR-19 PROGRAM CO-CHAIRS

Qingyao Ai (Tsinghua University, China)
Chung-Chi Chen (National Institute of Advanced Industrial Science and Technology (AIST), Japan)
Shoko Wakamiya (Nara Institute of Science and Technology (NAIST), Japan)

NTCIR-19 GENERAL CHAIRS

Charles Clarke (University of Waterloo, Canada)
Noriko Kando (National Institute of Informatics, Japan)
Makoto P. Kato (University of Tsukuba, Japan)
Yiqun Liu (Tsinghua University, China)

Contact

All questions about submissions should be emailed to ntc-secretariat@nii.ac.jp