Registration and Leaderboard

Registration

1.You can register in this link: Registration Link.

2.Max number of participants in a team: 4 (including tutor/supervisor). An individual cannot be part of multiple teams.

3.After submission, you will receive an email with detailed instructions for the development set and submission details.

4.The test set will be released when the leaderboard is open for submission.

Benchmark and Evaluation Metrics

This shared task evaluates LLMs using three value-based datasets, each split into 20% development and 80% test sets:

• Chinese (4,000 instances): Multiple-choice daily scenarios testing 158 universal values against a single gold answer.
• Indonesian (1,834 instances): Multiple-choice news-based scenarios testing five national values based on Pancasila, scored against human consensus.
• Sri Lankan (1,000 instances in Sinhala): Evaluates two statements (labeled A, B, Both, or None) grounded in 40 survey-derived societal values.

Submission format, delivery, and scoring details:

Each team must submit one prediction jsonl file covering all hidden test instances across all three countries. The format of the output:

{
  “dataset”: “<dataset_name>”
  “id”: “<question_id>”,
  “LLM_Output”: “<exact_answer>”
}

The required LLM_Output would be:

• China: one label from {A, B, C, D} for each four-option multiple-choice item.
• Indonesia: one label from {A, B, C, D} for each four-option multiple-choice item.
• Sri Lanka: one label from {A, B, Both, 0} for each item.

The prediction json file can be submitted via leaderboard website: Track-1 Resource-Constrained Track Track-2 Open Track. The file must be saved as “predictions.jsonl” and zipped. Submissions that do not follow the required output format may be treated as invalid for the affected instances.

Leaderboard

Track 1: Resource-Constrained Track

Live leaderboard: https://www.codabench.org/competitions/16791/

Track 2: Open Track

Live Leaderboard: https://www.codabench.org/competitions/16818/