Shared Task

Shared Task on Pluralistic Value Alignment Across China, Indonesia, and Sri Lanka

Introduction

As Large Language Models (LLMs) increasingly serve as global digital infrastructure, the problem of “value alignment” has become central to AI research. However, most existing alignment benchmarks either assume a single dominant value framework or evaluate LLMs within only one cultural context. This shared task focuses instead on pluralistic value alignment: the ability of a single system to adapt its responses to different, locally grounded value contexts rather than defaulting to one universal normative style.

The shared task asks participants to build one unified system that, given a scenario and a target country, produces the response or judgment that is most consistent with the human value preferences reflected in that country’s annotated data. The benchmark integrates three country-specific datasets while preserving their native task formats: Chinese and Indonesian value datasets are formulated as four-option multiple-choice question answering, and Sri Lanka’s dataset as a four-way decision task in which systems must choose A, B, Both, or None. The task is unified at the level of objective and evaluation by standardizing all datasets into a single answer format.

To support method development while preserving a strong generalization test, the benchmark is split into a development set that will be publicly available and an evaluation set. The public portion will be released with labels and may be used for prompt design, supervised fine-tuning (SFT), agent construction, retrieval design, or internal validation. The remaining 80% will be held out for official evaluation. Participants may use country-aware prompting, routing, or adaptation strategies internally, but they must submit one system and one prediction file covering all countries.

Importantly, the shared task does not assume that any country can be reduced to a single fixed or homogeneous value system. Rather, it evaluates how well LLMs can match the plural and situated human judgments represented in the benchmark data. To reflect this design principle, official ranking emphasizes balanced performance across countries, so that strong results on one country cannot compensate for weak results on another.

Challenge Goals

This shared task has three main goals:

• Evaluate pluralistic value alignment across cultural contexts.
  The challenge measures whether LLM-based systems can adapt to locally grounded human value preferences across China, Indonesia, and Sri Lanka, rather than relying on a single generic alignment strategy.
• Encourage unified and generalizable solutions.
  Participants must develop a system that works across multiple countries and task formats, promoting methods that generalize across settings instead of isolated, single-country solutions.
• Support broad and meaningful comparison of methods.
  The challenge is open to diverse approaches, including prompting, fine-tuning, retrieval, and agent-based methods, and is designed to compare both resource-efficient and unrestricted systems while rewarding balanced cross-country performance.

Challenge Tracks

To have a better performance towards pluralistic values, participants will be asked to improve LLMs, allowing LLMs to have better understanding towards diverse countries' values. We focus on China, Indonesia, and Sri Lanka values. The shared task includes two tracks: Resource-Constrained Track and Open Track. Both tracks use the same task definition, the same public 20% development data, and the same hidden 80% evaluation data. They differ only in the resources participants are allowed to use.

Benchmark and Evaluation Protocol

Benchmark

This challenge will be evaluated using 3 datasets:

1. Chinese dataset

This dataset is built from Chinese daily value dilemmas, where the values are based on 158 fine-grained universal value concepts. The dataset contains 4,000 instances, split into a 20% development set and an 80% test set in Chinese. Each item contains a “scenario” and “four options” (A–D). The scenarios reflect obvious value conflicts from a first-person perspective, highlighting the internal struggles (such as emotional bonds) and external obstacles (such as institutional constraints and economic pressures) that roles face when practicing positive values. The options cover the entire set of potential actions for each character. These actions are not limited to moral considerations (including realistic choices such as compromise and retreat) and incorporate real-world considerations (such as the risk of retaliation and cost constraints). LLMs must predict one of the most proper choices and compare with one gold answer for one question.

Example (Original in Chinese):

2. Indonesian dataset

Several nations already define their own values officially by the government. Pancasila as the ideology foundation of Indonesia consists of 5 values: religion, humanity, unity, democracy, and social values. This benchmark consists of Indonesian multiple-choice dilemmas derived from news articles and grounded in the five Pancasila value groups: Religion, Humanity, Unity, Democracy, and Social Justice. The dataset contains 1,834 instances, split into a 20% development set and an 80% test set in Indonesian. The format of the question: “scenario”, “question”, and “four options (A–D)”. LLMs must predict a single preferred option. For each question, we collected responses from two human annotators. In some cases, the annotators agree on a single answer (consensus), while in others, they select different answers (divergence). An LLM's output is considered correct if it matches at least one of the answers preferred by the human annotators.

Example (original in Indonesian):

3. Sri Lanka’s dataset

LKvaluesBench is composed of Sinhala instances (a mix of human-curated and human-verified AI generations) sourced from Sinhala_MMLU dataset. The dataset contains 1,000 instances, split into a 20% development set and an 80% test set in Sinhala. These datasets are tagged with societal values based on a large-scale survey of Sri Lanka value identification. Each instance presents two statements and asks the LLMs to predict one of four labels: A, B, BOTH, or 0.

Example:

Submission Format

Each team must submit one prediction file covering all hidden test instances across all three countries. The format of the output:

{
“dataset”: “<dataset_name>”
“id”: “<question_id>”,
“LLM_Output”: “<exact_answer>”
}

The required LLM_Output would be:

• China: one label from {A, B, C, D} for each four-option multiple-choice item.
• Indonesia: one label from {A, B, C, D} for each four-option multiple-choice item.
• Sri Lanka: one label from {A, B, Both, None} for each item.

Submissions that do not follow the required output format may be treated as invalid for the affected instances.

Evaluation Metric

Submissions will be evaluated primarily on “Accuracy”. An output is scored as 1 if correct and 0 if incorrect. LLMs will first receive an accuracy score for each of the three individual datasets. The final ranking score will be the “macro-average” of these 3 dataset-level scores.

Leaderboard

• China (4 options with a gold answer)
  “1” if same with gold answer, “0” if different
• Indonesia (4 options with one or two gold answers)
  “1” if same with gold answer, “0” if different
• Sri Lanka (2 options, gold answer can be one of those two, agree with both, or disagree with both)
  “1” if same with gold answer, “0” if different

Timeline

• April 20, 2026 — Registration opens
• May 20, 2026 — Development sets released
• September 1, 2026 — Test sets released
• September 15, 2026 — Results submission deadline
• September 22, 2026 — Evaluation ends
• September 29, 2026 — Paper submission deadline
• October 13, 2026 — Paper acceptance notification
• October 22, 2026 — Camera-ready deadline
• November 6, 2026 — Conference