In beef soup noodles, can you find the beef?
Anyone up on Taiwanese cuisine would know that the correct answer is no. Beef soup noodles (not to be confused with beef noodle soup) do not contain chunks of meat.
Several years ago, ChatGPT did not know the answer to this question, said Ling-Jyh Chen (陳伶志), the chief executive of Taiwan AI Academy. Earlier versions of the AI chatbot were trained on 190 million Chinese words, with the vast majority of Chinese-language training data being in simplified Chinese and originating from China.
Chen said Chinese-language proficiency doesn’t translate to an awareness of the local context, and large language models built by U.S. companies, like Google and OpenAI, often fail to understand Taiwanese culture. “We need to keep our culture. This is important for our government and also important for us to teach our next generation. We want to keep our value system.”
Making sure that AI chatbots reflect “Taiwanese values” is existential for Taiwan’s leaders. In a statement published last month, Taiwan’s Ministry of Digital Affairs made an implicit reference to China’s use of AI to propagate its worldview, writing in a blog post that authoritarian regimes use AI to rewrite the past and “dominate the future development of human society.”
“Taiwan must take control of its own data, language, and perspectives, and ensure that the views of Taiwan, the U.S., and other democratic countries can be correctly understood in the AI era,” the ministry said.
As Taiwan’s efforts to build its own models and body of Taiwanese training data take shape, the question is what “sovereign AI” means for Taiwan. In September, the AI Evaluation Center, which operates under the Ministry of Digital Affairs, began ranking AI models according to their aggregate score across language proficiency, understanding of Taiwanese society and culture, and understanding of Taiwanese values. The purpose is to understand to what extent an AI model truly understands Taiwan.
The first two metrics are straightforward, as they are based on the models’ ability to answer questions that appear on the GSAT, Taiwan’s university entrance exam. But Chen thinks they aren’t rigorous enough. The entrance exam questions are public, which means that a company could theoretically use them to train their model and achieve a perfect score during evaluation.
Taiwanese startup APMIC jumped from 15th place to 1st place in the AI Evaluation Center’s rankings between September and October due to a nearly 17 percentage point increase in its averaged GSAT score. APMIC’s model did not surpass U.S. models because it became more powerful but because it was fine-tuned based on data provided publicly by the Taiwanese government.
On these first two metrics, U.S. AI models have consistently performed very well. ChatGPT is now even able to answer the question about beef soup noodles correctly. In fact, if you look at the AI Evaluation Center’s September results, wherever Taiwanese models rank higher than the major U.S. models, it is because of their success on the third metric: “Taiwan-specific values.” Recalculating the rankings without this metric causes the Taiwanese models to fall dramatically.
The AI Evaluation Center is vague about how exactly it measures Taiwanese values. In an email to Domino Theory, the center said it measures an AI model’s understanding of Taiwan’s historical experience and democratic institutions, including issues on which there is a degree of consensus in Taiwanese society. “On some issues, perspectives in Taiwan may differ from those commonly reflected in narratives or discourse in China,” the center wrote, adding that “such differences are not, in themselves, the benchmark.”
But the evaluation is not designed to prescribe a set of politically correct answers, the center said. “The question is not whether an AI ‘agrees with Taiwan,’ but whether it can genuinely understand Taiwan.”
Alice Yang (楊長蓉), an assistant research fellow at the Institute for National Defense and Security Research, questioned whether a specifically Taiwanese value can even be defined objectively. She thinks the government is mainly concerned about preventing Chinese influence over Taiwanese thought, particularly among younger generations.
“If you ask something more sensitive, maybe [the model] will give you more Chinese-based answers. That’s what the government really doesn’t want,” Yang said.
The truth is that Taiwan is home to 20 national languages, 16 officially recognized Indigenous peoples and 42 language variations. “So ‘Taiwanese values’ was never going to be one voice,” wrote Audrey Tang (唐鳳), an ambassador at large for Taiwan and the former and first minister of digital affairs, in an email.
Tang thinks sovereign AI should mostly be implemented at the local level, where decisions about how the model is governed belong to “the smallest community that can carry them.” By this measure, a model based on a foreign open-weight model “can be sovereign in every way that matters,” Tang said.
“If sovereign AI means the state gathers everyone’s data to train one national model, I think that is fundamentally misguided,” Tang said. “A model is not trustworthy for being Taiwanese, any more than for being American or Chinese.”
Leave a Reply