Submission 15
Can Generative AI Reliably Code Treaty Texts? Evidence from Nuclear Non-Proliferation Treaties
Panel 2-Multi-Function Room 2 (19/F LAU)-02
Presented by: Saera Lee
Can generative AI be used to code international treaties, and how reliable are the results? As generative AI becomes increasingly integrated into both everyday live and academic research, scholars have begun exploring ways to incorporate these tools into their research workflows. This paper develops and evaluates a method for using generative AI to code treaty texts, using non-proliferation and arms control related treaties. We examine how prompting strategies influence coding outcomes and propose approaches that improve performance. We then assess the validity of the generated data by comparing outputs across multiple models and prompts. In addition, we compare coding results obtained through web-based interfaces and API access, providing practical guidance for researchers who are unfamiliar with API-based tools.
The findings suggest that generative AI can offer a cost-effective and time-efficient approach to treaty text data collection. However, expert review remains necessary to ensure the validity and reliability of the final dataset. The study contributes to methodological debates on the use of AI-assisted research tools in political science.
Can generative AI be used to code international treaties, and how reliable are the results? As generative AI becomes increasingly integrated into both everyday live and academic research, scholars have begun exploring ways to incorporate these tools into their research workflows. This paper develops and evaluates a method for using generative AI to code treaty texts, using non-proliferation and arms control related treaties. We examine how prompting strategies influence coding outcomes and propose approaches that improve performance. We then assess the validity of the generated data by comparing outputs across multiple models and prompts. In addition, we compare coding results obtained through web-based interfaces and API access, providing practical guidance for researchers who are unfamiliar with API-based tools.
The findings suggest that generative AI can offer a cost-effective and time-efficient approach to treaty text data collection. However, expert review remains necessary to ensure the validity and reliability of the final dataset. The study contributes to methodological debates on the use of AI-assisted research tools in political science.