15:15 - 16:00
Parallel sessions 5
Submission 209
A Hierarchy of Inferability: The DIRECT Framework for Defining Human Involvement in AI-Assisted Identification and Rating of Perceptual Indicators in Open Teacher Discourse
Presented by: Adi Yaakov Azaria
Adi Yaakov Azaria 1, Merav Rotary Saban 1, Anat Cohen 1, Guy Cohen 1, Alla Alla Bronshtein 2
1 School of Education, Tel Aviv University, Israel
2 Faculty of Medical & Health Sciences, Tel Aviv University, Israel

This study examines the conditions under which generative AI (GenAI) can support the identification and rating of perceptual indicators related to technology adoption in open-ended teacher discourse. Grounded in the Technology Acceptance Model (TAM), the study conceptualizes three levels of inferential demand: direct-evidence, synthesized-evidence, and TAM construct interpretation. Data from 174 K-12 teachers were analyzed through a human-AI comparative design using weighted Cohen’s kappa, before and after a structured training process. Findings indicate that agreement between GenAI and human raters varies systematically with inferential demand, with stronger alignment for direct-evidence indicators and more limited reliability & validity for higher-level, theory-driven constructs. Training improved alignment primarily for lower- and mid-level variables, while highlighting the role of “Insufficient Information” as a meaningful analytical outcome. The study introduces the DIRECT framework, which links inferential demand to the distribution of human-AI roles, offering a conceptual and practical guide for integrating GenAI into identification and rating of perceptual indicators.