‹
Back
15:15 - 16:00
Parallel sessions 5
›
15:15 - 16:00
Location: Room 247
The last presenter of the session is kindly asked to take over the moderation of the session helping to keep the time.
Each presenter is invited to use up to 15 min for presentation and up to 5 min for Q&A.
Submission 209
A Hierarchy of Inferability: The DIRECT Framework for Defining Human Involvement in AI-Assisted Identification and Rating of Perceptual Indicators in Open Teacher Discourse
Presented by: Adi Yaakov Azaria
Adi Yaakov Azaria 1, Merav Rotary Saban 1, Anat Cohen 1, Guy Cohen 1, Alla Alla Bronshtein 2
1 School of Education, Tel Aviv University, Israel
2 Faculty of Medical & Health Sciences, Tel Aviv University, Israel

This study examines the conditions under which generative AI (GenAI) can support the identification and rating of perceptual indicators related to technology adoption in open-ended teacher discourse. Grounded in the Technology Acceptance Model (TAM), the study conceptualizes three levels of inferential demand: direct-evidence, synthesized-evidence, and TAM construct interpretation. Data from 174 K-12 teachers were analyzed through a human-AI comparative design using weighted Cohen’s kappa, before and after a structured training process. Findings indicate that agreement between GenAI and human raters varies systematically with inferential demand, with stronger alignment for direct-evidence indicators and more limited reliability & validity for higher-level, theory-driven constructs. Training improved alignment primarily for lower- and mid-level variables, while highlighting the role of “Insufficient Information” as a meaningful analytical outcome. The study introduces the DIRECT framework, which links inferential demand to the distribution of human-AI roles, offering a conceptual and practical guide for integrating GenAI into identification and rating of perceptual indicators.