16:30 - 17:45
Parallel sessions 12
Submission 232
An Experience with an Educational Chatbot
Presented by: Rutinelli Favero
Rutinelli FaveroElton Vinicius Silva
Instituto Federal do Espírito Santo

The SophIA chatbot was created to assist Educational Design students (in the Foundations of Educational Design course), offering immediate support regarding content, Moodle navigation, and activities (Winkler & Söllner, 2018). It also aimed to provide a practical experience for the students, as professional practice requires familiarity with automation and technological curation (Filatro & Cavalcanti, 2017; Filatro, 2020). Its iterative and pedagogical development included data feeding, behavioral configuration (a five-dimensional prompt), and visual design, utilizing the no-code platform Chatbase (Chatbase, 2024) and the GPT-4o mini model (OpenAI, 2024). Based on Retrieval-Augmented Generation (RAG) to mitigate "hallucinations" (Lewis et al., 2020), its knowledge base integrated virtual classroom content, comprehensive Moodle documentation, and frequently asked questions (Q&A). The image of SophIA, conceptualized as a mature and welcoming 52-year-old woman, was generated via DALL-E (OpenAI, 2024). During the period from August 1 to December 31, 2024, SophIA recorded significant student engagement, with the highest concentration of access occurring in the afternoon (47%) and on Mondays. 53 real conversations were recorded, totaling 160 messages sent by the students. The peak of interactions occurred in August (64%), coinciding with the beginning of the course. Student demands were diversified, highlighting pedagogical organization: activities and deadlines (30%), academic content (25%), and technical use of Moodle (10%). The qualitative analysis revealed the effectiveness of the guardrails (safety guidelines) established in the prompt. In a specific interaction, a user indicated that they would provide their CPF to the platform; SophIA promptly interrupted the action. Other interactions included: tests conducted by us containing provocations about the teachers' conduct, in which the AI maintained a professional tone; and the use of the tutor in real-time during live lectures, responding as programmed in her personality. The implementation of SophIA demonstrated that the use of chatbots based on RAG architecture can be effective for providing faster assistance to students with specific questions. The technical accuracy, reflected in the confidence score of 0.82, validated the strategy of feeding the agent with hybrid sources (disciplinary content and Moodle technical documentation); however, the current methodology demands that the database composition be done non-automatically. Furthermore, the chatbot's ability to protect sensitive data, respond with some sense of humor, and maintain neutrality in provocative interactions proves that the success of an AI tutor in education does not reside solely in the language model used, but in the rigor of the Conversational Chatbot and the definition of behavioral guardrails. However, the experience revealed an important technical boundary: the disconnection between the chatbot and the dynamic data of Moodle. The inability of SophIA to access attendance reports or forum summaries in real time points to the need for future API integrations, allowing the virtual assistant to evolve from a static query repository to an active monitoring agent. It is necessary to consider the financial issues for this, as well as those related to the protection of sensitive data. The use of no-code platforms for creating intelligent tutors democratizes the access of teachers and educational designers to advanced technologies.