Submission 122
Algorithmic Critique: Prompt-sensitive analysis of how LLMs deploy theories
SP01-02
Presented by: Donghoon Jung
This study examines LLMs' capabilities and inclinations as literary critics. It will address questions such as how AI-generated literary criticism is different from human practice, whether they have preferred schools of critical theory, and how their built-in critical biases shape their claims. We define prompt design as specifying the interpretive rules for evidence, warrants, and permissible claims, and we foreground output sensitivity to prompt variables as a central object of inquiry rather than a mere implementation detail. This approach offers a test case for how generative AI models operationalize interpretation and reexamines how the field understands critical authority, originality of perspective, and interpretive plurality. By comparing a corpus of human criticism with LLM-generated texts and combining distant and close reading, the study measures patterns, perspectives, and interpretations of AI models. It then proposes prompt-design guidelines based on our analysis and maps the possibilities and limits of AI criticism to suggest how literary criticism may be repositioned as AI tools enter the field.LLMs are capable of producing prose that constitutes literary criticism. They are known for surface-level polish and coherent structure, but still struggle with layered semantic work, while also showing a tendency to privilege Western theory and to reproduce disciplinary commonplaces such as recurring arguments and canonical citation chains. Yet we lack concrete evaluations of how these abilities operate and where their limits lie. To address this, we combine distant reading of macropatterns with close reading of textual detail.The research proceeds along four axes. First, it examines whether LLM criticism shows regularities in narrative mode, lexical choice, and theoretical reference that distinguish it from human criticism, thereby identifying an AI-specific interpretive frame. Second, it assesses under- or over-representation biases in the representation of feminism, queer theory, postcolonial and decolonial studies, critical race theory, ecocriticism, and disability criticism, and analyzes how issues of gender, sexuality, race, class, disability, and the environment are represented in the process. Third, it systematically varies prompt instructions for the same work to evaluate the consistency of claims and conclusions and to track shifts in tone and focus. Fourth, drawing on these results, it proposes prompt-design guidelines and tests the potential for complementary integration between human and AI criticism.The methodology follows a sequential pipeline: data construction, AI-critique generation, distant analysis, and close reading. We fix the corpus at five English-language short stories from 1900 to 1922 (the modernist period featuring diverse style, with copyright clearance as an added merit), pairing two canonized works with three less mainstream Anglophone stories to diversify authorial and cultural vantage points: James Joyce, “The Dead” (1914); Katherine Mansfield, “The Garden Party” (1922); Sui Sin Far (Edith Maude Eaton), “Mrs. Spring Fragrance” (1912); Pauline E. Hopkins, “Talma Gordon” (1900); and Zitkala-Ša (Gertrude Bonnin), “The Soft-Hearted Sioux” (1901). For each work, the study collects journal articles, chapters from scholarly books, and single-authored monographs and book-length criticism. The theoretical lenses applied in each document are tagged as metadata to construct work × lens subcorpora. The target is at least thirty items per work, for a total of 150.We define prompt design as specifying the interpretive rules for evidence, warrants, and permissible claims. Because AI-produced criticism already circulates in scholarly discourse, output sensitivity to prompt variables is a central object of inquiry rather than an implementation detail. We therefore generate approximately 1,000-word AI critiques via API calls to snapshots of four models (claude-opus-4, gemini-2.5-pro, qwen-max, gpt-5). Prompts are implemented under three conditions that vary the degree and kind of interpretive constraint. In the basic condition, the model critiques the work as a whole without a specified theoretical lens and selects topics, structure, and citation strategy. In the direct-instruction condition, a theoretical lens is fixed as the analytic framework, within which topic selection, argumentative structure, and citation strategy proceed. In the implication-guided condition, we specify an analytic focus and an object of observation, omit explicit lens naming, require textual quotations as primary evidence, and require that concepts and categories be derived from that evidence. This design tests the stability, drift, and bias of interpretations across conditions and treats prompt sensitivity as a scholarly concern in its own right.In the distant reading stage, we compare human- and AI-criticism corpora to quantify biases in patterns and perspectives. For patterns, we cluster criticism texts using stylometry with most-frequent-words, extract seventeen linguistic features for identifying primary stylistic features, and use principal component analysis to visualize stylistic distribution and overlap. For perspectives, TF–IDF quantifies lexical overuse and avoidance and derives differential keywords, GPT-5 topic extraction generates document-level topics that we compare across work × lens combinations, and embeddings computed with text-embedding-3-large measure semantic distance and similarity among criticism texts. Together, these procedures translate stylistic and topical regularities into testable quantities and make biases in patterns and perspectives measurable.In the close reading stage, we qualitatively examine how argumentation operates within the texts. For each work × lens × prompt condition, we excerpt representative and contrastive cases, and annotate the propositions, evidence, and the development of interpretation at the paragraph level. We trace the role of quotation selection and placement in supporting claims and where and how theoretical concepts are invoked. The basic, direct-instruction, and implication-guided modes are contrasted under consistent criteria to assess lens appropriateness, the suppression or recurrence of summary regression, the pathways of concept derivation, and shifts in tone and focus.By combining the results of distant and close reading, the study offers condition-specific prompt guidelines. For each prompt mode, we offer systematized lens specification, principles for constructing claims, quotation-level requirements, summary-limitation rules, and guidelines for articulating the connection between evidence and claims. Collectively, these materials aim to increase understanding of how LLMs behave as critics and to support more informed choices about their use in research, teaching, and publishing. Verifying the necessity of human criticism and assessing both the value and limitations of AI-human collaboration, this study contributes to redefining the role and orientation of literary criticism as AI reshapes the environment of scholarly knowledge production.