The integration of early warning dissemination and public response is central to effective risk governance under extreme disasters. However, empirical evidence consistently shows that the issuance of warnings does not necessarily translate into timely public action. This gap reflects the lack of a valid, scalable, real-time measurement method for public risk and action awareness following the release of disaster warnings, leaving the warning institutes without timely insight into how the public understands risk or prepares to act. Addressing this challenge, this study focuses on developing a large-language-model-based (LLM-based) measurement method to assess and analyze real-time public response based on social media data. Specifically, the study developed a human-in-the-loop framework for integrating the theoretical construction of risk perception and protective action intentions into the LLM-based processing pipeline. First, we adopted different fine-tuned strategies including few-shots, chain-of-thoughts, and instruction prompt-based fine-tuned and parameter-based fine-tuned models to increase measurement stability. Second, a workflow was designed to address measurement uncertainty by generating multiple independent assessments through parallel model inference, and by systematically integrating outputs through consistency checks, result aggregation, and targeted validation. Third, we increase the content validity of the measurement by integrating human-validation in the framework. This process transforms a black-box LLM judgments into valid measurement for real-time and large-scale qualitative analysis. Experimental data were drawn from a case study in China, which include Sina Weibo posts during a super Typhoon Doksuri in 2023, to systematically evaluate different model configurations and analytical workflows. The findings identify a robust and valid measurement method for capturing public response upon the release of a disaster early warning. By doing so, the study provides an operational measurement tool for understanding the disconnection between warning dissemination and public response, and offers a transferable methodological reference for disaster governance research.