A New Metacognitive Confidence Calculation Approach Based on Large Language Models for Natural Language Explanations

Traditional methods used to measure metacognitive confidence rely on superficial features such as word count and ambiguity and therefore fail to adequately reflect the level of metacognitive confidence in statements. In this study, in addition to traditional text-based confidence measures, we used a Large Language Model (LLM) to directly analyze confidence and its ratio in statements. LLM-based confidence scores were calculated on statements from the e-SNLI and CoS-E open-source datasets, and the prediction performance obtained with traditional features was compared. The results indicate that traditional surface-level text features explain the variance in LLM-generated confidence scores to a limited extent, and that LLM-based scores are significantly correlated with independent researcher evaluations. This study contributes to the literature by offering a scalable and reproducible analytical framework for measuring metacognitive confidence. It is believed that the findings can be directly applied, particularly in applications where trust is critical, such as automated software test scenario generation, decision support systems, and risk management. The use of LLMs offers a framework for the LLM-based analysis of potential metacognitive confidence signals in natural language explanations by examining the sense of confidence and certainty with which statements are presented, rather than their factual accuracy.