Yilmazer Gundem, Merve; Sahin, Simay; Karakose, Mehmet

Traditional methods used to measure metacognitive confidence rely on superficial features such as word count and ambiguity and therefore fail to adequately reflect the level of metacognitive confidence in statements. In this study, in addition to traditional text-based confidence measures, we used a Large Language Model (LLM) to directly analyze confidence and its ratio in statements. LLM-based confidence scores were calculated on statements from the e-SNLI and CoS-E open-source datasets, and the prediction performance obtained with traditional features was compared. The results indicate that traditional surface-level text features explain the variance in LLM-generated confidence scores to a limited extent, and that LLM-based scores are significantly correlated with independent researcher evaluations. This study contributes to the literature by offering a scalable and reproducible analytical framework for measuring metacognitive confidence. It is believed that the findings can be directly applied, particularly in applications where trust is critical, such as automated software test scenario generation, decision support systems, and risk management. The use of LLMs offers a framework for the LLM-based analysis of potential metacognitive confidence signals in natural language explanations by examining the sense of confidence and certainty with which statements are presented, rather than their factual accuracy.

İncele

Gundem, Merve Yilmazer; Karakose, Mehmet

Large, user-interactive software systems require extensive software testing for safe deployment. Manually generated test cases by software testers require extensive time and effort, and their scope remains limited. In this study, a system based on LLM is proposed for automatic test cases and test steps generation from bug descriptions. In the proposed system, bug descriptions were first collected from the Defects4J Dataset. Keywords relevant to the bug scope were extracted from the descriptions. Structured automatic prompts were generated from the extracted keywords. The generated prompts were fed into the LLM as input, and software test cases appropriate for the bugs were generated. The method was evaluated on actual bug descriptions by comparing it to a general (baseline) prompt. Experimental results show that the lengths of the generated scenarios are similar, but the proposed method increases the keyword match rate with error descriptions to an average of 89 %. The findings reveal that keyword-driven prompts are effective in generating more reliable and error-specific test scenarios without requiring additional data.

İncele