Yazarlar
Yayınlar
- MakaleAçık Erişim2026Scopus SJRQ2A New Metacognitive Confidence Calculation Approach Based on Large Language Models for Natural Language Explanations
Yilmazer Gundem, Merve; Sahin, Simay; Karakose, Mehmet
Traditional methods used to measure metacognitive confidence rely on superficial features such as word count and ambiguity and therefore fail to adequately reflect the level of metacognitive confidence in statements. In this study, in addition to traditional text-based confidence measures, we used a Large Language Model (LLM) to directly analyze confidence and its ratio in statements. LLM-based confidence scores were calculated on statements from the e-SNLI and CoS-E open-source datasets, and the prediction performance obtained with traditional features was compared. The results indicate that traditional surface-level text features explain the variance in LLM-generated confidence scores to a limited extent, and that LLM-based scores are significantly correlated with independent researcher evaluations. This study contributes to the literature by offering a scalable and reproducible analytical framework for measuring metacognitive confidence. It is believed that the findings can be directly applied, particularly in applications where trust is critical, such as automated software test scenario generation, decision support systems, and risk management. The use of LLMs offers a framework for the LLM-based analysis of potential metacognitive confidence signals in natural language explanations by examining the sense of confidence and certainty with which statements are presented, rather than their factual accuracy.
- BildiriAçık Erişim2025LLM-Based Video Analytics Test Scenario Generation in Smart Cities
Yilmazer, Merve; Karakose, Mehmet
Rapid advances in the field of artificial intelligence have made significant contributions to the automation of software development and testing stages. Software created for use in various fields is tested with test scenarios created manually by software test experts or using test automation. Testing large-scale software with these methods complicates the testing phases because it requires increased human intervention and includes complex applications. In this study, an LLM-based scenario generation framework enhanced with prompt engineering is proposed for testing software to be used for video analysis in smart cities and smart campus areas. Thus, software test scenarios are created by strengthening large language models that are fast, flexible and have high learning ability using prompt engineering techniques. Test scenarios produced through LLM reinforced with prompt engineering techniques were evaluated with rarity and reality metrics and it was determined that more robust scenarios were produced compared to randomly generated test scenarios in the relevant field.
AtıfScopus3OpenAlex3
İncele
