The Relationship Between Digital Literacy, User Skepticism, and Verification Behavior in Evaluating Generative AI Responses

Abstract

Research Objective. This study is motivated by the transformation of information evaluation practices in the era of Generative Artificial Intelligence (Generative AI), in which users increasingly evaluate AI-generated responses rather than directly assess the credibility of sources. Given that Generative AI can produce highly fluent but not always accurate responses, digital literacy and user skepticism have become critical factors in the information evaluation process. This study aims to analyze the relationships among digital literacy, user skepticism, and verification behavior in the evaluation of Generative AI responses. Research Method. This study employed a quantitative cross-sectional design, using cross-tabulation and chi-square analysis. Analysis Data. Data were analyzed using cross-tabulation and chi-square tests in SPSS to identify distribution patterns and relationships among categorical variables. The study used secondary data from the Kaggle platform, involving 1000 respondents; therefore, the findings should be interpreted within the limitations of the dataset's representativeness. Results. The findings indicate that digital literacy is not significantly associated with age or educational attainment. In contrast, both digital literacy and educational level are significantly associated with users' skepticism toward Generative AI. However, no significant relationship was found between users' digital literacy levels and the fact-checking methods they employed. Conclusions. Within this dataset, age and formal education do not demonstrate significant relationships with digital literacy levels. Nevertheless, users with higher levels of digital literacy and educational attainment tend to exhibit greater skepticism toward Generative AI responses. These findings may indicate a tendency toward more proportionate patterns of trust, although this study does not directly measure calibrated trust. Digital literacy emerges as a potentially important factor associated with how users understand the limitations of AI systems, evaluate AI-generated responses, and determine whether additional verification is necessary.

Keywords: Generative AI Response Evaluation; AI User Skepticism; Digital Literacy