What undergraduate students need to know and actually know about generative AI

In November 2022, the release of ChatGPT sparked widespread adoption of generative AI chatbots among students, yet little is known about what undergraduate students actually understand about these tools or how accurately they perceive their capabilities. To address this gap, we propose a theoretically grounded framework for Generative AI (GenAI) literacy that integrates three forms of conceptual knowledge about large language models (LLMs)—foundations, capabilities, and limitations, and societal impact—with students' perceptions and folk theories of AI chatbots. We then developed and validated a GenAI literacy survey, including multiple-choice knowledge items and perception items, using expert review and item response theory (IRT) modeling. To examine GenAI literacy across different educational contexts, we conducted two complementary studies: Study 1 surveyed students enrolled in courses at a large public R1 university, and Study 2 surveyed a national sample of U.S. undergraduate students recruited via Prolific. Across both samples, approximately 60% of students reported using AI chatbots weekly or daily. However, many students overestimated chatbots’ capabilities, particularly on tasks involving reasoning and counting, and often anthropomorphized or treated chatbots as search engines. Knowledge scores were higher among computer science students at the R1 university and frequent chatbot users, while perception accuracy varied by group. Critically, greater conceptual knowledge was associated with less overestimation of chatbot abilities, suggesting that knowledge supports more calibrated and responsible use. This work introduces a validated framework and instrument for assessing GenAI literacy and highlights the need for AI literacy initiatives that move beyond tool usage to address misconceptions, beliefs, and responsible engagement with generative AI in higher education.