Objective To develop a scientific and reliable AI literacy assessment scale tailored to the characteristics of faculty in medical colleges, providing standardized tools and empirical evidence for evaluating AI literacy, optimizing training, and advancing digital transformation of education in this field.

Methods Based on the teacher competency model, constructivist teaching theory, and evidence-based medical education concepts, the scale dimensions and items were developed through literature research, expert consultation (Delphi method), pretesting, and formal testing. Exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) were employed to examine the structural validity of the scale, while reliability was assessed using Cronbach's α and composite reliability (CR). A total of 840 teachers from eight medical colleges at various levels were selected for empirical application to analyze the current status of teachers' AI literacy and its influencing factors.

Results The final AI literacy assessment scale for medical college teachers includes 5 dimensions and 32 items, namely Value Cognition and Ethical Foundation (7 items), Technological Understanding and Tool Application (8 items), Teaching Integration and Innovative Practice (7 items), Learning Assessment and Precise Empowerment (5 items), and Professional Development and Ecological co-construction (5 items). The overall Cronbach's α of the scale was 0.942, with α coefficients for each dimension ranging from 0.835 to 0.907. The CR values were all＞0.85, and the Average Variance Extracted (AVE) values were all＞0.50. Confirmatory factor analysis showed good model fit (χ²/df = 2.319, CFI = 0.958, TLI = 0.952, RMSEA = 0.056, SRMR = 0.048); Empirical application showed that the overall AI literacy of teachers in medical colleges was at a slightly above average level (total score 4.12 ± 0.58), with the highest scores in the dimension of Value Cognition and Ethical Foundation (4.37 ± 0.52), and the lowest scores in the dimension of Technological Understanding and Tool Application (3.89 ± 0.65). The AI literacy of teachers was significantly influenced by gender, years of teaching experience, discipline type, college level, and AI training experience (P ＜ 0.05 or P ＜ 0.001).

Conclusions The developed AI literacy assessment scale for teachers in medical colleges demonstrates good reliability and validity, and can be used as a standardized assessment tool for evaluating AI literacy of teachers in medical colleges. Currently, there is an imbalanced development across different dimensions of teachers’ AI literacy, therefore, it is necessary to optimize targeted training programs based on empirical findings to promote the deep integration of medical education and AI technology.