Clinical reasoning in nephrology: Comparing GPT-4 with junior fellows

Aim: This study compared the clinical decision-making abilities of junior nephrology fellows against two Generative Pre-trained Transformer 4 (GPT-4)-based artificial intelligence (AI) models—one standard and one specifically trained with nephrology guidelines—using simulated clinical scenarios.

Materials and Methods: Eleven nephrology fellows with two years or less of experience from four academic centers participated in the study. The performance of the participants and the artificial intelligence models (TrainedGPT, a custom GPT-4 model configured with a nephrology guideline-based internal repository, and GPT-4o) was evaluated based on their written responses to nine simulated nephrology case scenarios. Responses were scored by two blinded expert nephrologists using a 5-point Likert scale across three main stages: prediagnosis, diagnosis, and treatment.

Results: Both TrainedGPT (13.2 [interquartile range (IQR): 9.9 – 13.7]) and GPT-4o (12.8 [IQR: 8.6 – 13.7]) models significantly outperformed the fellow group (8.6 [IQR: 7.0 – 10.6]) (p = 0.001 and p = 0.016, respectively). This superiority was maintained throughout all stages of prediagnosis, diagnosis, and especially treatment. There was no significant difference between the two artificial intelligence models (p = 0.690).

Conclusion: These pilot findings suggest that GPT-4-based AI models may serve as useful adjuncts in nephrology education and structured clinical support, outperforming junior fellows in standardized, evidence-based decision-making tasks. Given the exploratory design and small sample size, these results should be interpreted cautiously with respect to generalizability. The lack of a significant difference between TrainedGPT and GPT-4o suggests that the foundational medical knowledge of current large language models may already be robust for standard nephrology scenarios.