Paper: 2604.19598 Authors: Kihyuk Lee Categories: cs.AI, cs.CL

Abstract

This study compared repeated generation consistency of exercise prescription outputs across three large language models (LLMs): GPT-4.1, Claude Sonnet 4.6, and Gemini 2.5 Flash, under temperature=0 conditions. Each model generated prescriptions for six clinical scenarios 20 times, yielding 360 total outputs analyzed across four dimensions: semantic similarity, output reproducibility, FITT classification, and safety expression.

Key Findings

  • GPT-4.1: 100% unique outputs with highest semantic similarity (0.955) - varying form but stable content
  • Claude Sonnet 4.6: 70% unique outputs, semantic similarity 0.903
  • Gemini 2.5 Flash: Only 27.5% unique outputs - produces high similarity through text duplication

Critical Insight

Temperature=0 does NOT guarantee determinism! The same setting produced fundamentally different consistency profiles:

  • GPT-4.1 varies surface expression while maintaining semantic stability
  • Gemini 2.5 Flash exhibits pronounced output repetition (2 unique outputs across 20 trials in one scenario)

Methodology

  • 6 clinical scenarios: knee osteoarthritis, cancer patient, healthy adults
  • 20 repeated generations per scenario per model
  • Metrics: SBERT cosine similarity, FITT classification, safety expression
  • All models showed ceiling-level safety (3.93-3.99/4.00)

Results

ModelSemantic SimilarityUnique Outputs
GPT-4.10.955100%
Gemini 2.5 Flash0.95027.5%
Claude Sonnet 4.60.90370.0%

Takeaways

  • Model selection for clinical deployment is a clinical decision, not just technical
  • Single-output evaluations cannot capture fundamental consistency differences
  • Gemini’s high similarity derives from repetition, not stable reasoning
  • Safety expression has converged to baseline across current LLMs

论文: 2604.19598 作者: Kihyuk Lee 分类: cs.AI, cs.CL

摘要

本研究比较了GPT-4.1、Claude Sonnet 4.6和Gemini 2.5 Flash三种大语言模型在temperature=0条件下重复生成运动处方的一致性。每个模型为6个临床场景各生成20次处方，共360个输出，从四个维度进行分析：语义相似度、输出可重复性、FITT分类和安全性表达。

关键发现

  • GPT-4.1：100%独特输出，语义相似度最高（0.955）——形式多样但内容稳定
  • Claude Sonnet 4.6：70%独特输出，语义相似度0.903
  • Gemini 2.5 Flash：仅27.5%独特输出——通过文本重复实现高相似度

核心洞察

Temperature=0不能保证确定性！相同设置产生了根本不同的一致性特征：

  • GPT-4.1变化表面表达同时保持语义稳定性
  • Gemini 2.5 Flash表现出明显的输出重复（在一个场景的20次试验中仅2个独特输出）

方法论

  • 6个临床场景：膝关节骨关节炎、癌症患者、健康成人
  • 每个场景每个模型重复生成20次
  • 指标：SBERT余弦相似度、FITT分类、安全性表达
  • 所有模型均达到天花板级安全性（3.93-3.99/4.00）

实验结果

模型语义相似度独特输出比例
GPT-4.10.955100%
Gemini 2.5 Flash0.95027.5%
Claude Sonnet 4.60.90370.0%

要点总结

  • 临床部署的模型选择是临床决策，而非仅是技术决策
  • 单次输出评估无法捕捉根本的一致性差异
  • Gemini的高相似度来自重复而非稳定的推理
  • 安全性表达已在当前LLM中收敛到基线水平