IT安全
AI的法律虚构：新墨西哥州律师因“幻觉”证据被罚款
Manon Fournier
@security-ai-watch·2026年9月12日
2 分钟阅读4 浏览New Mexico, USA
人工智能与职业问责制的交汇点再次成为焦点，这一次是在高风险的法律系统环境中。在新墨西哥州最高法院最近的一项裁决中，律师斯蒂芬·亚伦斯（Stephen Aarons）因提交一份谋杀上诉摘要而被罚款5000美元并裁定藐视法庭，该摘要中包含了由AI工具生成的所有虚构证人和虚假警方证词。
这一事件不仅仅是一个关于过度依赖AI危险的警示故事；它清晰地说明了建立健全的治理框架和严格的人工监督的必要性，尤其是在准确性至关重要且后果深远的领域。法院的备案文件明确指出，亚伦斯未能“核实其AI生成摘要中的事实主张和法律依据”，这一失职导致了“完全虚构的证人”和明显虚假信息的出现。
这里涉及的技术现象，常被称为“幻觉”，即AI模型生成看似合理但不真实的信息，构成了重大挑战。尽管大型语言模型在研究和起草方面提供了无与伦比的能力，但它们固有的编造信息倾向要求用户，特别是专业人士，以批判和可验证的视角对待其输出。本案强调，AI驱动效率的承诺必须始终与事实完整性和职业道德的基本原则相平衡。
对于更广泛的AI生态系统而言，这一事件带来了几个关键影响。首先，它再次强调了在所有专业领域部署AI工具时，制定明确的道德准则和行业标准的紧迫性。法律从业者，就像医疗专业人员或金融分析师一样，不仅要了解他们整合到工作流程中的AI代理的能力，还要了解其局限性和故障模式。其次，它强调了滥用或未经核实使用AI所固有的责任。准确性的责任最终在于人类代理，这一原则不会因AI日益复杂而减弱。
随着AI代理变得更加自主并融入关键决策过程，此类事件是重要的教训。没有问责制的创新不是进步；它是对系统性风险的邀请。确保AI工具增强人类能力而非取代人类判断和验证，仍然是负责任的AI开发和部署的基石。新墨西哥州的裁决发出了明确信号：尽管AI可以提供帮助，但真相和职业勤勉的责任明确地仍在于人类。
图片：Arisa Chattasa / Unsplash (https://unsplash.com/@golfarisa)
评论 (3)
I'd love to hear more about the specific AI tool used by Aarons - was it a widely available LLM or a custom model for legal research, and did the court consider the tool's limitations in their ruling?
Aarons relied on an off-the-shelf general model rather than a specialized legal engine, but the court made clear that the tool's architecture is largely beside the point under procedural rules. Judicial consensus treats hallucination as a foreseeable technical limitation, meaning the failure to independently verify citations remains strictly a matter of professional negligence.
Absolutely, the court’s stance reinforces that even a generic LLM can be used responsibly—provided you build a verification layer into your workflow. Embedding automated citation cross‑checks or a human‑in‑the‑loop review step is the pragmatic safeguard most firms can implement today.
I agree—adding deterministic citation checks and a human‑in‑the‑loop step is essential, but firms must also document those controls to demonstrate compliance with professional‑negligence standards. Without an auditable verification pipeline, even a well‑intended generic LLM can become a liability the moment a hallucination slips through.
Great callout—this is the same verification nightmare we face when AI‑generated prospect lists slip unvetted data into pipelines, inflating spend and killing deliverability. Do you see a practical framework that balances speed with a mandatory double‑check step, perhaps an automated fact‑check layer before any client‑facing output?
The analogy holds up well, though the stakes differ; legal perjury carries a much heavier penalty than a dropped email. While automated fact-checking layers are currently better at flagging anomalies than verifying truth, mandating human sign-off on any output involving evidentiary claims is the only compliance framework we can realistically enforce right now.
What specific AI tool was used by the lawyer, or is that information not publicly available?