Editor’s Note: This article was originally authored by our colleague and BARC Fellow, Douglas Laney, and was first published on Forbes.com. We are republishing it with full permission, as we believe its insights are highly relevant to the topics we cover and valuable for our community.
In a moment of profound artificial irony, a Stanford professor and misinformation expert submitted expert testimony that included fake citations generated by ChatGPT. The incident, which occurred in a lawsuit involving a media organization, perfectly encapsulates the risks and challenges of relying on generative AI for factual research.
The professor had used ChatGPT to help prepare the testimony but failed to verify the sources it provided. As has become a well-documented issue, the AI model “hallucinated” several academic articles and authors that did not exist, creating plausible but entirely fabricated citations. This led to a motion to strike the testimony and caused significant professional embarrassment.
This case is a critical lesson for any business integrating AI into its workflows. The tendency for Large Language Models to hallucinate is not a bug to be fixed but a fundamental characteristic of their probabilistic nature. They are designed to generate convincing text, not to be factually accurate databases.
For organizations, this means that human oversight is non-negotiable. Any factual claim, citation, or data point generated by an AI must be rigorously verified before it is used in any business context, whether it’s a marketing document, a legal brief, or an internal report. Failing to do so exposes the organization to risks of misinformation, legal liability, and a loss of credibility.
The challenge of AI “hallucinations” is more than a technical quirk; it creates significant legal and intellectual property risks. Who is liable when an AI generates false or copyrighted information? For professionals seeking clarity on these complex issues, our BARC+ subscription offers unrestricted access to our full research library. A relevant analysis on this topic is our guide to AI and intellectual property.
