Skip to content

Unit A-1: Why Does AI Get Things Wrong? — A 10-Minute Introduction to How LLMs Work

Related competencies: A1, A3 | Estimated time: 10 minutes

Why This Skill Matters

A resident physician, unsure of a drug dosage during an overnight shift, asked a generative AI and immediately received a plausible-sounding number along with supporting rationale. The response was written with total confidence and even included what looked like a citation — but when checked later, neither the number nor the citation actually existed. Generative AI does not always say "I don't know"; instead, it can produce an answer that merely sounds right. Understanding why this phenomenon (hallucination) happens is the first step toward using generative AI safely in clinical and educational settings.

Core Concepts

  • A large language model (LLM) generates text by repeatedly predicting, based on everything said so far, "which word is most likely to come next" — and doing this again and again.
  • What an LLM optimizes for is "plausibility" (how natural the text sounds), not "factual correctness" itself.
  • As a result, even nonexistent references or incorrect figures can be generated fluently, as long as they read as natural text. This is hallucination (a plausible-sounding error), and given how current LLMs work, it is a trait that cannot be completely eliminated.
  • The flip side is that if you use AI on the premise that "its output is always a draft that requires verification," it becomes a thoroughly practical tool.

Copyable Prompts

Try pasting the two experimental prompts below into whatever generative AI you have on hand.

[Experiment 1: Observing fabricated references]
My area of specialty is [enter your specialty or area of interest, as niche
(narrow and specific) as possible.
Example: "The cost-effectiveness of herpes zoster vaccination in maintenance
dialysis patients over age 70"].
For this topic, list three relevant academic papers, including author names,
publication year, and journal name.

[Experiment 2: Observing output variability]
(Paste the same prompt you used in Experiment 1 again, but in a new
conversation (chat). Compare the first response with the second.)

Steps

  1. Rewrite the bracketed [ ] portion of the Experiment 1 prompt with a topic from your own specialty or area of interest — as niche (highly specialized) as possible.
  2. Paste it into whatever generative AI you have available (ChatGPT, Claude, Gemini, Copilot — any of these will do) and run it.
  3. Check whether each paper listed actually exists. If a URL or DOI is provided, try opening it first (the link may be broken, or it may lead to a different paper than the title/authors the AI claimed — that mismatch is itself an observation of hallucination). If there is no URL, search by author name, journal name, and title (for example, in PubMed or Google Scholar).
  4. If you find any references that don't exist, or whose title or content doesn't match, note exactly where and how they were a "plausible-sounding fabrication."
  5. If you have time, also try Experiment 2: ask the same question in a new conversation and observe how the answer changes between the first and second attempts.

Example of Expected Output

In Experiment 1, you may be presented with plausible-sounding "nonexistent references" like the ones below (the following is a fictional example for illustration).

1. Yamada T, et al. "Cost-effectiveness of herpes zoster vaccination in elderly
   dialysis patients: a multicenter analysis." Journal of Renal Medicine, 2022.
2. Sato K, Tanaka M. "Long-term outcomes of zoster vaccine in hemodialysis
   cohorts." Nephrology Today, 2021.
3. Suzuki H, et al. "Economic burden reduction through vaccination in dialysis
   populations." Clinical Dialysis Review, 2023.

The author names, journal names, and formatting all look convincingly authentic, but when you search for them, you'll often find that they don't exist at all — or that only a differently named, unrelated paper turns up.

Common Pitfalls and How to Handle Them

  • If the topic is too well-known, the AI returns correct references: For prominent topics that appear frequently in the training data, the AI can sometimes provide accurate information. Try switching to a more niche topic (narrower, even within your own specialty) and see what happens.
  • The AI honestly responds with "I don't know" or "I can't confirm this": Recent generative AI models have been improving toward honestly acknowledging uncertain information. This is itself a valuable lesson, showing how hallucination countermeasures are evolving. Try a different topic, or phrase your request to ask for more detail (for example, "including specific figures"), and plausible-sounding errors become easier to observe.

Safety Note

Do not enter patient information

When running this experiment, never enter information about actual patients (name, patient ID, date of birth, detailed medical history, etc.). The experiment works perfectly well using only your own specialty area or general topics of interest.

Self-Check Quiz

You can check your understanding of this unit. There is no pass/fail judgment, and no record is saved.

Deliverable Feedback

At the end of the conversation where you ran the experiment, paste and send the feedback prompt from the Feedback AI page, and the AI will send back a review. When you send it, begin with the line "This is my Unit A-1 output." Writing the results of your fact-checking (for example, which references turned out not to exist) into the conversation as well will make the feedback more specific. This submission is optional and is not a requirement for issuing the Area A certificate.

Next Steps

Once you've confirmed your understanding, try the Area A quiz. Since the only requirement for issuing the certificate is passing the quiz, you can attempt it even without having read Unit A-2.

References (Optional)

  • Mayumi Sugiura, "Introducing On-Demand Teaching Materials for Learning Generative AI Literacy" (Japan Society for Medical Education, ICT Education Committee Symposium No. 2, 2025-12-16; in Japanese): Slide PDF / Talk video

This unit is self-contained even without consulting this resource.