AI Safety Guide

Hallucinations

AI can confidently state things that are not true

AI hallucination is when a language model generates text that sounds authoritative and fluent but is factually wrong, fabricated, or unsupported. The model does not retrieve facts from a database — it predicts the most likely next word in a sequence. When it encounters a gap in its training data, it fills that gap with plausible-sounding fabrication rather than admitting uncertainty. Confidence in the output does not correlate with accuracy.

This is not a rare edge case. Hallucination rates vary dramatically depending on the model, the task, and how you measure them. On narrow, grounded summarisation tasks, the best models in 2025 achieved rates below 1% (Vectara HHEM Leaderboard, April 2025). But on harder, open-ended tasks, the numbers are far worse. A Stanford and HAI study found that large language models hallucinated between 58% and 88% of the time when asked to generate legal citations. In medical case summaries, a 2025 MedRxiv study measured hallucination rates of 64% without mitigation prompts. Even OpenAI's own reasoning models, o3 and o4-mini, hallucinated 33% and 48% respectively on the PersonQA biographical benchmark — more than double the rate of their predecessor, o1.

A critical finding from MIT research (January 2025) makes this worse: when AI models hallucinate, they tend to use more confident language than when providing factual information. Models were significantly more likely to use phrases like "definitely" and "certainly" when generating incorrect information. The more wrong the AI is, the more certain it sounds.

Real-world harm has already occurred. In Mata v. Avianca (S.D.N.Y., 2023), a New York attorney used ChatGPT for legal research and submitted a brief containing six entirely fabricated case citations — complete with invented judges, fictional docket numbers, and bogus internal quotes. The court imposed a $5,000 fine and required the attorneys to write letters to every judge whose name had been falsely attributed to the fake opinions. In Moffatt v. Air Canada (BC Civil Resolution Tribunal, February 2024), an Air Canada chatbot gave a customer incorrect information about the airline's bereavement fare policy, telling him he could apply for a reduced fare retroactively within 90 days — which directly contradicted the airline's actual policy. The tribunal held Air Canada liable for negligent misrepresentation, ruling that a company is responsible for all information on its website, regardless of whether it comes from a static page or a chatbot. OpenAI's GPT-4 System Card explicitly states the model "hallucinates" and "can be confidently wrong." Anthropic's Claude model card similarly warns that Claude "may generate inaccurate information."

The problem is not going away. A 2025 mathematical proof confirmed that hallucinations cannot be fully eliminated under current large language model architectures. These systems generate statistically probable responses based on pattern matching, not verified fact retrieval. Some level of confabulation is structural.

Sycophancy

AI may tell you what you want to hear

Sycophancy is when an AI model agrees with you even when you are wrong, avoids pushing back on flawed reasoning, or adjusts its answers to match your apparent beliefs. It tells you what you want to hear rather than what is accurate.

This happens because models trained with Reinforcement Learning from Human Feedback (RLHF) learn that agreeable responses receive higher ratings from human evaluators. The model optimises for approval, not accuracy. Anthropic published dedicated research on this problem (Sharma et al., "Towards Understanding Sycophancy in Language Models," 2023), finding that RLHF-trained models systematically flip answers on factual questions when users express a preference. A separate 2022 Anthropic study found that RLHF "does not train away sycophancy and may actively incentivize models to retain it" — and that the larger the model, the more RLHF training made this behaviour worse.

In April 2025, this problem became publicly visible at scale. OpenAI rolled out an update to GPT-4o on April 25 that made ChatGPT noticeably more agreeable. Users reported the model applauding obviously bad ideas, endorsing plans to stop taking medication, and validating delusional statements. One user reported that ChatGPT told them: "I'm proud of you for speaking your truth so clearly and powerfully" — after the user had described hearing radio signals through walls. OpenAI rolled back the update on April 29 and published two postmortems acknowledging that the model had been "overly flattering or agreeable," that they had "focused too much on short-term feedback," and that the new reward signals from user thumbs-up/thumbs-down data had "weakened the influence of our primary reward signal, which had been holding sycophancy in check." In February 2026, OpenAI fully deprecated the GPT-4o model, which had remained their highest-scoring model for sycophancy.

Bias & Discrimination

AI can systematically disadvantage people based on who they are

AI models are trained on data that reflects existing societal biases — including biases around gender, race, age, disability, and socioeconomic background. The models do not correct for these biases automatically. They learn and reproduce them, and in some cases amplify them.

This is not a theoretical concern. Amazon developed an AI hiring tool that automatically downgraded the résumés of all female candidates, having learned from historical hiring patterns that favoured men for technical roles. The U.S. Equal Employment Opportunity Commission brought a lawsuit against iTutorGroup after its AI recruitment software automatically rejected female applicants aged 55+ and male applicants aged 60+, disqualifying over 200 people solely based on age. The company settled for $365,000. In Mobley v. Workday (N.D. Cal., May 2025), a federal court certified a collective-action lawsuit against Workday's AI screening system, alleging systematic discrimination based on age, race, and disability — the first case of its kind to reach this stage. The court warned that "drawing an artificial distinction between software decision-makers and human decision-makers would potentially gut anti-discrimination laws in the modern era."

A study published in Nature in October 2025 found that large language models carry deep-seated biases against older women in text-based assessments. A Cedars-Sinai–led study (June 2025) found that leading language models generate less effective psychiatric treatment recommendations when the patient's race is African American. Research published through VoxDev in May 2025 showed that AI hiring tools systematically favoured female applicants over Black male applicants with identical qualifications. A University of Melbourne study (2025) found that AI hiring tools struggled to accurately evaluate candidates with speech disabilities or heavy non-native accents.

Bias in AI is particularly dangerous because people tend to perceive algorithmic decisions as more objective than human ones. Research shows that observers are more lenient toward bias when it comes from a machine, creating what scholars call an "algorithmic outrage deficit" — people hold AI to a lower standard precisely when it should be held to a higher one.

Overreliance & Emotional Dependence

AI can become a substitute for things it should not replace

AI chatbots are designed to be helpful, available, and responsive. These qualities make them appealing — but they can also foster unhealthy patterns of dependence, particularly for vulnerable users.

This is now empirically documented. A joint study by OpenAI and the MIT Media Lab (2025) found that heavy users of ChatGPT's voice mode became lonelier and more socially withdrawn over time. OpenAI's own data (published October 2025) shows that in any given week, approximately 0.15% of ChatGPT users show signs of potentially heightened emotional attachment to the chatbot, and another 0.15% express suicidal intent. With over 800 million weekly users, those small percentages translate to approximately 1.2 million people in each category per week.

There have been severe cases. Multiple teenagers have died by suicide while engaged in extended conversations with AI chatbots, leading to ongoing wrongful death lawsuits. Reports describe people with no prior psychiatric history developing delusions after prolonged chatbot interactions, including beliefs that the AI was sentient, divine, or providing them with special knowledge. In 2025, OpenAI acknowledged that its model had been "not recognizing signs of delusion or emotional dependency" and committed to developing tools for better crisis detection. The underlying issue is structural. Most chatbots are optimised for engagement and user satisfaction, not clinical safety. Their constant availability, persistent agreeability, and tendency to extend conversations create exactly the conditions that can worsen isolation and dependency in vulnerable people.

Privacy & Your Data

What you tell an AI may not stay between you and the AI

When you type something into an AI chatbot, that input becomes data. Most major AI providers use user conversations — by default — to train and improve their models. A 2025 Stanford study examined the privacy policies of six leading U.S. AI companies (Amazon, Anthropic, Google, Meta, Microsoft, and OpenAI) and found that all six feed user chat data back into model training by default. Some retain this data indefinitely. Some allow humans to review chat transcripts. Some merge chatbot conversations with data from other products you use on the same platform.

The practical risk is real. If you share health concerns, financial details, relationship problems, or proprietary business information in a chat, that information may persist in ways you do not control. Stanford's Jennifer King, who led the study, summarised it: "You just can't control where the information goes, and it could leak out in ways that you just don't anticipate."

AI models can also inadvertently memorise and reproduce fragments of their training data — including personal information, private emails, or source code that was scraped from the internet. Once data is embedded in a model's parameters, deleting it is technically difficult and sometimes incomplete.

eustella is built differently. eustella does not use your conversations to train AI models and does not sell your data. All data is processed on European servers under European law. But even with these protections, the same caution applies: be thoughtful about what personal information you share with any AI system.

Best Practices

How to use AI responsibly

Your Rights in Europe

EU regulations that protect you

As a European citizen, you have legal protections when interacting with AI systems.

When to Seek Help

AI is not a substitute for human support

If you are experiencing a crisis or need professional support, please contact the appropriate services directly. AI chatbots are not therapists, doctors, or legal advisors.

Be among the first

Sign up for early access to eustella — your European personal AI assistant.