Know the Risks, Work Safely

As you saw in Lesson 2, a language model predicts the most likely continuation of a text, word by word — it does not look things up in a verified database and has no concept of true versus false. When the model lacks knowledge, it produces plausible-sounding content that is simply made up. This behaviour is called hallucination, and it is not a rare glitch but a fundamental property of the technology.

In a professional context, three forms are especially dangerous: invented sources (studies, court rulings or standards that do not exist), invented numbers (market data, deadlines, prices) and invented details in summaries. In the United States, lawyers have already been sanctioned by a court because their brief cited court decisions an AI had fabricated. The treacherous part: hallucinations look exactly as convincing as correct answers — complete with clean formatting, author names and publication years.

This leads to the most important professional rule for working with AI: no fact, figure, quote or legal reference leaves your desk before you have checked it against an independent source. The more important the statement, the more thorough the check. The AI gives you a draft — responsibility for the content stays with you.

AI models learn from existing data — and inherit its imbalances along the way. If certain groups are underrepresented in the training data or were historically disadvantaged, the model can quietly perpetuate those patterns. This is called bias: a systematic distortion that arises without malicious intent but has very real consequences.

Bias becomes critical wherever AI makes statements about people: pre-screening job applications, scoring customer enquiries or prioritising cases. An HR department that has CVs automatically pre-sorted can unknowingly disadvantage entire groups — with legal consequences, for instance under Germany's General Equal Treatment Act (AGG). It is no coincidence that the AI Act classifies AI systems involved in recruitment and candidate selection as high-risk applications.

For your daily work, this means: results that affect people are always reviewed by a human — not as a formality, but with a genuine look at the individual case. Two control questions help: Would I reach the same conclusion based on my own expertise? And does the result look conspicuously different for certain groups?

The GDPR applies in an AI chat just as it does in any email. The moment you enter names, addresses, contract numbers or other personal data of customers, colleagues or applicants into an AI tool, you are processing that data and transmitting it to the tool's provider. That requires a legal basis — and with public free tools, there usually is none. Hence the rule of thumb: no personal data of third parties in public AI tools.

Things are different with tools your company introduces officially. Here, the company signs a data processing agreement with the provider under Art. 28 GDPR: the provider may only process the data on the company's instructions, and the contract typically stipulates, among other things, that inputs are not used to train the models. This is exactly why an 'approved company tool' is something entirely different from 'the same tool on a private account'.

A third issue is where the servers are located. Many AI providers process data outside the EU; under the GDPR, such transfers to third countries are only permitted under additional conditions. Your company resolves this when selecting tools — the German data protection authorities have published dedicated joint guidance on 'Artificial Intelligence and Data Protection'. As a user, it is enough to know: this is one more reason to use approved tools only.

Whatever you enter into an AI tool leaves your company. Many providers store inputs in logs, and with free consumer versions in particular, providers often reserve the right to use inputs to improve their models. A prompt containing pricing calculations, customer lists or draft contracts can thus end up permanently in someone else's systems — and there is no way to get it back.

There is a legal dimension, too: under the German Trade Secrets Act, trade secrets are only protected if the company takes reasonable confidentiality measures. Carelessly entering confidential information into public tools can jeopardise that protection. Internal figures, strategies, source code and customer data therefore belong only in tools that have been explicitly approved for them.

Which brings us to shadow AI: employees using private accounts for company work because it is quick and convenient — usually without any bad intent. The problem: the company knows nothing about it, has no contract with the provider and no control over what happens to the data. If you are missing a tool, the right path is not a private account but a request to IT or your manager — many companies introduce approved alternatives once the need becomes visible.

For everyday work, two copyright essentials are enough. First: AI outputs can reproduce protected content from the training data so closely that using them infringes third-party rights — for example with images, longer text passages or program code. Anyone who publishes AI content is liable for it as if it were their own; 'the AI did it' is no defence.

Second: purely machine-generated content without a human creative contribution generally enjoys no copyright protection under German law, because only personal intellectual creations are protected. A fully AI-generated logo or advertising image may therefore, under certain circumstances, be freely used by anyone. Also check the terms of use of the tool in question — and do not upload third-party protected works you hold no rights to.

All the risks in this lesson can be managed with five simple rules. They are deliberately short — a checklist for every single use of AI, not a document for the filing cabinet.

These five rules are AI literacy in action, in the sense of Art. 4 of the AI Act, which has applied since 2 February 2025: you use the tools' strengths while knowing their limits. How all of this fits into the AI Act's overall system — risk classes, obligations, implementation in your company — is the subject of Lesson 5.