Limitations of generative AI
Generative AI is a technical system – not a thinking actor. Always relate critically to its output by comparing it with academic sources, using multiple prompts, and considering alternative perspectives .
Responses from generative AI are static predictions
When you use generative AI, it is important to understand that you are not in dialogue with a human intelligence. Generative AI operates according to mechanisms and logics that are fundamentally different from those of human thinking, and this has major implications for how you should interpret its output.
When a system such as ChatGPT produces text, it is simply continuing a sequence of words in a way that statistically resembles something a human might write.
Generative AI merely predicts the probability of what the next word in a sentence should be, and those predictions are not always correct or appropriate.
Hallucinations
Generative AI can hallucinate, meaning it may construct an answer that sounds completely correct but is factually wrong.
For example, if GenAI is asked to write a research paper on a topic where it lacks sufficient training data, it may still produce a response that appears plausible by combining familiar patterns. In such cases, references included in a research paper may be entirely fabricated. The models are designed to always produce an answer.
To minimise the risk of hallucinations, it is important to provide GenAI with sufficient context about the topic it is generating responses on, so that the word probabilities become more reliable and the output does not turn into guesswork. GenAI optimises for linguistic probability – not factual correctness .
Bias
Bias in generative AI can refer to distortions or biases that arise when an AI system is trained on data that collectively represents a skewed view of reality. Bias can also refer to bias introduced by the user. Be especially aware of the following types of bias :
The tendency to trust the system blindly – because it sounds convincing – even when it is incorrect or imprecise. As an engineer, you must always check the output against your academic knowledge and validated sources.
The first answer or idea from generative AI may become a fixed reference point, after which you adjust everything relative to it – even if it is not optimal. Try multiple prompts and perspectives to avoid being locked into a limited realm of solution.
Black Box
Generative AI models are often described as black boxes, because it can be difficult to understand exactly how they arrive at a particular answer.
The models consist of extremely complex neural networks with millions or billions of parameters, which have been adjusted through training on large datasets. Even though we understand the overall principles behind the models, it is in practice not possible to trace all internal calculations that lead to a specific output.This means that one cannot fully explain why a model formulates a specific response, or what parts of the training data has had the biggest impact on the result. Thus generative AI should not be viewed as a transparent source of knowledge.
This page is inspired by Aalborg University’s page on generative AI for students. Generative AI has been used in the preparation of this page.
