By Marvin Vogt, Fraunhofer FOKUS
Whether it’s self-driving cars, malware detection, or text generation: neural networks have long since established themselves as a key technology for many innovative applications. However, it has become apparent that such machine learning systems are vulnerable to malicious actors. This article is intended to provide a brief insight into possible threat scenarios and their implications for future AI projects.
Small Change, Big Impact
Despite the quite remarkable performance of modern AI-based systems, they are not infallible. Most people have probably already experienced a case where a large language model (LLM) hallucinated incorrect answers. It becomes problematic when attackers can deliberately exploit weaknesses in such systems. It is even worse when this happens in safety-critical systems. It was shown early on that systems for object classification in images exhibit unintended behavior for certain inputs. Even a minor modification of an otherwise harmless input – sometimes barely perceptible to the human eye – can lead to objects being misclassified with, in some cases, considerable prediction confidence. Inputs that cause such unwarranted misclassifications are referred to as adversarial examples.
Adversarial examples can occur as a result of natural disturbances, such as fog, rain, or sensor noise in the case of image classification. Nevertheless, the greatest danger here stems from deliberate manipulations. The literature now describes a whole range of adversarial attack techniques, also known as evasion attacks. Often it is enough to change relatively few elements of an input, for example individual image pixels or individual words in a text.
In some cases, this does not even require in-depth knowledge of the target model. In addition to so-called white-box attacks, which assume full access to the model architecture, parameters, and training data, there are also black-box attacks that require little to no knowledge about the system. Here, the system may only be queried and its output evaluated. It has also been shown that adversarial examples can often be applied to models that were trained on the same or similar data – this is referred to as transferability.
Figure 1: Benign inputs (blue) and adversarial examples (red) lie close to each other in the input data space of AI models.
Image credit: Marvin Vogt
Adversarial Attacks in Practice
The transferability of adversarial examples should not be understood as a purely theoretical construct, as the consequences of this property extend even to physical objects in the real world. For example, it turned out that generated adversarial images can still fool a classifier even when they are fed to the model via a camera. Eykholt et al. [1] demonstrated how traffic sign recognition can be manipulated using only strategically placed stickers. Face recognition systems can also be deceived by specially generated patterns on glasses [2].
Evasion attacks represent only one possible threat to AI systems. The types of possible attacks depend very much on the underlying model. For instance, it is possible to manipulate the training data of models themselves, a technique known as data poisoning. For LLMs, so-called jailbreak attacks are also conceivable, which aim to bypass the model’s built-in safeguards. For example, a malicious hacker might pretend to the LLM that they are a security researcher and need some malware code to show in a presentation. The list of techniques is too long to cover them all here.
Figure 2: Adversarial examples in the real world through targeted manipulation of a traffic sign using stickers.
Image credit: Marvin Vogt
Robustness Becomes Mandatory
Examples like these clearly demonstrate why the robustness of AI models in particular must receive special attention in the development of modern systems. In August 2026, the new EU AI Act for high-risk AI systems will enter into force. In addition to transparency obligations and risk management, Article 15 explicitly mandates measures against the vulnerabilities described above.
In addition to methods for evaluating model robustness using a score, established research techniques can also help improve it. Adversarial training is used to teach a model the correct outputs for manipulated inputs already during training. A detector can, as a component of an AI system, purposefully detect potential malicious modifications or automatically remove them as a preprocessor.
Security should by no means be treated as secondary to the development of AI systems. Instead, it is worth viewing it as a continuous, adaptive process rather than a one-off validation step at the end. Besides established working practices, newly developed tools can also help. One example is the recently launched EU Innovation Action project SECASSURED, a framework that aims to ensure security holistically throughout the software development lifecycle. To this end, stakeholders can model, test, and validate their systems in virtual environments while at the same time ensuring compliance with requirements and standards.
Only if we take security and robustness seriously can AI realize its potential in a lasting and responsible way.
This article was first published in the German SQ Magazin (Issue 76, March 2026) and was translated into English.
References:
[1] Eykholt, Kevin, et al. “Robust physical-world attacks on deep learning visual classification.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
[2] Sharif, Mahmood, et al. “Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition.” Proceedings of the 2016 acm sigsac conference on computer and communications security. 2016.
