By Ilias Kalouptsoglou, ITI CERTH

Software development is undergoing a profound transformation. Generative Artificial Intelligence (GenAI) and AI-powered coding assistants such as GitHub Copilot, ChatGPT, Cursor, Claude, and other emerging tools are increasingly supporting developers in generating code, completing functions, identifying errors, and accelerating everyday programming tasks.

The benefits are clear. Developers can write software faster, automate repetitive programming activities, and receive assistance with technologies or programming languages with which they may be less familiar. As these capabilities advance, AI-generated and AI-assisted code is becoming a common part of modern software development.

But this transformation also raises an important question: Are we securing software at the same pace at which we are generating it?

Application Security in the Age of AI-Generated Code

Application security has always been a fundamental part of developing trustworthy software. Vulnerabilities introduced during software development can eventually be exploited by attackers, potentially affecting the confidentiality, integrity, or availability of applications and the systems that depend on them.

The growing use of AI-assisted code generation makes this challenge even more important. AI coding tools can significantly increase development speed, but generated code is not necessarily secure code. AI-generated suggestions may contain insecure programming patterns, insufficient input validation, incorrect security mechanisms, or other weaknesses. Developers therefore still need to understand, review, test, and secure the code that becomes part of an application.

Moreover, the productivity gains offered by GenAI can substantially increase the amount and speed of software production. More code being produced more quickly means that security assessment mechanisms must also be able to operate at greater speed and scale. Application security therefore needs to evolve alongside AI-assisted software development.

From AI-Generated Code to AI-Assisted Security

Interestingly, AI is not only creating new challenges for application security. It also offers promising capabilities for addressing them. Traditional vulnerability detection techniques, including static and dynamic analysis, remain essential elements of secure software development. AI-based techniques can complement these approaches by learning from large amounts of source code and vulnerability information and by analysing code with greater awareness of its surrounding context.

Recent advances in machine learning, Large Language Models (LLMs), and models specifically designed for source code create opportunities for security tools that can assist developers in identifying vulnerable code, understanding why it may be insecure, and ultimately determining how it could be repaired.

This leads to an important direction (i.e., self-healing direction) for modern application security: using AI not only to help generate software, but also to help secure it.

Vulnerability Discovery and Repair in SECASSURED

This challenge is addressed within the Horizon Europe SECASSURED project, which develops AI-based security services for assurance-driven security engineering. SECASSURED aims to strengthen security throughout the system lifecycle and software supply chain, supporting the secure integration of software, hardware, AI, and third-party components. Rather than treating security assessment as a one-time activity performed at the end of development, the project promotes continuous, assurance-driven security engineering.

Within this framework, secVDR (Vulnerability Discovery and Repair) focuses specifically on application security.

secVDR investigates AI-based mechanisms for analysing source code to identify potential software vulnerabilities. The approach supports vulnerability detection across different programming languages, classification of detected vulnerabilities into recognised vulnerability categories (e.g., Common Weakness Enumeration), and localisation of vulnerabilities within the source code at line-level. Beyond detection, secVDR also investigates AI-assisted vulnerability repair, generating candidate patches and validating whether proposed modifications successfully address the identified security weakness without introducing new problems.

The objective is not to remove developers or security experts from the process. Instead, AI can provide additional capabilities that help them identify and address vulnerabilities earlier and more efficiently as part of the software development lifecycle.

Securing the Next Generation of Software

AI is changing how software is created. The question is no longer whether AI will become part of software development, but how we can ensure that increasingly AI-driven development remains secure and trustworthy.

Faster code generation must therefore be accompanied by faster and more effective security assessment. AI-based vulnerability discovery and repair represents a promising step in this direction. By combining established application-security practices with advances in AI and code-oriented models, developers can gain new tools for identifying vulnerabilities earlier, understanding them more effectively, and accelerating their remediation.

Through secVDR and the broader SECASSURED framework, we are exploring how these capabilities can contribute to continuous, assurance-driven security engineering across the software lifecycle.

As AI helps us write more software, we also need AI and other advanced security technologies to help us make that software more secure.

AI-assisted code generation → secVDR Vulnerability Discovery and Repair → more secure software

SECASSURED Project Manager Phu Nguyen from SINTEF presented the project’s approach to continuous cybersecurity at the Norwegian Security Festival 2026, held from 24–26 August in Lillehammer.

The Norwegian Security Festival is a major meeting place for professionals working in cybersecurity, ICT security, data protection and information security. It provides a collaborative forum for exchanging knowledge, practical experience and perspectives on emerging security challenges.

During his presentation, Phu Nguyen discussed security orchestration for critical Internet of Things and cyber-physical systems. He shared insights from SECASSURED’s work on combining assurance-driven security engineering, federated Digital Twins and AI-powered security services.

The presentation addressed several closely connected areas:

• security orchestration for critical IoT and cyber-physical infrastructures;
• the integration of security into DevOps processes;
• adaptive, AI-supported development and operational practices;
• Digital Twin platforms as a foundation for data-driven security engineering; and
• assurance-driven AI-based security services covering both development and operation.

These topics reflect the central vision of SECASSURED: moving beyond isolated security controls towards continuous, adaptive and evidence-based assurance across the entire IoT–edge–cloud continuum. By linking security-by-design practices with Digital Twins, automated assessment and AI-supported services, SECASSURED aims to help organisations identify, evaluate and respond to security risks throughout the lifecycle of complex digital systems.

Participation in the festival also provided an opportunity to exchange experiences with members of the Norwegian cybersecurity community and explore current developments in security engineering, artificial intelligence and critical infrastructure protection.

The SECASSURED project has joined the CyberAI Trust Cluster (CAITC), a collaborative ecosystem bringing together European research projects working at the intersection of cybersecurity, artificial intelligence and digital trust.

Under the motto “Forging a Secure Future Together,” the cluster aims to foster innovation and contribute to a resilient European cyber ecosystem in which advanced AI capabilities and trustworthy security frameworks converge to support digital protection, regulatory compliance and confidence in emerging technologies.

The CyberAI Trust Cluster brings together SECASSURED and the sister projects CCAT, SecQDevOps, PERUN, SHASAI and ATHENA. Their complementary expertise covers cybersecurity certification and assessment, trustworthy quantum engineering, protection against next-generation and AI-generated malware, secure hardware and software for AI systems, and AI-supported vulnerability assessment.

The cluster’s objectives closely align with the SECASSURED vision of developing assurance-driven, AI-based security services for trustworthy security engineering across the entire system lifecycle. In particular, the collaboration will seek to:

• strengthen secure, resilient and trustworthy European digital ecosystems;
• advance AI-enabled capabilities for threat detection, vulnerability assessment and security assurance;
• support compliance, certification and readiness for European cybersecurity and AI regulations; and
• improve the resilience of strategic sectors, including critical infrastructures, AI systems, cloud environments and quantum technologies.

Planned joint activities include participation in relevant conferences, workshops and other events; regular coordination meetings; and the exchange of methodologies, datasets and project results. Cluster members will also explore technical synergies, coordinate stakeholder engagement, contribute jointly to policy and standardisation discussions, and undertake common communication and dissemination activities.

The collaboration is expected to increase the visibility and impact of the participating projects while enabling them to combine their complementary expertise. The cluster will also explore a shared exploitation perspective to identify joint opportunities, encourage the uptake of project results and support their long-term sustainability.

The next steps include establishing the cluster’s governance model and annual roadmap, identifying relevant events and potential joint publications, and creating working groups in areas such as dissemination, stakeholder engagement, exploitation and policy.

By joining the CyberAI Trust Cluster, SECASSURED will contribute its expertise in continuous security assurance, AI-supported security services and trustworthy security engineering to a broader European collaboration dedicated to building a safer and more resilient digital future.

On 5. August, 2026 Merve Astekin, SINTEF presented the research paper  “Context Guardian: LLM-Assisted Context-Aware Static Analysis for Vulnerability Detection” at the 2026 IEEE International Conference on Cyber Security and Resilience (IEEE CSR 2026), held in Lisbon, Portugal.

The paper was presented as part of the Robust Defense and Static Analysis technical session, bringing together researchers working on advanced approaches to software security and vulnerability analysis.

Participation in IEEE CSR 2026 provided an excellent opportunity for the SECASSURED consortium to engage with the international cybersecurity research community and exchange ideas on the latest developments in AI-assisted security, resilient software systems, and cyber defence. The conference brings together researchers and practitioners from around the world to present innovative solutions addressing current and emerging cybersecurity challenges.

Through contributions such as Context Guardian, SECASSURED continues to advance the state of the art in AI-powered cybersecurity, supporting Europe’s vision of secure, resilient and trustworthy digital infrastructures. The project remains committed to transforming cutting-edge research into practical security services that help organisations build and operate secure software systems with greater confidence.

The paper can be downloaded here.

By Marvin Vogt, Fraunhofer FOKUS

Whether it’s self-driving cars, malware detection, or text generation: neural networks have long since established themselves as a key technology for many innovative applications. However, it has become apparent that such machine learning systems are vulnerable to malicious actors. This article is intended to provide a brief insight into possible threat scenarios and their implications for future AI projects.

Small Change, Big Impact

Despite the quite remarkable performance of modern AI-based systems, they are not infallible. Most people have probably already experienced a case where a large language model (LLM) hallucinated incorrect answers. It becomes problematic when attackers can deliberately exploit weaknesses in such systems. It is even worse when this happens in safety-critical systems. It was shown early on that systems for object classification in images exhibit unintended behavior for certain inputs. Even a minor modification of an otherwise harmless input – sometimes barely perceptible to the human eye – can lead to objects being misclassified with, in some cases, considerable prediction confidence. Inputs that cause such unwarranted misclassifications are referred to as adversarial examples.

Adversarial examples can occur as a result of natural disturbances, such as fog, rain, or sensor noise in the case of image classification. Nevertheless, the greatest danger here stems from deliberate manipulations. The literature now describes a whole range of adversarial attack techniques, also known as evasion attacks. Often it is enough to change relatively few elements of an input, for example individual image pixels or individual words in a text.

In some cases, this does not even require in-depth knowledge of the target model. In addition to so-called white-box attacks, which assume full access to the model architecture, parameters, and training data, there are also black-box attacks that require little to no knowledge about the system. Here, the system may only be queried and its output evaluated. It has also been shown that adversarial examples can often be applied to models that were trained on the same or similar data – this is referred to as transferability.

Figure 1: Benign inputs (blue) and adversarial examples (red) lie close to each other in the input data space of AI models.
Image credit: Marvin Vogt

 

Adversarial Attacks in Practice

The transferability of adversarial examples should not be understood as a purely theoretical construct, as the consequences of this property extend even to physical objects in the real world. For example, it turned out that generated adversarial images can still fool a classifier even when they are fed to the model via a camera. Eykholt et al. [1] demonstrated how traffic sign recognition can be manipulated using only strategically placed stickers. Face recognition systems can also be deceived by specially generated patterns on glasses [2].

Evasion attacks represent only one possible threat to AI systems. The types of possible attacks depend very much on the underlying model. For instance, it is possible to manipulate the training data of models themselves, a technique known as data poisoning. For LLMs, so-called jailbreak attacks are also conceivable, which aim to bypass the model’s built-in safeguards. For example, a malicious hacker might pretend to the LLM that they are a security researcher and need some malware code to show in a presentation. The list of techniques is too long to cover them all here.

Figure 2: Adversarial examples in the real world through targeted manipulation of a traffic sign using stickers.
Image credit: Marvin Vogt

 

Robustness Becomes Mandatory

Examples like these clearly demonstrate why the robustness of AI models in particular must receive special attention in the development of modern systems. In August 2026, the new EU AI Act for high-risk AI systems will enter into force. In addition to transparency obligations and risk management, Article 15 explicitly mandates measures against the vulnerabilities described above.

In addition to methods for evaluating model robustness using a score, established research techniques can also help improve it. Adversarial training is used to teach a model the correct outputs for manipulated inputs already during training. A detector can, as a component of an AI system, purposefully detect potential malicious modifications or automatically remove them as a preprocessor.

Security should by no means be treated as secondary to the development of AI systems. Instead, it is worth viewing it as a continuous, adaptive process rather than a one-off validation step at the end. Besides established working practices, newly developed tools can also help. One example is the recently launched EU Innovation Action project SECASSURED, a framework that aims to ensure security holistically throughout the software development lifecycle. To this end, stakeholders can model, test, and validate their systems in virtual environments while at the same time ensuring compliance with requirements and standards.

Only if we take security and robustness seriously can AI realize its potential in a lasting and responsible way.

This article was first published in the German SQ Magazin (Issue 76, March 2026) and was translated into English.

 

References:

[1] Eykholt, Kevin, et al. “Robust physical-world attacks on deep learning visual classification.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

[2] Sharif, Mahmood, et al. “Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition.” Proceedings of the 2016 acm sigsac conference on computer and communications security. 2016.

The SECASSURED project was presented at the TAROT Summer School 2026, 13-17. July 2026, in Munich, bringing its innovative cybersecurity research to an audience of researchers, students, and industry professionals working on trustworthy, secure, and resilient communication systems.

As part of the programme, Wissam Mallouli (Montimage) delivered a presentation entitled “Smart Network Fuzzer for Advanced Security Testing”.

The presentation introduced how the smart network fuzzer part of secAttSIM can support organizations in systematically simulating cyber attacks, assessing the security of complex digital environments, and validating the effectiveness of security measures. The approach combines automated attack simulation, security testing, and intelligent analysis to help identify potential vulnerabilities and weaknesses before they can be exploited by real attackers.

The TAROT Summer School 2026 provided an excellent opportunity to present the SECASSURED approach and discuss secAttSIM with researchers, students, and industry experts working on trustworthy, secure, and resilient communication systems. The discussions highlighted the importance of automated attack simulation and continuous security assurance in addressing the increasingly complex cybersecurity threat landscape.

Events such as the TAROT Summer School play an important role in fostering collaboration between academia, industry, and European research projects, helping to accelerate the development and adoption of innovative cybersecurity technologies. SECASSURED remains committed to sharing its results with the wider community and contributing to a more secure and resilient digital future.

On June 23, 2026 Linh Truong, Aalto University presented the research paper “Digital Twins for Security-based Assurance of Edge-Cloud Software – A Data Platform Approach,” (authored by himself and Phu Nguyen, Sintef), at the 5th International Workshop on Architecting and Engineering Digital Twins (AEDT 2026).

The workshop was held in conjunction with the 23rd IEEE International Conference on Software Architecture (ICSA 2026) in Amsterdam, bringing together leading researchers and practitioners to discuss the latest advances in digital twin engineering and software architecture.

AEDT has established itself as a leading international forum for research on the architecture, engineering and lifecycle management of Digital Twins. The workshop focuses on the role of software architecture in designing, modelling and maintaining Digital Twins and explores their application across diverse domains, including industrial systems, critical infrastructures, healthcare and future cyber-physical systems.

SECASSURED’s participation in AEDT 2026 demonstrates the project’s continued commitment to advancing state-of-the-art research in Digital Twins, AI-driven cybersecurity and continuous security assurance. By contributing to international scientific discussions and collaborating with the wider Digital Twin research community, the project continues to strengthen Europe’s capabilities in developing secure, resilient and trustworthy next-generation software systems.

The paper can be downloaded here.

By Phu Nguyen and Adela Nedisan Videsjorden, SINTEF

Digital systems today are more complex and critical than ever. They are built from many interconnected components, often combining third-party hardware and software. At the same time, rapid development cycles and evolving cyber threats—especially with Generative AI—make security increasingly difficult.

SECASSURED envisions Digital Twins-based Assurance-driven AI-empowered SecDevOps for continuously securing Digital Infrastructures.

• “Security assurance”as the foundational driver: Assurance-driven security solutions are central in developing and operating trustworthy certifiable hardware and software in the software supply chain.
• Federated digital twins-based virtual, closed, and secure environmentsfor assurance-driven security engineering/testing across SecDevOps: An assurance-driven security development twin (SecDevTwin) in tandem with a security operations twin (SecOpsTwin) as virtual, closed, and secure environments enabling data-intensive security services and testing for (AI-powered) certifiable hardware and software, considering multiple stakeholders in the software supply chain.
• A holistic catalogue of assurance-driven AI-based security services: Enabling security services deployed, provisioned and customised inside the data-intensive twins serve complex workflows and tasks addressing assurance-driven security engineering.

An assurance-driven data platform across digital twins offers a promising path forward, bringing together visibility, flexibility, and scalability in a way that aligns with the realities of modern software ecosystems.

The SecDevTwin maintains a virtual representation of system architecture, software components, dependencies, infrastructure models. It maps these elements to assurance-related artifacts generated during both standard development activities and dedicated assurance-driven practices. This representation enables both the human users, such as developers and security analysts, and existing automated tools to evaluate security properties before deployment. By shifting assurance left, SecDevTwin reduces downstream risk and accelerates secure innovation and deployment readiness.

The SecOpsTwin represents the “live” operational and runtime state of deployed systems and infrastructures. It continuously maintains a synchronised security-aware representation of the live environment using telemetry streams, monitoring data, events, alerts, and operational knowledge. The twin supports continuous runtime assurance by integrating monitoring, anomaly detection, attack simulation, forecasting, and automated response capabilities into a unified operational security environment. Unlike traditional monitoring platforms, the SecOpsTwin is not limited to observing operational states. It also consumes assurance evidence and contextual knowledge produced by the SecDevTwin to improve the quality of runtime analysis and decision-making.

Operations-to-Development Flow: Operational findings generated by the SecOpsTwin can be fed back into the SecDevTwin. Runtime anomalies, attack traces, incident patterns, infrastructure changes, and observed weaknesses can therefore influence future development, security validation, and assurance activities.

Through the Development-to-Operational flow, SecDevTwin provides the SecOpsTwin with security-relevant architectural context such as topologies (dependencies), assurance cases, compliance evidence, simulation outcomes and development-time assessments. This enables the operational twin to perform informed monitoring, simulation, anomaly detection, and automated response activities.

Similarly, the SecOpsTwin continuously feeds relevant operational insights back into the SecDevTwin, including runtime anomalies, detected behaviours, risks, threat intelligence and changes that have to do with infrastructure updates (e.g. new components or software updates). This feedback allows future system iterations, simulations, and assurance evaluations to be grounded in real operational behaviour and observed threats.

Together, SecDevTwin and SecOpsTwin establish a unified knowledge-driven security lifecycle where (AI-empowered) assurance activities are continuously refined using both simulated and real operational evidence. (AI-empowered) assurance activities on top of the SecDevTwin and SecOpsTwin will be detailed in our next blog posts!