24th e-Crime & Cybersecurity Congress – Invicti & S4Applications
While attending the e-Crime & Cybersecurity Congress in London with our partner Invicti, one session in particular stood out. Presented by Liam D’Amato Senior Solutions Engineer, the discussion focused on a rapidly growing but often overlooked risk: Shadow AI.
As organisations continue to embrace AI at speed, the session highlighted a critical reality: AI adoption is accelerating faster than the security and governance frameworks designed to control it. The result is a growing layer of unseen risk across businesses, driven by AI tools operating outside of formal oversight.
Traditionally, AppSec focuses on APIs, web apps, and backend systems. But once AI is embedded, whether it’s a chatbot, recommendation engine, or automated marketing tool, it becomes part of that same attack surface.
What Is Shadow AI and Why Should You Care?
Artificial intelligence is transforming how businesses operate. From marketing automation and customer service chatbots to advanced data analysis, AI is unlocking new levels of efficiency and innovation. But as organisations rush to adopt these technologies, a new and often overlooked risk is emerging in the background: Shadow AI.
Much like Shadow IT before it, Shadow AI refers to the use of AI tools and systems without proper oversight, governance, or security controls. And as AI adoption continues to outpace the frameworks designed to manage it, this hidden layer of risk is growing rapidly.
This could be a marketing team using AI tools to generate content, a developer integrating a chatbot into an application without security review, or employees pasting sensitive business data into publicly available AI platforms. These tools are often adopted with good intentions to save time, improve productivity, or gain insights, but without proper control, they can introduce serious vulnerabilities.
The issue is not the intent; it’s the lack of control.
AI systems are fundamentally different from traditional software. They learn, adapt, and rely heavily on data, making them harder to predict and, if left unmanaged, easier to exploit.
The New Risk Landscape: When AI Goes Wrong
AI doesn’t just improve efficiency; it amplifies both opportunity and risk. As organisations scale their use of AI, they also expand their attack surface.
One of the most immediate concerns is data leakage. AI systems often require access to large datasets to function effectively. If sensitive information such as customer data, financial records, or proprietary business insights is entered into unsecured tools, it can be exposed, stored externally, or even reused in unintended ways.
At the same time, attackers are learning how to manipulate AI systems through techniques like prompt injection, where carefully crafted inputs can override expected behaviour by tricking an AI system into revealing sensitive information or performing unintended actions simply by changing how they phrase a request.
AI is now being used to generate highly convincing phishing attacks, automate vulnerability discovery and exploitation through malicious code. Deepfakes and AI-powered social engineering are becoming mainstream attack vectors.
There are also broader concerns around bias and ethics. AI systems can produce misleading, biased, or inappropriate outputs, which can damage trust and brand reputation if left unchecked.
Bringing Shadow AI Into the Light
One of the most practical takeaways from the session was the importance of visibility. Before organisations can secure AI, they first need to understand where and how it is being used.
This starts with auditing existing AI systems, both officially approved tools and those adopted informally across teams. Many organisations are surprised to discover just how widely AI has already been adopted across the organisation.
From there, responsibilities need to evolve. AI is not just a tool; it becomes part of the operational fabric. This means redefining the roles of both AI systems and human analysts, ensuring that decisions made by AI are accountable, reviewable, and transparent.
Another critical factor is controlling AI’s “data appetite.” Without strong governance, AI systems can consume and expose sensitive information at scale. AI tools should not have unrestricted access to sensitive information, and clear governance policies should be in place to control how data is used.
At the same time, organisations must prepare for what’s next: the rise of agentic AI, where systems act more autonomously. This shift will require even stronger controls, monitoring, and safeguards.
It’s also essential to manage the data that feeds these systems. Perhaps most importantly, organisations need to adopt automated ways of identifying and testing AI-related risks by scanning all AI models and API endpoints.
Real-World Examples of AI Risks
These risks are not theoretical; they are already playing out in real-world scenarios.
In some cases, AI-powered customer service systems have delivered inconsistent or incorrect responses, leading to poor user experiences and reputational damage. Chatbots have been known to produce unexpected or inappropriate outputs when not properly controlled or tested.
Large organisations experimenting with AI have also encountered challenges with misleading or problematic responses, highlighting the importance of governance, monitoring, and accountability.
These examples underline a critical point: AI systems, if left unmanaged, can quickly become liabilities rather than assets.
Read more about our Entreprise solutions.
Securing AI Without Slowing Innovation
The goal is not to stop using AI. In fact, AI will only become more embedded in business operations over time. The real challenge is learning how to innovate safely.
This starts with a mindset shift. AI systems should be treated as critical assets, just like applications, databases, or infrastructure. Security should be built into their design from the beginning, rather than added as an afterthought.
Transparency and accountability are also key. Organisations need to understand how their AI systems work, what decisions they are making, and how those decisions can be reviewed or challenged.
Put simply, responsible AI is secure AI.
How Invicti Helps Secure AI-Powered Applications
This is where application security platforms like Invicti come into play.
As AI becomes embedded in modern applications, traditional security approaches need to evolve. Invicti extends application security testing to cover AI-powered systems, helping organisations identify and secure vulnerabilities specific to large language models (LLMs).
For example, AI applications can be tested for risks such as prompt injection, where malicious inputs manipulate system behaviour, or command injection, where attackers attempt to execute unauthorised actions. Other vulnerabilities include insecure handling of AI-generated outputs and server-side request forgery, where systems can be tricked into accessing unintended resources.
Invicti’s approach focuses on automatically detecting these issues across AI models and APIs, ensuring that AI-powered applications are treated with the same level of scrutiny as any other critical system.
Read more about Invicti in our Case Studies.
Building a Framework for Safe AI Adoption
To manage AI effectively at scale, organisations should align with established frameworks such as the AI Risk Management Framework from the National Institute of Standards and Technology, ISO standards for AI management systems, and emerging regulatory requirements like the EU AI Act.
A strong governance model typically follows a continuous cycle: identifying AI use cases, assessing risks, defining controls, monitoring performance, and continuously improving.
This approach ensures that AI adoption is not only innovative but also sustainable and secure.
The Bottom Line: Visibility Is Everything
Shadow AI is not a distant or future concern; it is already present in most organisations today. The speed of AI adoption has created a gap between innovation and control, and that gap is where risk lives.
Shadow AI becomes dangerous when it operates outside of security oversight. AppSec is how you bring it back under control.
By extending traditional application security principles to visibility, testing, validation, and monitoring into AI systems, organisations can continue to innovate while staying secure.
Contact us if you want to have a free consultation or to discuss Invicti and their offerings.