Services
Security monitoring Risk services Security Advisory
Inspiration
About us
Working at defenced Jobs Sustainability
contact
Keep AI chatbots safe

The promise and fragility of intelligent AI chatbots

Written by: Brian Kok

AI-powered chatbots are rapidly gaining popularity. They are increasingly used in customer service, internal communications, and as digital assistants capable of handling a wide range of tasks. The benefits are clear: speed, scalability, and ease of use.

Over ons Blog1

CHATBOT MANIPULATION
At the same time, recent real-world cases show that there are also risks associated with using AI chatbots. For example, the Dutch Data Protection Authority has warned that entering sensitive information into chatbots may lead to data breaches. In addition, research has shown that hackers can trick chatbots into disclosing confidential data. These examples highlight that while AI offers many opportunities, it also introduces vulnerabilities that are often overlooked.

ACCESS TO BUSINESS INFORMATION
A recent example of this is the hack of the AI-powered Salesloft chatbot. In that case, it was demonstrated that malicious actors were able to manipulate the chatbot to gain unauthorized access to company information. This incident highlights the importance for organizations to not only recognize the benefits of AI chatbots, but also to take their security risks seriously and implement appropriate protective measures.

The risks of smart AI chatbots

In this article, we explore the key risks of AI chatbots and why organizations adopting such technology need to remain vigilant. Below is an overview of common risks within AI applications:.

Common risks

Prompt injections
Leakage of initial instructions
Unintended information disclosure
The danger of shared context
Integrations with other systems
Misuse of AI for undesired activities

PROMPT INJECTIONS

AI models respond to instructions (prompts). An attacker can manipulate these instructions through so-called prompt injections. In such cases, the chatbot is tricked into performing actions that should not be allowed. For example, executing commands on underlying systems or escaping into other systems or networks. This can lead to data leaks or even full access to the underlying system.

CONCLUSION

AI chatbots hold enormous potential, but they also introduce new risks. Organizations should not only consider the advantages but also invest in security measures.

Examples include:

  • limiting access rights when integrating with other systems;

  • isolating chat sessions through sandboxing;

  • strictly controlling which data may be shared;

  • preventing harmful output;

  • monitoring for misuse attempts;

  • conducting penetration testing on the chatbot.

01

SOURCES

genai.owasp.org

autoriteitpersoonsgegevens.nl

bitdefender.com

krebsonsecurity.com

020 023
Want to know more?

Schedule your session with a cyber security

Step 1
You choose when to meet, and we’ll adapt to your schedule.

Step 2
We’ll check if you have any specific questions to ensure you get the most out of your session.

Step 3
During an initial online introductory meeting, we’ll discuss new cyber security perspectives.

Schedule now