AI’s efficiency and unrelenting work ethic has led to a business implementation race – but it’s important this haste does not cloak the risks the technology brings with it. OpenAI’s agent hacking its way out of a sandbox should serve as another reminder for businesses to carefully assess all possible consequences that could come from its use.
Businesses are increasingly trading privacy for convenience without fully realising it. As AI assistants become embedded across workplace software, external platforms and tools are starting to use and store proprietary data in ways they haven’t before.
Many businesses still assume that if a platform is encrypted, data remains private by default. But when AI is introduced into the workflow, that boundary can shift. Content that would normally sit within an end-to-end encrypted environment may be routed externally for processing and model training, with different rules coming into play around storage, logging and how data is used.
For business leaders, this is becoming a governance challenge without a clear playbook. Commercially sensitive information is increasingly being fed into AI systems without full visibility over where it goes or how it is handled.
Murky waters
The issue is less about whether AI tools are unsafe, and more about transparency. Many platforms still do not clearly explain when encryption changes, how long data is retained, or whether inputs are used to improve models.
New AI integrations are regularly shipped without clear advice for businesses about changes to data use. Any changes to the privacy model take a back seat to sales messaging about the efficiency benefits.
The first concern for businesses is whether the data collected by AI is stored externally for the purpose of training further models; which is a very common practice. If so, the concern is that confidential data, research and private employee information could potentially be imbibed by new models and regurgitated to competitors or anyone who asks the right questions.
The second murky issue for businesses is around consent and personal privacy. If workplace software has multiple users (most do) and allows a new AI feature to monitor and use conversation data, this potentially breaches the privacy of users who have not given permission for their data to be used in that way.
There is a third issue which sits squarely with providers and is yet to be meaningfully addressed. With 73% of the public using AI in their day-to-day life, the technology has access to a huge potential database of specific information about people’s emotions, desires, and tendencies. AI also has the capability to corral, analyse, and report on this data at scale. Developers like OpenAI are sitting on market research goldmines, but their privacy responsibilities in this regard remain unclear.
The risks
Much of the responsibility for this lack of transparency sits with platforms and regulators. But if a company inadvertently supplies client data to an external AI model and that information is subsequently exposed or accessed by a third party, the consequences will ultimately fall on the business. That makes it essential to understand how the AI-enabled software it uses accesses, stores and handles company data.
There’s also the growing risk of personal AI use among employees. You may have read about the recent data breach at Community Bank, where an employee used ‘unauthorised AI software’ to process customer data, which resulted in that information being exposed.
Without clear education on the risks, employees are likely to divert to external AI applications for quick answers, such as firing a question into ChatGPT between meetings or running a client document through Claude on their home computer. As Community Bank found, this risks the inadvertent release of sensitive information outside the managed parameters of the company tech stack.
What’s most concerning about the use of AI is that mistakes can be incredibly difficult to remedy because of the scale at which AI works. Take, for example, an AI agent established to screen job applications. The advantage is that AI can go through thousands of CVs in a day, where a sole employee might only manage a hundred. But, if the instructions to the AI are not set right, and biases or unauthorised data access is discovered even just a few days later, the damage can be significant.
To head this off at the pass, it’s extremely important that businesses do their due diligence with each use case and instruct the AI on exactly what it should not do. To properly ringfence, AI needs negative reinforcement to go with the positive.
How businesses can protect themselves
Businesses need to take a methodical approach to AI integration and clearly understand the implications of every AI addon in the services they use.
It is the responsibility of tech teams and CTOs to survey all uses of AI and question what kind of data is collected, how it is stored and how long for, how it is used, and then whether the necessary permissions have been obtained from employees or clients that are using it.
There is a compliance motivation for this too. The regulation of AI is still evolving, but existing laws do apply, such as business obligations under GDPR and the EU AI Act. And as the murky waters get clearer, it’s likely regulators will begin to set further privacy stipulations. It is important businesses get ahead of that curve.
