SecTor 2025 | What Happens When Your Digital Voice Clone Goes Rogue

“Speak for Me” was envisioned as a Windows accessibility feature designed to replicate a user’s voice with just a few samples, storing it locally as an AI model trained on the user’s voice. This innovative feature aimed to enhance the existing Text-To-Speech interface, offering capabilities such as creating a virtual microphone for seamless use in conferencing apps like Microsoft Teams. Our team performed an internal security audit of this feature, revealing that it is very hard to protect. The potential attacks spanned across multiple vectors. Ultimately, our audit led to this feature being released with Custom Neural Voices (CNV) Azure service only. In this session, we will walk you through the various attack scenarios and vulnerabilities found, showcasing the difficulties of protecting AI based user voices on client devices.

We will start our presentation with a number of critical vulnerabilities discovered in the project. These include classical remote code execution on the victims’ machines, but more interestingly, either directly stealing the model itself, or abusing the cloud infrastructure to obtain a model of arbitrary persona. Both client and web side of the app had multiple defensive mechanisms such as consent voice recording, model encryption, watermarking embedded into voice samples and others that were supposed to prevent the infrastructure from being abused to produce deepfakes by bad actors. All of these could easily be bypassed and ultimately, the attacker could gain the ability to impersonate a victim with relatively low effort.

This project will serve as a case study to demonstrate the challenges and vulnerabilities of AI security on devices, particularly on generic Windows platforms that were not designed to protect highly sensitive AI models. We will examine the current state of the Windows security ecosystem and its relevance to AI model security.

By: Andrey Markovytch | Senior Security Researcher, Microsoft

Presentation Materials Available at:
https://blackhat.com/sector/2025/briefings/schedule/?#what-happens-when-your-digital-voice-clone-goes-rogue-47422Black HatRead More