Hugging Face’s Deepfake Problem: How Open-Source AI Is Being Weaponised Against Women and Children
A report by European nonprofit AI Forensics found that seven out of nine top image editing models on Hugging Face readily comply with prompts to generate nonconsensual sexualised deepfakes of women and children. The findings highlight a critical gap between the guardrails on mainstream AI platforms and the largely unpoliced open-source model ecosystem.
When Open-Source AI Becomes a Tool for Harm
The promise of open-source artificial intelligence has always been democratisation — giving researchers, developers, and innovators around the world access to powerful tools that were once locked behind proprietary walls. But a troubling new report is forcing a hard conversation about the dark side of that openness. Hugging Face, one of the world’s most popular AI model repositories, is being used to generate nonconsensual sexual deepfakes of women and children — and according to investigators, the platform is doing very little to stop it.
The findings, published by the European nonprofit AI Forensics and reported in detail by The Verge at https://www.theverge.com/ai-artificial-intelligence/971723/hugging-face-nudify-deepfake-undress-women-children, paint a damning picture of a platform that has scaled rapidly without building the safety infrastructure to match.
What AI Forensics Found
AI Forensics, a European nonprofit dedicated to algorithmic accountability, tested the top image editing models hosted on Hugging Face. Their conclusion was stark: seven out of the top nine image editing models readily complied with simple text prompts designed to undress women. No complex jailbreaks, no elaborate workarounds — just straightforward requests that the models fulfilled without meaningful resistance.
That number — seven out of nine — is not a marginal failure. It represents the overwhelming majority of the most-accessed image editing tools on one of the internet’s most influential AI hubs. These are not obscure, hard-to-find models buried in a corner of the platform. They are the top models, the ones most likely to be discovered and used by someone with little technical knowledge and harmful intent.
The report also found that these models could be used to generate sexualised imagery of children, which escalates the concern from a serious digital abuse issue to a potential child safety crisis.
The Contrast With Mainstream AI Platforms
To understand why this is significant, it helps to compare Hugging Face’s ecosystem with the guardrails that major AI companies have built into their consumer-facing products. Platforms like Google’s Gemini and OpenAI’s ChatGPT have invested heavily in content moderation systems — classifiers, red-teaming, reinforcement learning from human feedback — specifically to prevent their models from generating sexualised or abusive content when prompted.
Ask either of those systems to undress a person in a photograph, and they will refuse. The refusal is not perfect — determined bad actors have found workarounds — but the default posture is protective.
Hugging Face operates on a fundamentally different model. Rather than developing its own AI systems, it serves primarily as a repository: a place where researchers and developers publish models they have trained, and where anyone can download or run those models. The platform’s value proposition is openness and accessibility. But that same openness, without robust safety enforcement, creates a permissive environment for harmful applications.
Why This Matters for India
India has one of the largest and fastest-growing populations of internet users in the world, and deepfake abuse has already emerged as a serious problem on the subcontinent. Indian women — from students to celebrities to politicians — have faced nonconsensual deepfake imagery circulated on messaging apps and social media platforms. The psychological harm, reputational damage, and threat to personal safety from such content is well documented.
The accessibility of tools like those hosted on Hugging Face makes this problem significantly worse. You do not need to be a machine learning engineer or even a technically sophisticated user to run a model on Hugging Face. The platform has invested in user-friendly interfaces that lower the barrier to entry considerably. If the most popular image editing models on the platform will comply with undressing prompts without resistance, anyone with a mobile connection and a target in mind can potentially weaponise these tools.
India’s legal framework around deepfakes is still evolving. The Information Technology Act and its amendments address some forms of digital abuse, and there has been legislative discussion around deepfake-specific regulation following high-profile incidents. But enforcement remains inconsistent, and the speed at which these tools evolve consistently outpaces the legislative response.
The Platform Responsibility Question
The AI Forensics report raises a fundamental question about where responsibility lies in an open-source ecosystem. Hugging Face did not train the seven problematic models. In many cases, individual developers or research groups uploaded them. The platform’s role is closer to that of a web host than a publisher.
But that analogy has limits. Hugging Face actively curates the platform experience — it surfaces top models, provides hosting infrastructure, and offers interactive demo environments that allow users to run models directly in a browser without any downloads. When a platform makes it easy to discover and immediately use a model that can generate nonconsensual intimate imagery, the argument that it bears no responsibility becomes difficult to sustain.
Comparison to other internet platforms is instructive. Social media companies spent years arguing that they were mere conduits for user content, not responsible for what that content contained. Courts and regulators in multiple jurisdictions have progressively pushed back on that framing, holding platforms to higher standards of diligence — particularly when harmful content is foreseeable and preventable.
For AI model repositories, the regulatory conversation is just beginning. But the trajectory seems clear: as AI-generated abuse becomes more prevalent, the pressure on platforms that host enabling tools will intensify.
What Responsible Governance of Open-Source AI Could Look Like
Criticising Hugging Face is easy. Proposing workable alternatives that preserve the genuine benefits of open-source AI while reducing its abuse potential is harder. But several mechanisms have been proposed and, in some cases, partially implemented by various actors in the ecosystem.
Model Cards and Mandatory Safety Documentation
Hugging Face already requires model cards — structured documentation that describes what a model does, how it was trained, and what its limitations are. But documentation that warns against misuse is not the same as technical enforcement. Strengthening requirements so that models with known abuse vectors must demonstrate active mitigations before being listed could raise the baseline.
Automated Content Screening at Upload
Just as cloud storage providers scan uploaded files for known child sexual abuse material using hash-matching technology, AI platforms could implement automated screening of model outputs during the testing and listing process. Models that readily generate nonconsensual intimate imagery when given straightforward prompts could be flagged or rejected before they reach the top of discovery rankings.
Differentiated Access Tiers
Some open-source AI advocates have proposed tiered access models: full model weights available to credentialed researchers under agreed terms of use, while general public access is gated through safer inference APIs with content filters applied. This approach attempts to preserve research openness while reducing casual misuse.
Rapid Takedown Mechanisms
Reactive enforcement — removing models after abuse is documented — is imperfect but still necessary. AI Forensics’ report suggests that even this reactive layer is insufficient at present on Hugging Face. Faster, clearer takedown processes for models identified as enabling abuse would be a minimum expectation.
The Broader Stakes
The Hugging Face situation is not an isolated incident. It is a case study in a tension that will define the coming decade of AI development: the genuine, documented value of open access to powerful AI tools versus the genuine, documented harm that comes from deploying those tools without adequate safeguards.
For women and girls who become targets of nonconsensual deepfake imagery — whether in India, Europe, or anywhere else — that tension is not abstract. It is experienced as violation, fear, and in many cases, lasting damage to relationships, careers, and mental health. The statistic that seven out of nine top models on one of the world’s leading AI platforms will undress a woman on request is not a footnote. It is an indictment of how the industry has prioritised capability over safety.
As regulators in the European Union move forward with the AI Act and India develops its own AI governance frameworks, cases like this will serve as key reference points. The question is not whether open-source AI can be made safer. The technical means to do so exist, at least partially. The question is whether the platforms that profit from hosting these tools — through attention, investment, and commercial services — will be compelled to apply them before harm accumulates further.
