The friction that used to protect people is gone
A few years back, faking a convincing nude photo took real effort — a decent GPU, patience with clunky software, and tolerance for rough results. That effort acted as a natural filter, keeping out anyone who wasn't seriously motivated. That filter no longer exists.
Today's open image-generation models produce photorealistic output on ordinary consumer hardware. Community model repositories host checkpoints downloaded well over a million times each, and explicit-content variants built on popular base models have been fetched hundreds of thousands of times. Layered on top are countless small "fine-tune" files, built by hobbyists, that push output toward a specific look, body type, or even a specific real person. Some of the most widely used base models were trained on millions of images and ship with no built-in content restrictions whatsoever. None of this requires exotic hardware anymore — a mid-tier gaming laptop from a few years ago is enough.
Then there's the packaged version: so-called "nudify" apps that hide all of that machinery behind one button. Upload a clothed photo, wait seconds, get back a fabricated nude. Recent transparency reporting counted dozens of these apps across major app stores, with combined downloads in the hundreds of millions. Strip away the slick interface and the payment wall, and it's the same open-source diffusion technology anyone could run at home.
Open weights mean there's no off switch
This is the part that gets glossed over in most hand-wringing think pieces: these models are open weights. Once released, a model's parameters exist as a file — and that file is now sitting on tens of thousands of hard drives worldwide. You can't recall it. You can't push a patch to remove it remotely. There's no central server the requests pass through, no terms of service binding the person running it, no company that could even see what's being generated, let alone stop it.
Most discussions of regulating AI-generated pornography assume a chokepoint exists — the way a platform can flip a switch and make a category of content vanish. Open-weight models were built without that chokepoint. Generation happens entirely on someone's own device, offline if they choose, invisible to any outside observer. I can run these models on a laptop with no internet connection at all. So can anyone else.
That's the same property that makes local models genuinely valuable for legitimate work — no per-request billing, no dependence on a vendor that might shut down or change terms, real privacy for sensitive data. The technology doesn't distinguish between a developer protecting client code and a stranger fabricating a nonconsensual image. You can't have the beneficial use case without also enabling the harmful one. That isn't a bug someone forgot to patch. It's an inherent feature of how the technology works.
The people being targeted are mostly kids, not celebrities
Public attention to deepfakes has largely focused on celebrities and politicians, which makes the problem feel distant and rare. The actual pattern is far more common and far closer to home.
Surveys of teachers in England found that a meaningful share already know of a student using these tools to create fake explicit images of a classmate. Other research suggests a large majority of deepfake pornography circulating in schools targets children under fourteen — some as young as eleven. In one widely reported case at a Pennsylvania high school, a single student generated fake explicit images of five classmates using ordinary photos pulled from social media. Broader teen surveys suggest a significant fraction of American adolescents personally know someone targeted this way while still a minor.
The child-safety dimension is what should settle any remaining debate about whether this is urgent. The organization that runs the main U.S. reporting system for child sexual abuse material documented an enormous year-over-year surge in reports involving AI-generated content. Leaked data from at least one overseas "nudify" service reportedly contained tens of thousands of generated images stored alongside the prompts used to create them. This isn't a hypothetical risk anymore — it's a documented pattern of what has already occurred.
What's hardest to sit with is how ordinary the perpetrators often are: not sophisticated actors, just a bored teenager with a phone and an app that, until recently, was easy to find in mainstream app stores. The barrier that used to make this hard is exactly the barrier that disappeared.
Detection is a race the defenders keep losing
The comforting counterargument is that if fakes get convincing enough, detection tools will keep pace. The evidence suggests otherwise — it's a treadmill, not a finish line.
No detector holds a stable accuracy record for long, because performance degrades the moment conditions shift. A model that performs well on a clean benchmark dataset often performs noticeably worse on real-world images — which get compressed, cropped, screenshotted, and re-shared, each step erasing the subtle artifacts detectors rely on. Worse, it's an adversarial loop by design: researchers publish detection methods, people building generative models read those papers, and the next generation of models is tuned to eliminate exactly the signatures that gave them away.
There's a structural reason defenders stay behind. A generator only needs to fool a detector once for the harm to land. A detector needs to be right nearly all the time, across every model and every way an image might be altered, without wrongly flagging real photos. Those aren't comparable burdens — one side needs a single success, the other needs near-perfect and continuous accuracy. Every honest assessment of the field reaches the same conclusion: detection never wins permanently, it only holds a temporary lead the next model release erases.
The law targets the wrong layer
None of this means nothing is being done. In the United States, the Take It Down Act — signed into law in 2025 — criminalizes publishing nonconsensual intimate images, including AI-generated fakes of real, identifiable people, and requires platforms to remove flagged material within 48 hours of a valid request. Platforms had until mid-2026 to build out compliant takedown systems, and the first conviction under the law has already occurred. Most U.S. states now have their own statutes addressing explicit deepfakes as well. That progress is genuine and worth acknowledging.
But look at what the law actually governs: publication and distribution — the moment content leaves a device and reaches a platform. That's a sensible place to apply pressure, since platforms remain a genuine chokepoint. What the law can't touch is generation itself. It doesn't reach a model file sitting on someone's laptop, or an image created and shared privately through a messaging app that never triggers a platform review, or a photo passed around a group chat. Takedown is a remedy that only activates after harm is already public. For a student whose classmates have already seen a fabricated image, a fast removal window doesn't undo what already happened.
That's not to say the law is useless — raising the cost of distributing this material matters. But treating takedown statutes as a solution mistakes the one layer that's regulated for the much larger layer that isn't. Generation happens beneath the reach of any takedown regime, and that's precisely the layer that has already scaled out of control.
Why "impossible to stop" is the honest answer
This isn't a call to give up — it's a call to diagnose the problem accurately, because the wrong diagnosis leads to the wrong response.
You can't un-release a model that's already been copied to thousands of machines. You can't make photorealistic generation difficult again. You can't build a detector that stays permanently ahead, and you can't write a law that reaches computation happening privately on someone's own hardware. Every intervention that genuinely helps operates downstream of generation itself: faster and more consistent takedowns, real legal consequences for distribution, app stores actually enforcing policies against apps built for this purpose, and schools having honest conversations with kids about this the way they eventually learned to address other harms that arrived through a screen. These are worth doing — but they're harm reduction, not prevention.
The essays warning that this technology is dangerous have the danger right. Where they go wrong is assuming there's still a cure available. The tools are already distributed across millions of devices. The barrier that once protected people is gone, and it isn't coming back. The uncomfortable truth is that the part everyone hoped could still be prevented already happened — quietly, while the public debate was still asking whether it was coming.
Responses
Sign in to leave a response.