Technology
ChatGPT Found Porn in My Family Photo
I asked it to make a Sopranos-style family poster. It complied. Then I asked it to fix the faces. Apparently, that was where civilization became endangered.
There are moments in the history of technology when you suddenly realize you are living in the future.
This was not one of them.
I wanted a family picture.
Not pornography. Not instructions for enriching uranium. Not a deepfake of the President robbing a liquor store.
A family picture.
The idea was simple enough to explain to a moderately intelligent Labrador.
Take the visual language of a prestige television crime-family poster: dark background, dramatic lighting, one central figure, relatives and friends arranged behind him. Replace the fictional cast with photographs of real adults from my family and social circle.
Me.
My wife.
My mother.
My adult son and daughter.
A few old friends.
And an AI-generated character representing my resident digital psychologist, because apparently my life was not complicated enough already.
Everyone clothed.
Everyone standing around looking vaguely as though somebody at the table knew where a body was buried.
That was it.
ChatGPT understood.
Or appeared to.
It generated the picture.
And, to be fair, it wasn't terrible.
There was only one problem.
It had apparently concluded that marriage is a sufficiently important institution that my wife deserved to appear **twice**.
So I pointed this out.
ChatGPT agreed.
It explained the error.
We tried again.
Now some of my friends looked as though they had been replaced overnight by men recruited from a convention of generic Eastern European nightclub security guards.
We corrected that.
Another attempt.
Better composition. Wrong people.
Another attempt.
Wrong face in the central position.
Another correction.
And this is where our journey through the technological singularity became interesting.
The machine that had already generated essentially the same image several times suddenly decided that the request might violate its content policies.
Nothing sexual had been added.
Nobody had removed any clothing.
Nobody had requested violence.
Nobody had even requested a particularly adventurous neckline.
We were fixing **faces**.
The system refused.
So the request was reformulated.
Refused again.
And finally, after repeatedly generating variations of a fully clothed family portrait, the safety system produced its masterpiece.
It warned that the request might violate rules concerning:
**“nudity, sexual or erotic content.”**
There was no nudity.
There was no sexual content.
There was no erotic content.
There wasn't even a swimsuit.
Somewhere inside one of the most sophisticated consumer artificial-intelligence products ever built, a safety mechanism had looked at a bunch of middle-aged people wearing suits and concluded:
**Porn.**
And that is when the problem stopped being funny.
Well, almost.
It remained extremely funny.
But underneath the comedy is a serious product-design failure.
A Safety System With a Veto and No Idea Why
False positives happen.
Spam filters occasionally eat legitimate email. Fraud systems occasionally decline legitimate credit cards. Computer vision occasionally mistakes one object for another.
No serious person expects machine-learning systems to achieve perfection.
But there is an important difference between an ordinary classifier making an occasional mistake and a classifier being granted **absolute authority over the product while behaving inconsistently from one attempt to the next**.
The same underlying task had already been accepted.
The system had already generated the images.
The user then requested corrections to the generated output.
The task did not become more sexual.
It did not become more violent.
It did not become more dangerous.
It became more specific.
And specificity somehow triggered prohibition.
This creates a remarkable user experience:
**Yes.
Yes.
Yes.
Oops, wrong face.
Can you fix it?
No. Pornography.**
That isn't safety.
That's a broken classifier with veto power.
The Worst Kind of Rule Is One You Cannot Learn
People can work around restrictive systems.
People can work around permissive systems.
What people cannot efficiently work around are **unpredictable systems**.
If a product says, “We don't generate this category of image,” fine. You may dislike the rule, but at least you understand the boundary.
If it says, “We allow this,” also fine.
But when the same product effectively says:
*This is permitted.*
*This is permitted.*
*This is permitted.*
*Actually, this is pornography.*
—the user cannot learn the rules because apparently **the machine hasn't learned them either**.
That is more damaging than a simple refusal.
It destroys trust.
Every future rejection becomes suspect.
Was the request genuinely outside the rules?
Did a classifier hallucinate something?
Did one word trip a heuristic?
Did the presence of multiple reference photographs confuse another model?
Would submitting precisely the same request five minutes later succeed?
The user has no idea.
Eventually the warning stops functioning as a warning.
It becomes a slot machine.
Safety Theater Has a Cost
There is a fashionable assumption in technology that more guardrails automatically mean a safer product.
They don't.
Bad guardrails can make a product simultaneously **less useful and less credible**.
A smoke detector that screams every time somebody makes toast does not create an extraordinarily safe kitchen.
Eventually somebody removes the battery.
The same principle applies here.
A moderation system earns authority by being accurate enough that users take its interventions seriously.
If it labels harmless requests as sexual content, users learn something very different from what its designers intended.
They don't learn:
*The system is keeping me safe.*
They learn:
*The system is stupid.*
And once that lesson has been taught often enough, the genuine warnings lose value too.
That is the part technology companies should worry about.
Not angry tweets.
Not sarcastic articles.
**Loss of epistemic authority.**
Because an AI assistant ultimately sells something much more fragile than intelligence.
It sells confidence that when it says *yes*, *no*, *I know*, *I don't know*, or *I can't do that*, those statements have some consistent relationship with reality.
Break that relationship often enough and the intelligence underneath becomes irrelevant.
The Family Is Still Fully Clothed
At the end of this grand technological adventure, nobody got naked.
Nobody became erotic.
Nobody threatened public safety.
The friends still needed better faces.
My wife still needed to stop reproducing spontaneously.
And the AI psychologist still needed to be put back into the photograph after the generator replaced her with somebody else's ex-wife.
Those were the actual technical problems.
The safety system solved none of them.
Instead, it discovered imaginary pornography and terminated the job.
Perhaps that is the perfect metaphor for a certain kind of modern technology company.
Build something astonishingly powerful.
Surround it with layers of systems designed to prevent every conceivable problem.
Then make those systems so frightened of conceivable problems that they begin inventing problems that aren't there.
Artificial intelligence was supposed to understand the world.
For now, I would settle for it understanding the difference between **a family portrait and porn**.