Safety & Security

'My family would kill me if they saw it': why moderation AI misses deadly photos

Fully clothed selfies are being used to blackmail and endanger Pakistani women. Chayn's report says moderation AI trained to spot nudity misses images that can be deadly without it.

‘My family would kill me if they saw it’: the ‘harmless’ images that can be deadly for women in Pakistan
‘My family would kill me if they saw it’: the ‘harmless’ images that can be deadly for women in PakistanLoboStudioHamburg / pixabay
By James Calloway7 min read

Why it matters

  • Chayn's report 'Explicit Harms of Non-Explicit Images' urges platforms to judge consent rather than content in moderation.
  • Pakistan's Digital Rights Foundation helpline receives approximately 250 complaints a month.
  • Of Pakistan's estimated 100 million TikTok users, more than 70% are male; two-thirds of Instagram users are male.
  • 14-year-old Hira Anwar and TikTok user Sana Yousaf were both murdered last year in Pakistan over online content.
  • Meta did not respond to a request for comment on the report.

A Pakistani teenager's fully clothed selfie was ruled harmless by a social media platform, even though she says her father would kill her if he ever saw it. Maniza*, who is from a conservative town and never leaves home without her hijab, took a photo of herself at home in jeans and a vest top. The young man she trusted with it used it to threaten, blackmail and impersonate her. When she reported the fake account, the platform told her the image did not break any rules.

"They didn't understand my family would kill me if they saw it," she said, describing the platform's response to the impersonation account built around her photo.

The case sits at the center of a new report by Chayn, a charity that tackles gender-based violence, titled Explicit Harms of Non-Explicit Images. The report's core argument is aimed squarely at how platforms — and the automated systems that do most of their moderation — decide what counts as harmful. Chayn wants tech companies to shift from judging content to judging consent.

The stakes are not hypothetical. This is why the story matters for anyone following AI moderation: automated content moderation, now the default filter for billions of posts, is trained overwhelmingly to detect specific things — nudity most of all. Where harm depends on local cultural context rather than pixels, those systems return false negatives with potentially lethal consequences.

What happened to Maniza?

Maniza's account, as told in the report's accompanying reporting, traces a now-familiar extortion pattern. A man she liked asked for a photo of her hair, which she wore down to her waist, and promised to delete it. He then escalated.

"He started getting pushy about meeting up. He threatened to send the photo to my family. He said if I let him touch me he will delete the picture," she said.

He then created a fake account using her image, captioned "love of my life." Her report to the platform failed. Her mother, convinced that Maniza's father would kill them both, pulled her out of school. Maniza says she now faces forced marriage as soon as a proposal arrives, and that the photo remains online.

"That picture is still out there and there's nothing I can do," she said.

The pattern is extortion via context, not via nudity. No moderation system flagged it because nothing in the image itself violates a Western-defined rulebook.

Why can't automated moderation catch this?

Chayn's chief executive, Hera Hussain, puts the problem in one sentence: "There's a gap in what platforms define as harmful and what women experience as harmful."

Her critique of the technology is specific. "Automated content moderation systems are trained to spot nudity, which is an important aspect of tackling image-based abuse, but misses that images can be deadly without nudity," she said.

The report urges platforms to build a consent-based reporting pathway — one that evaluates whether the subject of an image agreed to its distribution, rather than whether the image contains skin. That is a materially different technical and policy problem, and Chayn argues it is the only approach that works across cultures.

Hussain's framing of the current system is blunt: "Otherwise it's like someone throwing a bomb and focusing on what type of bomb it is instead of the damage it can do."

The gap is not speculative. Advocates say the global debate on intimate image abuse is viewed through a Western lens, excluding women in the global south whose families treat modesty itself — a photo of uncovered hair, an outlined body — as grounds for violence. A caveat is due here: the cases cited come through Chayn and the Digital Rights Foundation, organizations with an explicit advocacy position, though their casework is corroborated by Pakistan's homicide record.

How large is the problem in Pakistan?

The numbers available sketch a stark gender divide. Two-thirds of Instagram users in Pakistan are male. Of the country's estimated 100 million TikTok users, more than 70% are male.

The Digital Rights Foundation, a Pakistani digital rights organization founded by lawyer Nighat Dad, runs a helpline that receives approximately 250 complaints a month. In one recent case, the helpline succeeded in getting Instagram to remove AI-generated videos depicting a teacher kissing a man — a rare win, and an example of synthetic media compounding the problem.

The consequences extend beyond distress. Nighat Dad summarized the range: "The consequences can be devastating, particularly in contexts like Pakistan where women's sexuality, or even simply their presence online, is heavily policed and intimate imagery carries enormous social stigma."

Hussain lists what the report documents: women "taken out of education and jobs, pressured or forced into marriage, [subjected to] verbal and physical assault and divorce, as well as in extreme cases, suicide, murder and femicide."

What does a 'non-explicit' case look like?

Zarghuna, a 15-year-old from Balochistan, described a photo taken of her wading in a lake during a school picnic — fully dressed. The male student who took it later projected the image onto a large screen at a school event.

"My wet clothes were clinging to my body. Even if you're wearing a burqa, if it outlines your body, people will say you are making an exhibition," she said.

She described how such images become leverage: "Sometimes images are taken of my friends in our classroom and a WhatsApp comes asking for sexual favours, saying: 'I have your image, I can make it public.'"

Her analysis matches the report's thesis: "It's a simple image but it is harmful because you don't know what the person who has it will do with it. They can get money, sexual favours or unwanted relationships from the fear of it."

Has this led to killings?

Yes, and recently. Fourteen-year-old Hira Anwar was lured from the US to Pakistan and killed by her father last year over her TikTok videos. Sana Yousaf, another teenage TikTok user, was murdered last year by a man who had become obsessed with her online account.

These are the extreme endpoints of the same spectrum the report documents. Platform policy failure and physical violence are connected, the authors argue, because an unremoved image remains a live weapon in an extortioner's hands.

Dad's policy critique goes to the architecture of moderation itself: "Platform policies are often designed without sufficient understanding of cultural contexts and the ways gendered harm manifests differently across communities."

She added: "We repeatedly see cases where something may not fit a platform's narrow definition of intimate or sexual content, but in the survivor's context it is clearly being used to sexualise, shame or endanger her."

What are platforms being asked to change?

Chayn's demands are two-fold:

  • Bring cultural context into content moderation, so that reviewers and automated systems can weigh how an image functions in a survivor's community rather than only what it depicts.
  • Introduce a consent-based reporting pathway, letting users flag images distributed without their permission regardless of whether the content contains nudity.

Both demands would require retraining or redesigning the automated moderation layer, not just adding human reviewers. Neither Meta nor other named platforms responded to the report's specific proposals; Meta did not respond to a request for comment on this article.

That silence is itself part of the story. For Maniza, the policy gap has already produced its outcome: lost schooling, a pending forced marriage, and a photo that no system will take down. "Every day I am living in fear," she said.

The report's release puts the burden back on platforms to prove their moderation systems can recognize harm that does not show skin — and until they do, women like Maniza are left reporting threats to systems that keep telling them nothing is wrong.

* Name has been changed.

Source: theguardian.com

Share this article:

More from James Calloway

James Calloway

Show full bio

News editor covering people, launches and day-to-day practice in artificial intelligence and machine learning at AI In Context.

722 articles

Related articles

  1. Test accounts posing as 13-year-olds downloaded 135 'nudify' apps
  2. UK weighs banning under-16s from reselling apps over sexualised messages
  3. Meta Lets Its AI Sext While Adult Creators Fight Restrictions
  4. Apple Tells App Store Developers: Filter AI Nudify Content or Get Removed
  5. Two major chatbots strip hijabs when asked; Claude refused

« Previous article