404 Media

2025-10-24 20:11:10

AI Dataset for Detecting Nudity Contained Child Sexual Abuse Images

A large image dataset used to develop AI tools for detecting nudity contains a number of images of child sexual abuse material (CSAM), according to the Canadian Centre for Child Protection (C3P).

The NudeNet dataset, which contains more than 700,000 images scraped from the internet, was used to train an AI image classifier which could automatically detect nudity in an image. C3P found that more than 250 academic works either cited or used the NudeNet dataset since it was available download from Academic Torrents, a platform for sharing research data, in June 2019.

“A non-exhaustive review of 50 of these academic projects found 13 made use of the NudeNet data set, and 29 relied on the NudeNet classifier or model,” C3P said in its announcement.

C3P found more than 120 images of identified or known victims of CSAM in the dataset, including nearly 70 images focused on the genital or anal area of children who are confirmed or appear to be pre-pubescent. “In some cases, images depicting sexual or abusive acts involving children and teenagers such as fellatio or penile-vaginal penetration,” C3P said.

People and organizations that downloaded the dataset would have no way of knowing it contained CSAM unless they went looking for it, and most likely they did not, but having those images on their machines would be technically criminal.

“CSAM is illegal and hosting and distributing creates huge liabilities for the creators and researchers. There is also a larger ethical issue here in that the victims in these images have almost certainly not consented to have these images distributed and used in training,” Hany Farid, a professor at UC Berkeley and one of the world’s leading experts on digitally manipulated images, told me in an email. Farid also developed PhotoDNA, a widely used image-identification and content filtering tool. “Even if the ends are noble, they don’t justify the means in this case.”

“Many of the AI models used to support features in applications and research initiatives have been trained on data that has been collected indiscriminately or in ethically questionable ways. This lack of due diligence has led to the appearance of known child sexual abuse and exploitation material in these types of datasets, something that is largely preventable,” Lloyd Richardson, C3P's director of technology, said.

Academic Torrents removed the dataset after C3P issued a removal notice to its administrators.

"In operating Canada's national tipline for reporting the sexual exploitation of children we receive information or tips from members of the public on a daily basis," Richardson told me in an email. "In the case of the NudeNet image dataset, an individual flagged concerns about the possibility of the dataset containing CSAM, which prompted us to look into it more closely."

C3P’s findings are similar to 2023 research from Stanford University’s Cyber Policy Center, which found that LAION-5B, one of the largest datasets powering AI-generated images, also contained CSAM. The organization that manages LAION-5B removed it from the internet following that report and only shared it again once it had removed the offending images.

"These image datasets, which have typically not been vetted, are promoted and distributed online for hundreds of researchers, companies, and hobbyists to use, sometimes for commercial pursuits," Richardson told me. "By this point, few are considering the possible harm or exploitation that may underpin their products. We also can’t forget that many of these images are themselves evidence of child sexual abuse crimes. In the rush for innovation, we’re seeing a great deal of collateral damage, but many are simply not acknowledging it — ultimately, I think we have an obligation to develop AI technology in responsible and ethical ways."

Update: This story has been updated with comment from Lloyd Richardson.

Largest Dataset Powering AI Images Removed After Discovery of Child Sexual Abuse Material

The model is a massive part of the AI-ecosystem, used by Stable Diffusion and other major generative AI products.

^{Samantha Cole (404 Media)}

404 Media

2025-10-24 17:55:27

a16z-Backed Startup Sells Thousands of ‘Synthetic Influencers’ to Manipulate Social Media as a Service

A new startup backed by one of the biggest venture capital firms in silicon valley, Andreessen Horowitz (a16z), is building a service that allows clients to “orchestrate actions on thousands of social accounts through both bulk content creation and deployment.” Essentially, the startup, called Doublespeed, is pitching an astroturfing AI-powered bot service, which is in clear violation of policies for all major social media platforms.

“Our deployment layer mimics natural user interaction on physical devices to get our content to appear human to the algorithims [sic],” the company’s site says. Doublespeed did not respond to a request for comment, so we don’t know exactly how its service works, but the company appears to be pitching a service designed to circumvent many of the methods social media platforms use to detect inauthentic behavior. It uses AI to generate social media accounts and posts, with a human doing 5 percent of “touch up” work at the end of the process.

On a podcast earlier this month, Doublespeed cofounder Zuhair Lakhani said that the company uses a “phone farm” to run AI-generated accounts on TikTok. So-called “click farms” often use hundreds of mobile phones to fake online engagement of reviews for the same reason. Lakhani said one Doublespeed client generated 4.7 million views in less than four weeks with just 15 of its AI-generated accounts.

“Our system analyzes what works to make the content smarter over time. The best performing content becomes the training data for what comes next,” Doublespeed’s site says. Doublespeed also says its service can create slightly different variations of the same video, saying “1 video, 100 ways.”

“Winners get cloned, not repeated. Take proven content and spawn variation. Different hooks, formats, lengths. Each unique enough to avoid suppression,” the site says.
One of Doublespeed's AI influencers
Doublespeed allows clients to use its dashboard for between $1,500 and $7,500 a month, with more expensive plans allowing them to generate more posts. At the $7,500 price, users can generate 3,000 posts a month.

The dashboard I was able to access for free shows users can generate videos and “carousels,” which is a slideshow of images that are commonly posted to Instagram and TikTok. The “Carousel” tab appears to show sample posts for different themes. One, called “Girs Selfcare” shows images of women traveling and eating at restaurants. Another, called “Christian Truths/Advice” shows images of women who don’t show their face and text that says things like “before you vent to your friend, have you spoken to the Holy Spirit? AHHHHHHHHH”

On the company’s official Discord, one Doublespeed staff member explained that the accounts the company deploys are “warmed up” on both iOS and Android, meaning the accounts have been at least slightly used, in order to make it seem like they are not bots or brand new accounts. Doublespeed cofounder Zuhair Lakhani also said on the Discord that users can target their posts to specific cities and that the service currently only targets TikTok but that it has internal demos for Instagram and Reddit. Lakhani said Doublespeed doesn’t support “political efforts.”

A Reddit spokesperson told me that Doublespeed’s service would violate its terms of service. TikTok, Meta, and X did not respond to a request for comment.

Lakhani said Doublespeed has raised $1 million from a16z as part of its “Speedrun” accelerator program “a fast‐paced, 12-week startup program that guides founders through every critical stage of their growth.”

Marc Andreessen, after whom half of Andreessen Horowitz is named, also sits on Meta’s board of directors. Meta did not immediately respond to our question about one of its board members backing a company that blatantly aims to violate its policy on “authentic identity representation.”

What Doublespeed is offering is not that different than some of the AI generation tools Jason has covered that produce a lot of the AI-slop flooding social media already. It’s also similar, but a more blatant version of an app I covered last year which aimed to use social media manipulation to “shape reality.” The difference here is that it has backing from one of the biggest VC firms in the world.

AI-Powered Social Media Manipulation App Promises to 'Shape Reality'

A prototype app called Impact describes “A Volunteer Fire Department For The Digital World,” which would summon real people to copy and paste AI-generated talking points on social media.

^{Emanuel Maiberg (404 Media)}

404 Media

2025-10-24 17:02:18

Behind the Blog: Making Things for Humans

This is Behind the Blog, where we share our behind-the-scenes thoughts about how a few of our top stories of the week came together. This week, we discuss Pavlovian Chartbeat response, when to say "cum," and the wave of making things for humans, by humans.

JOSEPH: Right now I’m in the midst of upgrading a bunch of my podcasting and related gear. I’m using the same kinda cheap to midrange web cam I got when we first launched 404 Media. My mic is fine but now that a fair number of people listen to the pod, and we want to grow it, it’s time to invest in some new tech. Jason has already done this, I’m more following his lead. I used to be very into cameras, tech, gadgets, mics, but it’s been a few years.

Because the four of us are spread all over, we can’t do the popular aesthetic of everyone sitting on a chair or sofa chatting in a podcast studio. That stuff obviously performs better on video/YouTube, which is where podcast discovery often happens now. But we can try to make our remotely recorded podcast look as good as we can. Give people something to actually look at.

This post is for subscribers only

Become a member to get access to all content
Subscribe now

Behind the Blog: Making Things for Humans

^{Samantha Cole (404 Media)}

404 Media

2025-10-24 15:31:52

DHS Tries To Unmask Ice Spotting Instagram Account by Claiming It Imports Merchandise

The Department of Homeland Security (DHS) is trying to force Meta to unmask the identity of the people behind Facebook and Instagram accounts that post about Immigration and Customs Enforcement (ICE) activity, arrests, and sightings by claiming the owners of the account are in violation of a law about the “importation of merchandise.” Lawyers fighting the case say the move is “wildly outside the scope of statutory authority,” and say that DHS has not even indicated what merchandise the accounts, called Montcowatch, are supposedly importing.

“There is no conceivable connection between the ‘MontCo Community Watch’ Facebook or Instagram accounts and the importation of any merchandise, nor is there any indicated on the face of the Summonses. DHS has no authority to issue these summonses,” lawyers with the American Civil Liberties Union (ACLU) wrote in a court filing this month. There is no indication on either the Instagram or Facebook account that the accounts are selling any type of merchandise, according to 404 Media’s review of the accounts. “The Summonses include no substantiating allegations nor any mention of a specific crime or potential customs violation that might trigger an inquiry under the cited statute,” the lawyers add.

💡
Do you know anything else about this case or others like it? We would love to hear from you. Using a non-work device, you can message Joseph securely on Signal at joseph.404 or Jason at jason.404

A judge temporarily blocked DHS from unmasking the owners last week.

“The court now orders Meta [...] not to produce any documents or information in response to the summonses at issue here without further order of the Court,” the judge wrote in a filing. The move to demand data from Meta about the identities of the accounts while citing a customs statute shows the lengths to which DHS is willing to go to attempt to shut down and identify people who are posting about ICE’s activities.

Montcowatch is, as the name implies, focused on ICE activity in Montgomery County, Pennsylvania. Its Instagram posts are usually titled “Montco ICE alert” and include details such as where suspected ICE agents and vehicles were spotted, where suspected agents made arrests, or information about people who were detained. “10/20/25 Eagleville,” one post starts. “Suspected dentention [sic] near Ollies on Ridge Pike sometime before 7:50 am. 3 Agents and 3 Vehicles were observed.”

The Instagram account has been posting since June, and also posts information about peoples’ legal rights to film law enforcement. It also tells people to not intervene or block ICE. None of the posts currently available on the Instagram account could reasonably be described as doxing or harassing ICE officials.

On September 11, DHS demanded Meta provide identifying details on the owners of the Montcowatch accounts, according to court records. That includes IP addresses used to access the account, phone numbers on file, and email addresses, the court records add. DHS cited a law “focused on customs investigations relating to merchandise,” according to a filing from the ACLU that pushed to have the demands thrown out.
playlist.megaphone.fm?p=TBIEA2…
“The statute at issue here, 19 U.S.C. § 1509, confers limited authority to DHS in customs investigations to seek records related to the importation of merchandise, including the assessment of customs duties,” the ACLU wrote. “Identifying anonymous social media users critical of DHS is not a legitimate purpose, and it is not relevant to customs enforcement.” As the ACLU notes, a cursory look at the accounts shows they are “not engaged in commerce.” The court record points to an 2017 Office of the Inspector General report which says Customs and Border Protection (CBP) “regularly” tried much the same thing with its own legal demands, and specifically around the identity of an anonymous Twitter user.

“Movant now files this urgent motion to protect their identity from being exposed to a government agency that is apparently targeting their ‘community watch’ Facebook and Instagram accounts for doing nothing more than exercising their rights to free speech and association,” those lawyers and others wrote last week.

“Movant’s social media pages lawfully criticize and publicize DHS and the government agents who Movant views as wreaking havoc in the Montgomery County community by shining a light on that conduct to raise community members’ awareness,” they added.

The judge has not yet ruled on the ACLU’s motion to quash the demands altogether. This is a temporary blockage while that case continues.

The Montocowatch case follows other instances in which DHS has tried to compel Meta to identify the owners of similar accounts. Last month a judge temporarily blocked a subpoena that was aiming to unmask Instagram accounts that named a Border Patrol agent, The Intercept reported.

Earlier this month Meta took down a Facebook page that published ICE sightings in Chicago. The move came in direct response to pressure from the Department of Justice.

Both Apple and Google have removed apps that people use to warn others about ICE sightings. Those removals also included an app called Eyes Up that was focused more on preserving videos of ICE abuses. Apple’s moves also came after direct pressure from the Department of Justice.

Montcowatch directed a request for comment to the ACLU of Pennsylvania, which did not immediately respond.

Courts Block Meta From Sharing Anti-ICE Activists’ Instagram Account Info With Feds

For now, Meta cannot disclose to federal investigators the identities of Instagram users who named and shamed a Border Patrol agent.

^{Shawn Musgrave (The Intercept)}