404 Media

2025-07-23 13:42:58

ChatGPT Hallucinated a Feature, Forcing Human Developers to Add It

In what might be a first, a programmer added a feature to a piece of software because ChatGPT hallucinated it, and customers kept attempting to force the software to do it.. The developers of the sheet music scanning app Soundslice, a site that lets people digitize and edit sheet music, added additional functionality to their site because the LLM kept telling people it existed. Rather than fight the LLM, Soundslice indulged the hallucination.

Adrian Holovaty, one of Soundslices’ developers, noticed something strange in the site's error logs a few months ago. Users kept uploading ASCII tablature—a basic system for notating music for guitar, despite the fact that Soundslice wasn’t set up to process it, and had never advertised that it could. The error logs included pictures of what users had uploaded, and many of them were screenshots of ChatGPT conversations where the LLM had churned out ASCII tabs and told the users to send them to Soundslice.
playlist.megaphone.fm?p=TBIEA2…
“It was around 5-10 images daily, for a period of a month or two. Definitely enough where I was like, ‘What the heck is going on here?’” Holovaty told 404 Media. Rather than fight the LLM, Soundslice decided to add the feature ChatGPT had hallucinated. Holovaty said it only took his team a few hours to write up the code, which was a major factor in adding the feature.

“The main reason we did this was to prevent disappointment,” he said. “I highly doubt many people are going to sign up for Soundslice purely to use our ASCII tab importer […] we were motivated by the, frankly, galling reality that ChatGPT was setting Soundslice users up for failure. I mean, from our perspective, here were the options:

“1. Ignore it, and endure the psychological pain of knowing people were getting frustrated by our product for reasons out of our control.

“2. Put annoying banners on our site saying: ‘On the off chance that you're using ChatGPT and it told you about a Soundslice ASCII tab feature, that doesn't exist.’ That's disproportional and lame.

“3. Just spend a few hours and develop the feature.”

There’s also no way to tell ChatGPT the feature doesn’t exist. In an ideal world, OpenAI would have a formal procedure for removing content from its model, similar to the ability to request the removal of a site from Google’s index. “Obviously with an LLM it's much harder to do this technically, but I'm sure they can figure it out, given the absurdly high salaries their researchers are earning,” Holovaty said.

He added that the situation made him realize how powerful ChatGPT has become as an influencer of consumer behavior. “It's making product recommendations—for existent and nonexistent features alike—to massive audiences, with zero transparency into why it made those particular recommendations. And zero recourse.”

This may be the first time that developers have added a feature to a piece of software because ChatGPT hallucinated it, but it won’t be the last. In a personal blog, developer Niki Tonsky dubbed this phenomenon “gaslight-driven development” and shared a recent experience that’s similar to Holovaty’s.

One of Tonsky’s projects is a database for frontends called Instant. An update method for the app used a text document called “update” but LLMs that interacted with Instant kept calling the file “create.” Tonsky told 404 Media that, rather than fight the LLMs, his team just added the text file with the name the systems wanted. “In general I agree `create` is more obvious, it’s just weird that we arrived at this through LLM,” he said.

He told 404 media that programmers will probably need to account for the “tastes” of LLMs in the future. “You kinda already have to. It’s not programming for AI, but AI as a tool changes how we do programming,” he said.

Holovaty doesn’t hate AI—Soundslice uses machine learning to do its magic—but is mixed on LLMs. He compared his experience with ChatGPT to dealing with an overzealous sales team selling a feature that doesn’t exist. He also doesn’t trust LLMs to write code. He experimented with it, but found it caused more problems than it solved.

“I don't trust it for my production Soundslice code,” he said. “Plus: writing code is fun! Why would I choose to deny myself fun? To appease the capitalism gods? No thanks.”

Soundslice | Create living sheet music

Learn music better with our living sheet music.

^Soundslice

404 Media

2025-07-21 17:19:12

The NIH Is Capping Research Proposals Because It's Overwhelmed by AI Submissions

The National Institutes of Health claims it’s being strained by an onslaught of AI-generated research applications and is capping the number of proposals researchers can submit in a year.

In a new policy announcement on July 17, titled “Supporting Fairness and Originality in NIH Research Applications,” the NIH wrote that it has recently “observed instances of Principal Investigators submitting large numbers of applications, some of which may have been generated with AI tools,” and that this influx of submissions “may unfairly strain NIH’s application review process.”

“The percentage of applications from Principal Investigators submitting an average of more than six applications per year is relatively low; however, there is evidence that the use of AI tools has enabled Principal Investigators to submit more than 40 distinct applications in a single application submission round,” the NIH policy announcement says. “NIH will not consider applications that are either substantially developed by AI, or contain sections substantially developed by AI, to be original ideas of applicants. If the detection of AI is identified post award, NIH may refer the matter to the Office of Research Integrity to determine whether there is research misconduct while simultaneously taking enforcement actions including but not limited to disallowing costs, withholding future awards, wholly or in part suspending the grant, and possible termination.”

Starting on September 25, NIH will only accept six “new, renewal, resubmission, or revision applications” from individual principal investigators or program directors in a calendar year.

Earlier this year, 404 Media investigated AI used in published scientific papers by searching for the phrase “as of my last knowledge update” on Google Scholar, and found more than 100 results—indicating that at least some of the papers relied on ChatGPT, which updates its knowledge base periodically. And in February, a journal published a paper with several clearly AI-generated images, including one of a rat with a giant penis. In 2023, Nature reported that academic journals retracted 10,000 "sham papers," and the Wiley-owned Hindawi journals retracted over 8,000 fraudulent paper-mill articles. Wiley discontinued the 19 journals overseen by Hindawi. AI-generated submissions affect non-research publications, too: The science fiction and fantasy magazine Clarkesworld stopped accepting new submissions in 2023 because editors were overwhelmed by AI-generated stories.

According to an analysis published in the Journal of the American Medical Association, from February 28 to April 8, the Trump administration terminated $1.81 billion in NIH grants, in subjects including aging, cancer, child health, diabetes, mental health and neurological disorders, NBC reported.

Just before the submission limit announcement, on July 14, Nature reported that the NIH would “soon disinvite dozens of scientists who were about to take positions on advisory councils that make final decisions on grant applications for the agency,” and that staff members “have been instructed to nominate replacements who are aligned with the priorities of the administration of US President Donald Trump—and have been warned that political appointees might still override their suggestions and hand-pick alternative reviewers.”

The NIH Office of Science Policy did not immediately respond to a request for comment.

Trump administration cut more than $1.8 billion in NIH grants

The Trump administration terminated $1.81 billion in National Institutes of Health grants in less than 40 days, including $544 million in as-yet-unspent funds.

^{Aria Bendix (NBC News)}

404 Media

2025-07-21 13:05:07

A Startup is Selling Data Hacked from Peoples’ Computers to Debt Collectors

When your laptop is infected with infostealing malware, it’s not just hackers that might get your passwords, billing and email addresses, and a list of sites or services you’ve created accounts on, potentially including some embarrassing ones. A private intelligence company run by a young founder is now taking that hacked data from what it says are more than 50 million computers, and reselling it for profit to a wide range of different industries, including debt collectors; couples in divorce proceedings; and even companies looking to poach their rivals’ customers. Essentially, the company is presenting itself as a legitimate, legal business, but is selling the same sort of data that was previously typically sold by anonymous criminals on shady forums or underground channels.

Multiple experts 404 Media spoke to called the practice deeply unethical, and in some cases the use of that data probably illegal. The company is also selling access to a subset of the data to anyone for as little as $50, and 404 Media used it to uncover unsuspecting victims’ addresses.

The activities of the company, called Farnsworth Intelligence, show a dramatic shift in the bevvy of companies that collect and sell access to so-called open source intelligence, or OSINT. Historically, OSINT has included things like public social media profiles or flight data. Now, companies increasingly see data extracted from peoples’ personal or corporate machines and then posted online as fair game not just to use in their own investigations, but to repackage and sell too.

“To put it plainly this company is profiting off of selling stolen data, re-victimizing people who have already had their personal devices compromised and their data stolen,” Cooper Quintin, senior public interest technologist at the Electronic Frontier Foundation (EFF), told 404 Media. “This data will likely be used to further harm people by police using it for surveillance without a warrant, stalkers using it to gather information on their targets, high level scams, and other damaging motives.”

💡
Do you know anything else about people selling data to debt collectors or these other industries? I would love to hear from you. Using a non-work device, you can message me securely on Signal at joseph.404 or send me an email at joseph@404media.co.

Infostealers are pieces of malware, often stealthily bundled in a piece of pirated software, that steal a victim’s cookies, login credentials, and often more information stored in their browser too. On its website, Farnsworth lays out several potential uses for that stolen data. This includes “skip tacing,” presumably a typo of skip tracing, which is where a private individual or company tracks someone down who owes a debt. The website says users can “find debtors up-to-date addresses.” Another use case is to “Find high impact evidence that can make/break the case of million dollar lawsuits, high value divorce cases, etc.” A third is to “generate lead lists of customers/users from competitors [sic] companies,” because the data could show which competing products they have login credentials for, and, presumably, use.

Calli Schroeder, senior counsel at the Electronic Privacy Information Center (EPIC), told 404 Media that the use cases Farnsworth offers are “not only morally questionable [...] but may not be legal or usable in some cases.” For the litigation one, courts are split on using stolen information as evidence in legal proceedings. When hackers targeted the dating site Ashley Madison, for example, a judge ruled that despite the data being publicly published it was still confidential and stolen and couldn’t be used. Most judges will not allow illegally obtained evidence in divorce proceedings either, Schroeder said.

Then for using the data to build a list of customers of competitors, Schroeder said that “may very well fall under corporate espionage and trade secrets violations, depending on what information is taken.”

“This is so gross and predatory. They are facilitating and enabling further exploitation of victims of a crime and bragging about how multiple criminal acts make their business better. Moral bankruptcy is common in this industry, but I rarely see a company so proud of it,” Schroeder added.
playlist.megaphone.fm?p=TBIEA2…
Farnsworth did not respond to multiple requests for comment. Aidan Raney, the company’s 23 year-old founder, did not respond to multiple Signal messages sent to an account he has previously used to communicate with 404 Media.

Farnsworth offers two infostealer related products. The first is Farnsworth’s “Infostealer Data Platform,” which lists those above use cases. This can display hacking victims’ full text passwords, and requires potential users to contact Farnsworth for access. The company asks applicants to explain their use case, and can include “private investigations, intelligence, journalism, law enforcement, cyber security, compliance, IP/brand protection,” and several others, according to its website.

The second product is infostealers.info, a publicly available service that requires no due diligence to enter. It only asks for a minimum of $50 to search through the results. These don’t include victims’ full passwords, but the platform still includes a wide range of sensitive information. Recently infostealers.info introduced the ability to search for data stored in a hacking victim’s autofill. That is, data stored in the browser for convenience that can automatically populate when filling out a form, such as a billing address. Using this tool, 404 Media was able to extract multiple peoples’ billing addresses. One was in Staten Island, New York, which appeared to be someone’s private residence. Another address was in India.

Inside the Massive Crime Industry That is Hacking Billion Dollar Companies
When you download that piece of pirated software, you might be also getting a piece of infostealer malware, and entering a highly complex hacking ecosystem that is fueling some of the biggest breaches on the planet.
404 MediaJoseph Cox

In other words, these people had been hacked, and now anyone with $50 was able to search through data stolen from their computer.

“This should also be an example of how once your data is lost in a breach you can't control what will happen to it. It can be used by law enforcement, stalkers, scammers, advertisers, or anyone with access to it. It's a stark reminder of why digital security is important even if you think you have nothing to hide,” Quintin from the EFF added.

Hackers running infostealer operations often create Telegram channels where they upload personal data their malware has stolen. Other criminals can then pay to access this stolen data. The administrator of one prolific infostealer campaign previously told 404 Media “this brings us good income, but I am not ready to disclose specific amounts.” Infostealers operators often then publish stolen credentials on Telegram for free, likely as a way to advertise their paid offerings. Farnsworth did not respond when asked if it is buying this stolen data from hackers to then put into its product.

Cybersecurity researchers have used infostealer data to unmask criminals. Hudson Rock, another company that sells infostealer-related services, used it to uncover information on two alleged fraudsters on the FBI’s Most Wanted List. Last year cybersecurity firm RecordedFuture said it found 3,334 unique credentials used to access child abuse imagery websites. It says it used that data to identify two individuals. In a LinkedIn post on Tuesday, Raney said the company has explored its own dataset in a similar way.

But those are different use cases to selling infostealer data on the open market or for potentially illegal use cases.

Quintin said “It would be illegal and unethical to sell stolen cell phones even if you didn't steal them yourself, and I don't see how this is any different.”

Infostealer data can unmask Tor users and attribute the users/admins of CSAM distribution forums...

Infostealer data can unmask Tor users and attribute the users/admins of CSAM distribution forums... After exploring our infostealer data at Farnsworth Intelligence we learned that the users and admins of countless dark web forums, such as those dist…

^{Aidan Raney (www.linkedin.com)}