404 Media

2025-07-07 13:16:50

The Open-Source Software Saving the Internet From AI Bot Scrapers

For someone who says she is fighting AI bot scrapers just in her free time, Xe Iaso seems to be putting up an impressive fight. Since she launched it in January, Anubis, a “program is designed to help protect the small internet from the endless storm of requests that flood in from AI companies,” has been downloaded nearly 200,000 times, and is being used by notable organizations including GNOME, the popular open-source desktop environment for Linux, FFmpeg, the open-source software project for handling video and other media, and UNESCO, the United Nations organization for educations, science, and culture.

Iaso decided to develop Anubis after discovering that her own Git server was struggling with AI scrapers, bots that crawl the web hoovering up anything that can be used for the training data that power AI models. Like many libraries, archives, and other small organizations, Iaso discovered her Git server was getting slammed only when it stopped working.

“I wasn't able to load it in my browser. I thought, huh, that's strange,” Iaso told me on a call. “So I looked at the logs and I figured out that it's restarted about 500 times in the last two days. So I looked in the access logs and I saw that [an] Amazon [bot] was clicking on every single link.”

Iaso knew it was an Amazon bot because it self identified as such. She said she considered withdrawing the Git server from the open web but that because she wants to keep some of the source code hosted there open to the public, she tried to stop the Amazon bot instead.

“I tried some things that I can’t admit in a recorded environment. None of them worked. So I had a bad idea,” she said. “I implemented some code. I put it up on GitHub in an experimental project dumping ground, and then the GNOME desktop environment started using it as a Hail Mary. And that's about when I knew that I had something on my hands.”

There are several ways people and organizations are trying to stop bots at the moment. Historically, robots.txt, a file sites could use to tell automated tools not to scrape, was a respected and sufficient norm for this purpose, but since the generative AI boom, major AI companies as well as less established companies and even individuals, often ignored it. CAPTCHAs, the little tests users take to prove they’re not a robot, aren’t great, Iaso said, because some AI bot scrapers have CAPTCHA solvers built in. Some developers have created “infinite mazes” that send AI bot scrapers from useless link to useless link, diverting them from the actual sites humans use and wasting their time. Cloudflare, the ubiquitous internet infrastructure company, has created a similar “AI labyrinth” feature to trap bots.

Iaso, who said she deals with some generative AI at her day job, told me that “from what I have learned, poisoning datasets doesn't work. It makes you feel good, but it ends up using more compute than you end up saving. I don't know the polite way to say this, but if you piss in an ocean, the ocean does not turn into piss.”

In other words, Iaso thinks that it might be fun to mess with the AI bots that are trying to mess with the internet, but in many cases it’s not practical to send them on these wild goose chases because it requires resources Cloudflare might have, but small organizations and individuals don’t.

“Anubis is an uncaptcha,” Iaso explains on her site. “It uses features of your browser to automate a lot of the work that a CAPTCHA would, and right now the main implementation is by having it run a bunch of cryptographic math with JavaScript to prove that you can run JavaScript in a way that can be validated on the server.”

Essentially, Anubis verifies that any visitor to a site is a human using a browser as opposed to a bot. One of the ways it does this is by making the browser do a type of cryptographic math with JavaScript or other subtle checks that browsers do by default but bots have to be explicitly programmed to do. This check is invisible to the user, and most browsers since 2022 are able to complete this test. In theory, bot scrapers could pretend to be users with browsers as well, but the additional computational cost of doing so on the scale of scraping the entire internet would be huge. This way, Anubis creates a computational cost that is prohibitively expensive for AI scrapers that are hitting millions and millions of sites, but marginal for an individual user who is just using the internet like a human.

Anubis is free, open source, lightweight, can be self-hosted, and can be implemented almost anywhere. It also appears to be a pretty good solution for what we’ve repeatedly reported is a widespread problem across the internet, which helps explain its popularity. But Iaso is still putting a lot of work into improving it and adding features. She told me she’s working on a non cryptographic challenge so it taxes users’ CPUs less, and also thinking about a version that doesn’t require JavaScript, which some privacy-minded disable in their browsers.

The biggest challenge in developing Anubis, Iaso said, is finding the balance.

“The balance between figuring out how to block things without people being blocked, without affecting too many people with false positives,” she said. “And also making sure that the people running the bots can't figure out what pattern they're hitting, while also letting people that are caught in the web be able to figure out what pattern they're hitting, so that they can contact the organization and get help. So that's like, you know, the standard, impossible scenario.”

Iaso has a Patreon and is also supported by sponsors on Github who use Anubis, but she said she still doesn’t have enough financial support to develop it full time. She said that if she had the funding, she’d also hire one of the main contributors to the project. Ultimately, Anubis will always need more work because it is a never ending cat and mouse game between AI bot scrapers and the people trying to stop them.

Iaso said she thinks AI companies follow her work, and that if they really want to stop her and Anubis they just need to distract her.

“If you are working at an AI company, here's how you can sabotage Anubis development as easily and quickly as possible,” she wrote on her site. “So first is quit your job, second is work for Square Enix, and third is make absolute banger stuff for Final Fantasy XIV. That’s how you can sabotage this the best.”

Developer Creates Infinite Maze That Traps AI Training Bots

"Nepenthes generates random links that always point back to itself - the crawler downloads those new links. Nepenthes happily just returns more and more lists of links pointing back to itself."

^{Jason Koebler (404 Media)}

minioctt

2025-07-07 18:38:39

passatanza crostanzica non salvata ma perduta (il casino coi salvataggi su ACNH)

Ieri ho casualmente menzionato gli Animali Crostini? E dunque, per volere del solito mio destino contorsionista, oggi proprio a riguardo dei Crostini Animali scopro una cosa che a suo modo fa sicuramente ridere, anche se non è per nulla divertente. Ho aperto il gioco su Swiss, circa al volo, perché mi serviva un’immagine, per scrivere un altro post per cui non trovo quello che voglio su Pinterest… e forse mi servivano dei nuovi vestiti… e quindi stavo per andare dalle Sorelle Ago e Filo sperando di trovare qualcosa… al che mi sono accorta che il negozio non ci sta… non si trova nemmeno sulla mappa. 💀

Mooolto strano, ho pensato, perché ricordo di averlo avuto in passato — e con assoluta certezza, perché ricordo perfettamente alcuni momenti di gioco relativi ad esso… ho sicuramente anche screenshot, da qualche parte. Il mio cervello ha, come al solito, immediatamente iniziato a pensare ai complotti, e quindi subito ho trovato una discussione su Reddit “THE ABLE SISTERS DISAPPEARED!?!?“, dove non solo chi ha creato il post, ma anche altri gamer, riportano l’inspiegabile sparizione del preciso negozio… e questa singola paginetta sembra il solo posto sul web in cui se ne parla. 😩

Già così la cosa è pericolosamente strana, ma comunque sia purtroppo il danno c’è, e quindi devo indagare oltre. Seguendo il consiglio di un commento nel post (che, a rivederlo ora, è evidentemente dato totalmente alla cazzo di cane; classico momento Reddit), vado a parlare con Nook, per fare come per spostare l’edificio della sartoria… ma, giustamente, nella lista non c’è. Ho pure perso un botto di tempo nei sottomenu di ‘sto procione, perché con il gioco in giapponese non ci capisco granché… finché, giusto per scrupolo, non ho parlato con Fuffi, che mi ha detto di parlare con Nook, perché “ci sono delle cose da fare” o roba del genere (la prima opzione nel menu), e a quel punto lui ha parlato di Project K… e io sono morta fulminata sul colpo. 💔

Cioè, quasi. Project K sarebbe l’ultima fase del prologo di New Horizons, quella che fa venire K.(K). Slider sull’isola, e porta anche allo sblocco degli strumenti di terraforming… ma io, anche questo, lo avevo fatto — e anzi, sono ancora più sicura di averlo fatto, perché, a parte il concerto del cane che si pensa VIP, ricordo perfettamente come ho distrutto malamente l’isola, una settimanella prima di abbandonare virtualmente per sempre il gioco, nel lontano 2020, prima che qualche anno dopo non ripristinassi un backup precedente del salvataggio, per provare almeno a vedere se mi tornasse un minimo la voglia di giocare e…

…MANNAGGIA! Ho realizzato solo in questo istante, dopo letterali anni dall’azione maledetta, di aver ripristinato un backup troppo vecchio. Si perché, anche qui, ricordo benissimo di aver fatto il ripristino con il solo intento di riportare l’isola ad uno stato giusto precedente alla mia opera di spacc, e non con lo scopo di portare indietro la progressione del gioco. All’epoca usai un homebrew fatto apposta per i Nuovi Crostini, prima che gli altri gestori di salvataggi supportassero il formato usato dal gioco (associato alla console, anziché agli utenti), che non aveva una GUI per selezionare diversi salvataggi… e quindi, quello che credo sia successo è che avrò sbagliato a spostare le cartelle da PC per scegliere il backup da ripristinare, prendendone uno troppo vecchio (…o, forse, semplicemente non ne avevo uno più nuovo che non fosse già distrutto); fatto sta che solo ora mi sto accorgendo dell’errore. 😭

Il danno ora è chiaro, ma la beffa stavolta è più subdola del solito: al momento, su due piedi, non trovo più questi miei fantomatici backup di ACNH, né tantomeno quel tool (che ormai sarà deprecato), di cui non ricordo nemmeno il nome! Mentre credo che il secondo non sia un vero problema, perché JKSM dovrebbe comunque leggere tranquillamente la cartella esportata, ad ogni modo senza le copie dei salvataggi non si va da nessuna parte. Se va bene, forse staranno da qualche parte tra i miei fin troppi e fin troppo disordinati dischi e dispositivi, dove almeno una o due volte avrò ciclicamente svuotato la microSD di Switch per liberare spazio o per formattare… ma, francamente, a questo punto, accetto il bruh momento e piuttosto inizio un nuovo salvataggio di New Leaf (e sarebbe l’ennesimo…) ma in giapponese (…perché ai tempi Animal Crossing era region-locked e il salvataggio non è cross-compatibile, ho provato), e forse riuscirò a copare. 😤
E questa qui, ovviamente, non è la foto che cercavo all’inizio. Rappresenta sempre adeguatamente il mio stato d’animo e d’essere, ma solo relativamente a questa ennesima rivelazione dei miei infiniti problemi di skill, e non a cos’altro avevo da scrivere prima. Non si capisce nemmeno se sto piangendo, o starnutendo, o mi voglio semplicemente buttare nel fiume così da farla finita con tutte le merdate che per via del software mi accadono (in questo caso per colpa mia, ok, ma comunque)… e ciò non era voluto, ma forse è meglio così. (E, c’è da dire, le foto sceniche sono davvero l’unica cosa che manca a New Leaf, a parte il poter piazzare mobili fuori.) 😾
#ACNH #AnimalCrossing #AnimalCrossingNewHorizons #BruhMoment #hacking #Mannaggia #NewHorizons #salvataggi #savegame #SkillIssue

404 Media

2025-07-07 13:16:50

The Open-Source Software Saving the Internet From AI Bot Scrapers

For someone who says she is fighting AI bot scrapers just in her free time, Xe Iaso seems to be putting up an impressive fight. Since she launched it in January, Anubis, a “program is designed to help protect the small internet from the endless storm of requests that flood in from AI companies,” has been downloaded nearly 200,000 times, and is being used by notable organizations including GNOME, the popular open-source desktop environment for Linux, FFmpeg, the open-source software project for handling video and other media, and UNESCO, the United Nations organization for educations, science, and culture.

Iaso decided to develop Anubis after discovering that her own Git server was struggling with AI scrapers, bots that crawl the web hoovering up anything that can be used for the training data that power AI models. Like many libraries, archives, and other small organizations, Iaso discovered her Git server was getting slammed only when it stopped working.

“I wasn't able to load it in my browser. I thought, huh, that's strange,” Iaso told me on a call. “So I looked at the logs and I figured out that it's restarted about 500 times in the last two days. So I looked in the access logs and I saw that [an] Amazon [bot] was clicking on every single link.”

Iaso knew it was an Amazon bot because it self identified as such. She said she considered withdrawing the Git server from the open web but that because she wants to keep some of the source code hosted there open to the public, she tried to stop the Amazon bot instead.

“I tried some things that I can’t admit in a recorded environment. None of them worked. So I had a bad idea,” she said. “I implemented some code. I put it up on GitHub in an experimental project dumping ground, and then the GNOME desktop environment started using it as a Hail Mary. And that's about when I knew that I had something on my hands.”

There are several ways people and organizations are trying to stop bots at the moment. Historically, robots.txt, a file sites could use to tell automated tools not to scrape, was a respected and sufficient norm for this purpose, but since the generative AI boom, major AI companies as well as less established companies and even individuals, often ignored it. CAPTCHAs, the little tests users take to prove they’re not a robot, aren’t great, Iaso said, because some AI bot scrapers have CAPTCHA solvers built in. Some developers have created “infinite mazes” that send AI bot scrapers from useless link to useless link, diverting them from the actual sites humans use and wasting their time. Cloudflare, the ubiquitous internet infrastructure company, has created a similar “AI labyrinth” feature to trap bots.

Iaso, who said she deals with some generative AI at her day job, told me that “from what I have learned, poisoning datasets doesn't work. It makes you feel good, but it ends up using more compute than you end up saving. I don't know the polite way to say this, but if you piss in an ocean, the ocean does not turn into piss.”

In other words, Iaso thinks that it might be fun to mess with the AI bots that are trying to mess with the internet, but in many cases it’s not practical to send them on these wild goose chases because it requires resources Cloudflare might have, but small organizations and individuals don’t.

“Anubis is an uncaptcha,” Iaso explains on her site. “It uses features of your browser to automate a lot of the work that a CAPTCHA would, and right now the main implementation is by having it run a bunch of cryptographic math with JavaScript to prove that you can run JavaScript in a way that can be validated on the server.”

Essentially, Anubis verifies that any visitor to a site is a human using a browser as opposed to a bot. One of the ways it does this is by making the browser do a type of cryptographic math with JavaScript or other subtle checks that browsers do by default but bots have to be explicitly programmed to do. This check is invisible to the user, and most browsers since 2022 are able to complete this test. In theory, bot scrapers could pretend to be users with browsers as well, but the additional computational cost of doing so on the scale of scraping the entire internet would be huge. This way, Anubis creates a computational cost that is prohibitively expensive for AI scrapers that are hitting millions and millions of sites, but marginal for an individual user who is just using the internet like a human.

Anubis is free, open source, lightweight, can be self-hosted, and can be implemented almost anywhere. It also appears to be a pretty good solution for what we’ve repeatedly reported is a widespread problem across the internet, which helps explain its popularity. But Iaso is still putting a lot of work into improving it and adding features. She told me she’s working on a non cryptographic challenge so it taxes users’ CPUs less, and also thinking about a version that doesn’t require JavaScript, which some privacy-minded disable in their browsers.

The biggest challenge in developing Anubis, Iaso said, is finding the balance.

“The balance between figuring out how to block things without people being blocked, without affecting too many people with false positives,” she said. “And also making sure that the people running the bots can't figure out what pattern they're hitting, while also letting people that are caught in the web be able to figure out what pattern they're hitting, so that they can contact the organization and get help. So that's like, you know, the standard, impossible scenario.”

Iaso has a Patreon and is also supported by sponsors on Github who use Anubis, but she said she still doesn’t have enough financial support to develop it full time. She said that if she had the funding, she’d also hire one of the main contributors to the project. Ultimately, Anubis will always need more work because it is a never ending cat and mouse game between AI bot scrapers and the people trying to stop them.

Iaso said she thinks AI companies follow her work, and that if they really want to stop her and Anubis they just need to distract her.

“If you are working at an AI company, here's how you can sabotage Anubis development as easily and quickly as possible,” she wrote on her site. “So first is quit your job, second is work for Square Enix, and third is make absolute banger stuff for Final Fantasy XIV. That’s how you can sabotage this the best.”

Developer Creates Infinite Maze That Traps AI Training Bots

"Nepenthes generates random links that always point back to itself - the crawler downloads those new links. Nepenthes happily just returns more and more lists of links pointing back to itself."

^{Jason Koebler (404 Media)}