Salta al contenuto principale






The Open-Source Software Saving the Internet From AI Bot Scrapers




The Open-Source Software Saving the Internet From AI Bot Scrapers


For someone who says she is fighting AI bot scrapers just in her free time, Xe Iaso seems to be putting up an impressive fight. Since she launched it in January, Anubis, a “program is designed to help protect the small internet from the endless storm of requests that flood in from AI companies,” has been downloaded nearly 200,000 times, and is being used by notable organizations including GNOME, the popular open-source desktop environment for Linux, FFmpeg, the open-source software project for handling video and other media, and UNESCO, the United Nations organization for educations, science, and culture.

Iaso decided to develop Anubis after discovering that her own Git server was struggling with AI scrapers, bots that crawl the web hoovering up anything that can be used for the training data that power AI models. Like many libraries, archives, and other small organizations, Iaso discovered her Git server was getting slammed only when it stopped working.

“I wasn't able to load it in my browser. I thought, huh, that's strange,” Iaso told me on a call. “So I looked at the logs and I figured out that it's restarted about 500 times in the last two days. So I looked in the access logs and I saw that [an] Amazon [bot] was clicking on every single link.”

Iaso knew it was an Amazon bot because it self identified as such. She said she considered withdrawing the Git server from the open web but that because she wants to keep some of the source code hosted there open to the public, she tried to stop the Amazon bot instead.

“I tried some things that I can’t admit in a recorded environment. None of them worked. So I had a bad idea,” she said. “I implemented some code. I put it up on GitHub in an experimental project dumping ground, and then the GNOME desktop environment started using it as a Hail Mary. And that's about when I knew that I had something on my hands.”

There are several ways people and organizations are trying to stop bots at the moment. Historically, robots.txt, a file sites could use to tell automated tools not to scrape, was a respected and sufficient norm for this purpose, but since the generative AI boom, major AI companies as well as less established companies and even individuals, often ignored it. CAPTCHAs, the little tests users take to prove they’re not a robot, aren’t great, Iaso said, because some AI bot scrapers have CAPTCHA solvers built in. Some developers have created “infinite mazes” that send AI bot scrapers from useless link to useless link, diverting them from the actual sites humans use and wasting their time. Cloudflare, the ubiquitous internet infrastructure company, has created a similar “AI labyrinth” feature to trap bots.

Iaso, who said she deals with some generative AI at her day job, told me that “from what I have learned, poisoning datasets doesn't work. It makes you feel good, but it ends up using more compute than you end up saving. I don't know the polite way to say this, but if you piss in an ocean, the ocean does not turn into piss.”

In other words, Iaso thinks that it might be fun to mess with the AI bots that are trying to mess with the internet, but in many cases it’s not practical to send them on these wild goose chases because it requires resources Cloudflare might have, but small organizations and individuals don’t.

“Anubis is an uncaptcha,” Iaso explains on her site. “It uses features of your browser to automate a lot of the work that a CAPTCHA would, and right now the main implementation is by having it run a bunch of cryptographic math with JavaScript to prove that you can run JavaScript in a way that can be validated on the server.”

Essentially, Anubis verifies that any visitor to a site is a human using a browser as opposed to a bot. One of the ways it does this is by making the browser do a type of cryptographic math with JavaScript or other subtle checks that browsers do by default but bots have to be explicitly programmed to do. This check is invisible to the user, and most browsers since 2022 are able to complete this test. In theory, bot scrapers could pretend to be users with browsers as well, but the additional computational cost of doing so on the scale of scraping the entire internet would be huge. This way, Anubis creates a computational cost that is prohibitively expensive for AI scrapers that are hitting millions and millions of sites, but marginal for an individual user who is just using the internet like a human.

Anubis is free, open source, lightweight, can be self-hosted, and can be implemented almost anywhere. It also appears to be a pretty good solution for what we’ve repeatedly reported is a widespread problem across the internet, which helps explain its popularity. But Iaso is still putting a lot of work into improving it and adding features. She told me she’s working on a non cryptographic challenge so it taxes users’ CPUs less, and also thinking about a version that doesn’t require JavaScript, which some privacy-minded disable in their browsers.

The biggest challenge in developing Anubis, Iaso said, is finding the balance.

“The balance between figuring out how to block things without people being blocked, without affecting too many people with false positives,” she said. “And also making sure that the people running the bots can't figure out what pattern they're hitting, while also letting people that are caught in the web be able to figure out what pattern they're hitting, so that they can contact the organization and get help. So that's like, you know, the standard, impossible scenario.”

Iaso has a Patreon and is also supported by sponsors on Github who use Anubis, but she said she still doesn’t have enough financial support to develop it full time. She said that if she had the funding, she’d also hire one of the main contributors to the project. Ultimately, Anubis will always need more work because it is a never ending cat and mouse game between AI bot scrapers and the people trying to stop them.

Iaso said she thinks AI companies follow her work, and that if they really want to stop her and Anubis they just need to distract her.

“If you are working at an AI company, here's how you can sabotage Anubis development as easily and quickly as possible,” she wrote on her site. “So first is quit your job, second is work for Square Enix, and third is make absolute banger stuff for Final Fantasy XIV. That’s how you can sabotage this the best.”






Samsung's profits plummet, Nvidia-related woes continue


Samsung Electronics is a leading supplier of system memory. However, it would like to supply High Bandwidth Memory to Nvidia for the biggest profits, but it is losing the competitive battle in this area. Due to the lack of demand for memory for AI chips, the South Korean company’s profits fell by 39 percent this quarter.


AI chip startup Groq expands with first European data center


US AI chip developer Groq has opened its first European data center in Helsinki, Finland. This marks a step forward in the company’s international expansion. It also responds to the growing demand for fast and scalable AI inferencing in Europe.



Apple’s 5th Ave store spray-painted to protest ‘climate hypocrisy’


Protesters chanted “Dump Trump, Apple!”


Brussels AI ambitions risk fuelling Big Tech dominance, watchdog warns


Public support for European AI start-ups could end up strengthening the dominance of US tech giants like Microsoft and Amazon, a new report finds.
Questa voce è stata modificata (2 mesi fa)



The Chinese EV Carmaker That Is Quietly Crushing Tesla


For the second straight quarter, Tesla’s sales have declined while Chinese rivals like BYD and NIO are surging, a sign of a major power shift in the global electric vehicle market.




Jack Dorsey working on Bluetooth messaging app, Bitchat


Block CEO and Twitter co-founder Jack Dorsey wrote in an X post on Sunday night that he spent his weekend creating an app called Bitchat (presumably a combination of the words “bit” and “chat,” and not the other combination of words that it looks like).
in reply to BrikoX

It's licensed under the "unlicense" which is public domain adjacent. But yes Briar exists and I'm not sure what this really does any better that's any more usable. It's at least interesting to see that he may have finally understood what decentralized actually means with this though. And competition is always a good thing even if it's over things that are mostly useless to the public.
in reply to Eldritch

My bad. The article didn't mention the license, but they compared it to Bridgefy so I assumed it was the same crap.


Epic reaches mystery settlement with Samsung days before new Galaxy phones


Epic is done suing over Auto Blocker.


Case file: s3.documentcloud.org/documents…



[Technology Connections] Induction lamps: fluorescent lighting's final form [30:05]


Pro doesn't like this.




Tesla’s real struggles have only just begun


Trump’s budget bill could cost Tesla billions of dollars in credit sales. And Elon Musk’s promise to create a third political party is creating its own headaches.







[Opinion] Amid Gaza’s Devastated Infrastructure, Donkey Carts Function as Ambulances


Protecting donkeys helps protect the people who depend on them to survive the ongoing crisis, says a Gaza veterinarian.


Archived version: archive.is/newest/truthout.org…


Disclaimer: The article linked is from a single source with a single perspective. Make sure to cross-check information against multiple sources to get a comprehensive view on the situation.

Questa voce è stata modificata (2 mesi fa)


[Announcement] Secrets of the Atlas Fan Art Competition


It's been a while since we've had a fan art competition for community artists, so we're very excited to bring it back! We're launching a fan art competition themed around the Secrets of the Atlas expansion and everything introduced in it. The prize pool includes Path of Exile merchandise and in-game microtransactions.

Prizes


Top Three Winners

  • Forum avatar based on the winning submission
  • Path of Exile 1 Art Book
  • Your choice of one signed Path of Exile poster
  • Your choice of two Path of Exile T-shirts
  • Chaos Orb Socks
  • A random Path of Exile Pin
  • Your choice of an Armour Set (an Armour Set from the list below)
  • Your choice of a Portal Effect (a Portal Effect from the list below)
  • Your choice of Footprints Effect (any Footprints Effect from the store)

4-5th Place

  • Path of Exile 1 Art Book
  • Your choice of one Path of Exile T-shirt
  • Chaos Orb Socks
  • A random Path of Exile Pin
  • Your choice of an Armour Set (an Armour Set from the list below)
  • Your choice of a Portal Effect (a Portal Effect from the list below)
  • Your choice of Footprints Effect (any Footprints Effect from the store)

6-10th Place

  • Your choice of one Path of Exile T-shirt
  • Chaos Orb Socks
  • A random Path of Exile Pin
  • Your choice of a Portal Effect (a Portal Effect from the list below)
  • Your choice of Footprints Effect (any Footprints Effect from the store)

11-20th Place

  • Your choice of a Portal Effect (a Portal Effect from the list below)

Runners-Up

  • Your choice of a Helmet/Helmet Effect (a Helmet/Helmet Effect from the list below)


Microtransaction Prize Pool


::: spoiler List of Armour Sets

  • Arcane Armour Pack
  • Alabaster Seraph Pack
  • Arctic Crystal Armour Pack
  • Automaton Armour Pack
  • Black Armour Pack
  • Bleached Demon King Armour Pack
  • Blood Guard Armour Set
  • Carnage Armour Pack
  • Coldsnap Armour Pack
  • Coliseum Armour Pack
  • Corsair Armour Set
  • Darkprism Armour Pack
  • Deicide Armour Pack
  • Demon Parasite Armour Set
  • Deepwater Armour Pack
  • Desert Armour Pack
  • Dragon Armour Pack
  • Elven Knight Armour Pack
  • Elite Armour Pack
  • Executioner Armour Pack
  • Extinction Armour Set
  • Faith Guard Armour Set
  • Fallen Angel Armour Pack
  • Fire Armour Pack
  • Gargoyle Armour Set
  • Ghostflame Armour Pack
  • Gloom Armour Pack
  • Gore Armour Pack
  • Gorgon Armour Pack
  • Gothic Armour Pack
  • Gryffon Armour Pack
  • Harpyon Armour Pack
  • Heartseeker Armour Pack
  • Huntsman Armour Set
  • Ice Armour Pack
  • Innocence Armour Pack
  • Ivory Chaos Armour Pack
  • Lightning Armour Pack
  • Lunaris Armour Pack
  • Mageguard Armour Set
  • Malachai Armour Pack
  • Merchant Armour Pack
  • Miracle Armour Set
  • Mystic Armour Set
  • Necrotic Armour Pack
  • Obsidian Seraph Pack
  • Project Armour Pack
  • Pure Light Armour Pack
  • Purple Crystal Armour Pack
  • Purple Pyre Knight Armour Pack
  • Pyre Knight Armour Pack
  • Radiant Armour Pack
  • Raven Armour Pack
  • Ringmaster Armour Set
  • Scientist Armour Pack
  • Seawitch Armour Pack
  • Sin Armour Pack
  • Sphinx Armour Pack
  • Steam-powered Armour Pack
  • Stormcaller Armour Pack
  • Ultimate Chaos Armour Pack
  • Vampiric Armour Pack
  • Verdant Armour Pack
  • Viper Armour Set
  • Warlock Armour Pack
  • Wasteland Armour Set
  • Wasteland Warrior Armour Pack
  • White Armour Pack
  • Wrangler Armour Set
    :::

::: spoiler List of Portal Effects

  • Arcane Portal
  • Arcane Voidgate Portal Effect
  • Arctic Crystal Portal Effect
  • Arctic Portal Effect
  • Automaton Portal
  • Balefire Portal Effect
  • Black Portal Effect
  • Corsair Portal
  • Daresso Portal Effect
  • Darkprism Portal Effect
  • Demon King Portal Effect
  • Divine Arcane Portal
  • Divine Demon King Portal
  • Divine Stygian Portal Effect
  • Dreadspire Portal Effect
  • Fire Portal Effect
  • Frost Viking Portal Effect
  • Ghostflame Stygian Portal Effect
  • Golden Angel Portal
  • Gore Portal Effect
  • Havenwood Portal
  • Huntsman Portal
  • Infernal Demon King Portal
  • Infernal Stygian Portal Effect
  • Innocence Portal
  • Kaom Portal Effect
  • Kraken Portal Effect
  • Legion Portal Effect
  • Madcap Portal
  • Malachai Portal Effect
  • Maraketh Portal
  • Miracle Portal
  • Mist Portal Effect
  • Myrmidon Portal Effect
  • Mystic Portal
  • Necrotic Demon King Portal
  • Project Portal Effect
  • Pure Light Portal Effect
  • Ringmaster Portal Effect
  • Scientist Portal Effect
  • Seraph Portal Effect
  • Spectre Portal Effect
  • Steam-powered Portal Effect
  • Stormcaller Stygian Portal Effect
  • Sulphite Portal
  • Temple Portal Effect
  • Vampiric Portal
  • Wasteland Portal
  • Wild Portal
    :::

::: spoiler List of Helmets and Helmet Effects

  • Abyss Eyes
  • Antlers
  • Apocalypse Helmet Attachment
  • Aquila Crest Helmet Attachment
  • Arcane Flaming Head Helmet Attachment
  • Arcane Halo
  • Arcane Mark of the Phoenix
  • Arcane Visage
  • Arctic Crown
  • Arctic Skull
  • Assassin Halo Helmet Attachment
  • Assassin Seer Helmet Attachment
  • Aureate Halo
  • Automaton Halo
  • Bent Horns
  • Black Headband
  • Black Pirate Bandana
  • Bleached Skull Helmet
  • Blood Guard Helmet Attachment
  • Brilliant Crown
  • Brilliant Feathers
  • Cephalopod Tricorne
  • Chaos Orb Helmet
  • Coliseum Helmet Attachment
  • Corrupted Skull
  • Cow Skull
  • Curved Horns
  • Darkprism Helmet Attachment
  • Demon King Horns
  • Despair Helmet Attachment
  • Divine Ice Crown
  • Divine Sign
  • Divine Skull
  • Dragon Mask
  • Druidic Helmet
  • Eye Patch
  • Faith Guard Helmet Attachment
  • Fiery Eyes
  • Fiery Visage
  • Flaming Head Helmet Attachment
  • Flambeau Helmet
  • Frost Viking Alternate Helmet
  • Frost Viking Helmet Attachment
  • Ghostflame Flaming Head Helmet Attachment
  • Ghostflame Skull
  • Glowing Red Eyes
  • Glorious Helmet
  • Gore Skull
  • Gorgon Snakes Helmet Attachment
  • Guardian Mask
  • Harmonious Helmet Attachment
  • Harmonious Hood
  • Holy Eyes
  • Ice Crown
  • Ice Horns
  • Incursion Hood
  • Infernal Skull
  • Innocence Laurel Helmet Attachment
  • Iron Brand
  • Iron Maiden
  • Lightning Eyes
  • Lightning Skull
  • Mark of the Phoenix
  • Medusa Snakes
  • Misery Flaming Head Helmet Attachment
  • Misery Helmet Attachment
  • Nightfall Helmet Attachment
  • Oriath Helmet Attachment
  • Oshabi's Headdress
  • Phoenix Helmet
  • Plague Doctor Helmet
  • Purple Divine Sign
  • Purple Eyes
  • Purple Flaming Head Helmet Attachment
  • Purple Skull Helmet
  • Purple Smoking Hood
  • Rapture Helmet Attachment
  • Red Headband
  • Red Pirate Bandana
  • Rose Skull
  • Serrated Mohawk
  • Shardcrown
  • Sin and Innocence Helmet Attachment
  • Sin Talon Helmet Attachment
  • Skull Hood
  • Smoking Hood
  • Spirit Charm
  • Steam-powered Skull
  • Suffering Helmet Attachment
  • Sunrise Helmet Attachment
  • Swan Crest Helmet Attachment
  • Thorned Skull
  • Twilight Helmet Attachment
  • Twisted Bone Horns
  • Twisted Horns
  • Undead Outcast Helmet
  • Vaal Orb Helmet
  • Wasteland Horns
    :::


How to Enter


Create still art that is inspired by anything introduced in the Secrets of the Atlas expansion. This excludes video submissions but does include any painting, drawing or model creation. Then just submit your work in this thread.

Multiple submissions are welcome and encouraged. Please note that the submission needs to be your own work and needs to have been created after the launch of the competition.

If you're unsure how to share an image on the forum, just follow these steps:


    1. Go to imgur.com



    1. Go to 'new post' and drag your images into the window.



    1. Once the image has been uploaded, right click it and click 'Copy image address'.



    1. Click reply to this post and paste the link into the window with at the end.



    1. When you click 'post', your image should be displayed!



Conditions


  • The competition starts at the time of this post and will end on Tuesday the 29th of July at 10PM PDT. Submissions made after this time will not be counted. We'll post the winners sometime in the following days.
  • Any work that is submitted must be your own and must be created after the launch of this competition.
  • AI generated art will not be accepted. We reserve the right to disqualify your entry if we believe that it is AI generated.
  • Multiple submissions are welcome.

Best of luck to everyone who is participating in the competition!










in reply to ray

This is frightning, google giving law enforcement a list of users who did a particular keyword search.

I am glad it helped solve the murder case but it also implies that my search history when using google services will always be stored and can be shared without my permission.
Given that its almost impossible to not use google unless you want to be frustrated while trying to do basic stuff like email, searches etc. This basically mean every bit of data generated my anyone is permanently stored and its just about time until it will be searched for any useful stuff in case there is a situation like this again which there always will be.

Questa voce è stata modificata (2 mesi fa)
in reply to tfowinder

I use duckduckgo for searches and proton for emails for 4+ years, and I have been less frustrated than with google services
in reply to ray

Consider the people who were killed here also.

The first fire truck arrived at 2:47 am. By then, the inferno had shattered the windows and plumed the air with smoke. The stench of burning wood filled the neighborhood. When firefighters subdued the blaze enough to get in the front door, they found the small body of a child. Djiby’s daughter Khadija had been two months shy of her second birthday. Farther in sprawled Djiby himself and his 23-year-old wife, Adja.

Next to Adja lay Djiby’s 25-year-old sister, Hassan. She’d only been living in the house for three months. Like Adja, she had dreamed of going back to school to study nursing. She died with her arms still wrapped around her 7-month-old daughter, Hawa Beye. Medical examiners would later conclude that all five died of smoke inhalation, airways coated in black soot, internal organs and muscles burnished “cherry-red” from the heat.