I get the centralization concerns, but I would think that's on the consumer since there are other options. As for the fascist content, as another commenter said, they could risk their safe harbor if they started stated regulating content that they weren't legally required to regulate.
They’re not. They’re using this as an excuse to become paid gatekeepers of the internet as we know it. All that’s happening is that Cloudflare is using this to menuever into position where they can say “nice traffic you’ve got there - would be a shame if something happened to it”.
This feels like cloudflare trying to collect rent from both sides instead of doing what’s best for the website owners.
There is a problem with AI crawlers, but these technologies are essentially doing a search, fetching a several pages, scanning/summarizing them, then presenting the findings to the user.
I don’t really think that’s wrong, it’s just a faster version of rummaging through the SEO shit you do when you Google something.
(I’ve never used perplexity, I do use Kagi’s ki assistant for similar search. It runs 3 searches and scans the top results and then provides citations)
On the flip side, most websites are so ad-ridden these days a reader mode or other summary tool is almost required for normal browsing. Not saying that AI is the right move, but I can understand not wanting to visit the actual page any more.
Search engines been going relatively fine for decades now. But the crawlers from AI companies basically DDOS hosts in comparison, sending so many requests in such a short interval. Crawling dynamic links as well that are expensive to render compared to a static page, ignoring the robots.txt entirely, or even using it discover unlinked pages.
Servers have finite resources, especially self hosted sites, while AI companies have disproportinately more at their disposal, easily grinding other systems to a halt by overwhelming them with requests.
Cloudflare runs as a CDN/cache/gateway service in front of a ton of websites. Their service is to help protect against DDOS and malicious traffic.
A few weeks ago cloudflare announced they were going to block AI crawling (good, in my opinion). However they also added a paid service that these AI crawlers can use, so it actually becomes a revenue source for them.
This is a response to that from Perplexity who run an AI search company. I don’t actually know how their service works, but they were specifically called out in the announcement and Cloudflare accused them of “stealth scraping” and ignoring robots.txt and other things.
A few weeks ago cloudflare announced they were going to block AI crawling (good, in my opinion). However they also added a paid service that these AI crawlers can use, so it actually becomes a revenue source for them.
I think it's also worth pointing out that all of the big AI companies are currently burning through cash at an absolutely astonishing rate, and none of them are anywhere close to being profitable. So pay-walling the data they use is probably gonna be pretty painful for their already-tortured bottom line (good).
Perplexity (an "AI search engine" company with 500 million in funding) can't bypass cloudflare's anti-bot checks. For each search Perplexity scrapes the top results and summarizes them for the user. Cloudflare intentionally blocks perplexity's scrapers because they ignore robots.txt and mimic real users to get around cloudflare's blocking features. Perplexity argues that their scraping is acceptable because it's user initiated.
Personally I think cloudflare is in the right here. The scraped sites get 0 revenue from Perplexity searches (unless the user decides to go through the sources section and click the links) and Perplexity's scraping is unnecessarily traffic intensive since they don't cache the scraped data.
…and Perplexity's scraping is unnecessarily traffic intensive since they don't cache the scraped data.
That seems almost maliciously stupid. We need to train a new model. Hey, where’d the data go? Oh well, let’s just go scrape it all again. Wait, did we already scrape this site? No idea, let’s scrape it again just to be sure.
I think it boils down to "consent" and "remuneration".
I run a website, that I do not consent to being accessed for LLMs. However, should LLMs use my content, I should be compensated for such use.
So, these LLM startups ignore both consent, and the idea of remuneration.
Most of these concepts have already been figured out for the purpose of law, if we consider websites much akin to real estate: Then, the typical trespass laws, compensatory usage, and hell, even eminent domain if needed ie, a city government can "take over" the boosted post feature to make sure alerts get pushed as widely and quickly as possible.
That all sounds very vague to me, and I don't expect it to be captured properly by law any time soon. Being accessed for LLM? What does it mean for you and how is it different from being accessed by a user? Imagine you host a weather forecast. If that information is public, what kind of compensation do you expect from anyone or anything who accesses that data?
Is it okay for a person to access your site? Is it okay for a script written by that person to fetch data every day automatically? Would it be okay for a user to dump a page of your site with a headless browser? Would it be okay to let an LLM take a look at it to extract info required by a user? Have you heard about changedetection.io project? If some of these sound unfair to you, you might want to put a DRM on your data or something.
Would you expect a compensation from me after reading your comment?
That all sounds very vague to me, and I don’t expect it to be captured properly by law any time soon.
It already has been captured, properly in law, in most places. We can use the US as an example: Both intellectual property and real property have laws already that cover these very items.
What does it mean for you and how is it different from being accessed by a user?
Well, does a user burn up gigawatts of power, to access my site every time? That's a huge different.
Imagine you host a weather forecast. If that information is public, what kind of compensation do you expect from anyone or anything who accesses that data?
Depends on the terms of service I set for that service.
Is it okay for a person to access your site?
Sure!
Is it okay for a script written by that person to fetch data every day automatically?
Sure! As long as it doesn't cause problems for me, the creator and hoster of said content.
Would it be okay for a user to dump a page of your site with a headless browser?
See above. Both power usage and causing problems for me.
Would it be okay to let an LLM take a look at it to extract info required by a user?
No. I said, I do not want my content and services to be used by and for LLMs.
Have you heard about changedetection.io project?
I have now. And should a user want to use that service, that service, which charges 8.99/month for it needs to pay me a portion of that, or risk having their service blocked.
There no need to use it, as I already provide RSS feeds for my content. Use the RSS feed, if you want updates.
If some of these sound unfair to you, you might want to put a DRM on your data or something.
Or, I can just block them, via a service like Cloud Flare. Which I do.
Would you expect a compensation from me after reading your comment?
None. Unless you're wanting to access if via an LLM. Then I want compensation for the profit driven access to my content.
Both intellectual property and real property have laws already that cover these very items.
And it causes a lot of trouble to many people and pains me specifically. Information should not be gated or owned in a way that would make it illegal for anyone to access it under proper conditions. License expiration causing digital work to die out, DRM causing software to break, idiotic license owners not providing appropriate service, etc.
Well, does a user burn up gigawatts of power, to access my site every time?
Doing a GET request doesn't do that.
As long as it doesn't cause problems for me, the creator and hoster of said content.
What kind of problems that would be?
Both power usage and causing problems for me.
?? How? And what?
do not want my content and services to be used by and for LLMs.
You have to agree that at one point "be used by LLM" would not be different from "be used by a user".
which charges 8.99/month
It's self-hosted and free.
Use the RSS feed, if you want updates.
How does that prohibit usage and processing of your info? That sounds like "I won't be providing any comments on Lemmy website, if you want my opinion you can mail me at a@b.com"
I can just block them, via a service like Cloud Flare. Which I do.
That will never block all of them. Your info will be used without your consent and you will not feel troubled from it. So you might not feel troubled if more things do the same.
None. Unless you're wanting to access if via an LLM. Then I want compensation for the profit driven access to my content.
What if I use my local hosted LLM? Anyway, the point is, selling text can't work well, and you're going to spend much more resources on collecting and summarizing data about how your text was used and how others benefited from it, in order to get compensation, than it worths.
Also, it might be the case that some information is actually worthless when compared to a service provided by things like LLM, even though they use that worthless information in the process.
I'm all for killing off LLMs, btw. Concerns of site makers who think they are being damaged by things like Perplexity are nothing compared to what LLMs do to the world. Maybe laws should instead make it illegal to waste energy. Before energy becomes the main currency.
Information should not be gated or owned in a way that would make it illegal for anyone to access it under proper conditions.
Then you don't believe content creators should have any control over their own works?
The "proper conditions" are deemed by the content creator, not the consumers.
Doing a GET request doesn’t do that.
Not at all. It consumes at most, a watt.
What kind of problems that would be?
Increasing my hosting bill, to accommodate the senseless traffic being sent my way?
Outages for my site, making my content unavailable for legitimate users?
You have to agree that at one point “be used by LLM” would not be different from “be used by a user”.
Not at all. LLMs are not users.
It’s self-hosted and free.
If you want, or they charge for the hosted version. If they want to use a paid for version, then they can divert some of that revenue to me, the creator, because without creators, they would have no product.
How does that prohibit usage and processing of your info? That sounds like “I won’t be providing any comments on Lemmy website, if you want my opinion you can mail me at a@b.com”
That's a apples and oranges comparison, and you know it.
That will never block all of them. Your info will be used without your consent and you will not feel troubled from it. So you might not feel troubled if more things do the same.
Perplexity seems to be troubled by it.
What if I use my local hosted LLM? Anyway, the point is, selling text can’t work well, and you’re going to spend much more resources on collecting and summarizing data about how your text was used and how others benefited from it, in order to get compensation, than it worths.
If selling text can't work well, then why do LLM products insist on using my text, to sell it?
Also, it might be the case that some information is actually worthless when compared to a service provided by things like LLM, even though they use that worthless information in the process.
LLMs are a net negative, as far as costs go. They consume far more in resources than they provide in benefit. If my information was worthless without an LLM, it's worthless with an LLM, therefore, LLMs don't need to access it. Periodt.
The bottom line? Content creators get the first say in how their content is used, and consumed. You are not entitled to their labor, for free, and without condition.
Don't feel like spending time on this anymore. To me you are not different from idiots who destroys information once they can't sell it anymore, who sue webarchive, who calls pirated copy a lost sale, who shut down game servers etc. LLM might be worse than those but Perplexity is certainly a lesser player in the field.
LLM might be worse than those but Perplexity is certainly a lesser player in the field.
Its a good thing I don't just block Perplexity, but all of the LLMs.
And I wont comment on the rest of this, but lets consider another form of property: Real estate.
You own a plot of land. Should others be able to use it, however they feel, whenever they feel like? Or should you have a say in how it gets used?
If you feel like you should have exclusive say in how real estate you own is used and when and by whom, why is intellectual property any different? There must be value in using it, so what's wrong with revenues generated by that use being shared (At least) with the creator?
Last I checked, I'm not seeing rev shares from any of these LLMs that have certainly used my code and other content to train?
they cant get their ai to check a box that says "I am not a robot"? I'd think thatd be a first year comp sci student level task. And robots.txt files were basically always voluntary compliance anyway.
Cloudflare actually fully fingerprints your browser and even sells that data. Thats your IP, TLS, operating system, full browser environment, installed extensions, GPU capabilities etc. It's all tracked before the box even shows up, in fact the box is there to give the runtime more time to fingerprint you.
This post refers to Google ReCaptcha v2 (not the latest version)
Recently Google introduced a simplified "captcha" verification system (video) that enables users to pass the "captcha" just by clic...
you're not wrong, but it also allows more than 99.8% of the bot traffic through too on text challenges. Its like the TSA of website security. Its mostly there to keep the user busy while cloudflare places itself in a man in the middle of your encrypted connection to a third party. The only difference between cloudflare and a malicious attacker is cloudflares stated intention not to be evil. With that and 3 dollars I can buy myself a single hard shell taco from tacobell.
When a firm outright admits to bypassing or trying to bypass measures taken to keep them out, you think that would be a slam dunk case of unauthorized access under the CFAA with felony enhancements.
Fuck that. I don't need prosecutors and the courts to rule that accessing publicly available information in a way that the website owner doesn't want is literally a crime. That logic would extend to ad blockers and editing HTML/js in an "inspect element" tag.
That logic would not extend to ad blockers, as the point of concern is gaining unauthorized access to a computer system or asset. Blocking ads would not be considered gaining unauthorized access to anything. In fact it would be the opposite of that.
Ehhhh, you are gaining access to content due to assumption you are going to interact with ads and thus, bring revenue to the person and/or company producing said content. If you block ads, you remove authorisation brought to you by ads.
Carefull, this way even not looking at an ads positioned at the bottom of the page (or anyway not visible without scrolling) would mean to remove authorisation brought to you by ads.
And my point is that defining "unauthorized" to include visitors using unauthorized tools/methods to access a publicly visible resource would be a policy disaster.
If I put a banner on my site that says "by visiting my site you agree not to modify the scripts or ads displayed on the site," does that make my visit with an ad blocker "unauthorized" under the CFAA? I think the answer should obviously be "no," and that the way to define "authorization" is whether the website puts up some kind of login/authentication mechanism to block or allow specific users, not to put a simple request to the visiting public to please respect the rules of the site.
To me, a robots.txt is more like a friendly request to unauthenticated visitors than it is a technical implementation of some kind of authentication mechanism.
Scraping isn't hacking. I agree with the Third Circuit and the EFF: If the website owner makes a resource available to visitors without authentication, then accessing those resources isn't a crime, even if the website owner didn't intend for site visitors to use that specific method.
Andrew “Weev” Auernheimer was convicted of violating the Computer Fraud and Abuse Act ("CFAA") in New Jersey federal court and sentenced to 41 months in federal prison in March of 2013 for revealing to media outlets that AT...
When sites put challenges like Anubis or other measures to authenticate that the viewer isn't a robot, and scrapers then employ measures to thwart that authentication (via spoofing or other means) I think that's a reasonable violation of the CFAA in spirit — especially since these mass scraping activities are getting attention for the damage they are causing to site operators (another factor in the CFAA, and one that would promote this to felony activity.)
The fact is these laws are already on the books, we may as well utilize them to shut down this objectively harmful activity AI scrapers are doing.
Nah, that would also mean using Newpipe, YoutubeDL, Revanced, and Tachiyomi would be a crime, and it would only take the re-introduction of WEI to extend that criminalization to the rest of the web ecosystem. It would be extremely shortsighted and foolish of me to cheer on the criminalization of user spoofing and browser automation because of this.
Do you think DoS/DDoS activities should be criminal?
If you're a site operator and the mass AI scraping is genuinely causing operational problems (not hard to imagine, I've seen what it does to my hosted repositories pages) should there be recourse? Especially if you're actively trying to prevent that activity (revoking consent in cookies, authorization captchas).
In general I think the idea of "your right to swing your fists ends at my face" applies reasonably well here — these AI scraping companies are giving lots of admins bloody noses and need to be held accountable.
I really am amenable to arguments wrt the right to an open web, but look at how many sites are hiding behind CF and other portals, or outright becoming hostile to any scraping at all; we're already seeing the rapid death of the ideal because of these malicious scrapers, and we should be using all available recourse to stop this bleeding.
DoS attacks are already a crime, so of course the need for some kind of solution is clear. But any proposal that gatekeeps the internet and restricts the freedoms with which the user can interact with it is no solution at all. To me, the openness of the web shouldn't be something that people just consider, or are amenable to. It should be the foundation in which all reasonable proposals should consider as a principle truth.
How “open” a website is, is up to the owner, and that’s all.
As someone who registered this account on this platform in response to Reddit's API restrictions, it would be hypocritical of me to accept such a belief.
Well, until we abolish capitalism, that’s the state of things.
I can see that things are the way things are. Accepting it is a different matter.
Unless you feel like Nazis MUST be freely given access to everything too?
To me, the "access" that I am referring to (the interface with which you gain access to a service) and that "access" (your behavior once you have gained access to a service) are different topics. The same distinction can be made with the concern over DoS attacks mentioned earlier in the thread. The user's behavior of overwhelming a site's traffic is the root concern, not the interface that the user is connecting with.
Site owners currently do and should have the freedom to decide who is and is not allowed to access the data, and to decide for what purpose it gets used for. Idgaf if you think scraping is malicious or not, it is and should be illegal to violate clear and obvious barriers against them at the cost of the owners and unsanctioned profit of the scrapers off of the work of the site owners.
Yeah, fuck everything about that. If I'm a site visitor I should be able to do what I want with the data you send me. If I bypass your ads, or use your words to write a newspaper article that you don't like, tough shit. Publishing information is choosing not to control what happens to the information after it leaves your control.
Don't like it? Make me sign an NDA. And even then, violating an NDA isn't a crime, much less a felony punishable by years of prison time.
Interpreting the CFAA to cover scraping is absurd and draconian.
If you want anybody and everyone to be able to use everything you post for any purpose, right on, good for you, but don't try to force your morality on others who rely on their writing, programming, and artworks to make a living to survive.
I'm gonna continue to use ad blockers and yt-dlp, and if you think I'm a criminal for doing so, I'm gonna say you don't understand either technology or criminal law.
Thats a crime yeah and if Alphabet co wants to sue you for $1.34 damages then they have that right, just as we should have the right to sue them if their AI crawlers make our site unusable and plagiarize our work to the effect of thousands of dollars, or even press charges for the criminal act of intentional disruption of services.
Thats a crime yeah and if Alphabet co wants to sue you for $1.34 damages then they have that right
So yeah, I stand by my statement that anyone thinks this is a crime, or should be a crime, has a poor understanding of either the technology or the law. In this case, even mentioning Alphabet suing for damages means that you don't know the difference between criminal law and civil law.
press charges for the criminal act of intentional disruption of services
That's not a crime, and again reveals gaps in your knowledge on this topic.
Who said anything about DDoS? I'm using ad blockers and saving/caching/archiving websites with a single computer, and not causing damage. I'm just using the website in a way the owner doesn't like. That's not a crime, nor should it be.
just as we should have the right to sue them if their AI crawlers make our site unusable and plagiarize our work to the effect of thousands of dollars, or even press charges for the criminal act of intentional disruption of services.
I've developed my own agent for assisting me with researching a topic I'm passionate about, and I ran into the exact same barrier: Cloudflare intercepts my request and is clearly checking if I'm a human using a web browser. (For my network requests, I've defined my own user agent.)
So I use that as a signal that the website doesn't want automated tools scraping their data. That's fine with me: my agent just tells me that there might be interesting content on the site and gives me a deep link. I can extract the data and carry on my research on my own.
I completely understand where Perplexity is coming from, but at scale, implementations like ~~this~~ Perplexity's are awful for the web.
I hate to break it to you but not only does Cloudflare do this sort of thing, but so does Akamai, AWS, and virtually every other CDN provider out there. And far from being awful, it’s actually protecting the web.
We use Akamai where I work, and they inform us in real time when a request comes from a bot, and they further classify it as one of a dozen or so bots (search engine crawlers, analytics bots, advertising bots, social networks, AI bots, etc). It also informs us if it’s somebody impersonating a well known bot like Google, etc. So we can easily allow search engines to crawl our site while blocking AI bots, bots impersonating Google, and so on.
What I meant with "things like this are awful for the web," I meant that automation through AI is awful for the web. It takes away from the original content creators without any attribution and hits their bottom line.
My story was supposed to be one about responsible AI, but somehow I screwed that up in my summary.
You'd think that a competent technology company, with their own AI would be able to figure out a way to spoof Cloudflare's checks. I'd still think that.
Or find a more efficient way to manage data, since their current approach is basically DDOSing the internet for training data and also for responding to user interactions.
Perplexity argues that Cloudflare is mischaracterizing AI Assistants as web crawlers, saying that they should not be subject to the same restrictions since they are user-initiated assistants.
Personally I think that claim is a decent one: user-initiated request should not be subject to robot limitations, and are not the source of DDOS attack to web sites.
I think the solution is quite clear, though: either make use of the user identity to walz through the blocks, or even make use of the user browser to do it. Once a captcha appears, let the user solve it.
Though technically making all this happen flawlessly is quite a big task.
Personally I think that claim is a decent one: user-initiated request should not be subject to robot limitations, and are not the source of DDOS attack to web sites.
They are one of the sources!
The AI scraping when a user enters a prompt is DDOSing sites in addition to the scraping for training data that is DDOSing sites. These shitty companies are repeatedly slamming the same sites over and over again in the least efficient way because they are not using the scraped data from training when they process a user prompt that does a web search.
Scraping once extensively and scraping a bit less but far more frequently have similar impacts.
When user enters a prompt, the backend may retrieve a handful a pages to serve that prompt. It won't retrieve all the pages of a site. Hardly different from a user using a search engine and clicking 5 topmost links into tabs. If that is not a DoS attack, then an agent doing the same isn't a DDoS attack.
Constructing the training material in the first place is a different matter, but if you're asking about fresh events or new APIs, the training data just doesn't cut it. The training, and subsequenctly the material retrieval, has been done a long time ago.
puts on evil hat CloudFlare should DRM their protection then DMCA Perplexity and other US based "AI" companies to oblivion. Side effect, might break the Internet.
It's insane that anyone would side with Cloudflare here. To this day I cant visit many websites like nexusmods just because I run Firefox on Linux. The Cloudflare turnstile just refreshes infinitely and has been for months now.
Cloudflare is the biggest cancer on the web, fucking burn it.
I'm on Linux with Firefox and have never had that issue before (particularly nexusmods which I use regularly). Something else is probably wrong with your setup.
Linux and Firefox here. No problem at all with Cloudflare, despite having more or less as much privacy preserving add-on as possible. I even spoof my user agent to the latest Firefox ESR on Linux.
Thats not how it works. Cf uses thousands of variables to estimate a trust score and block people so just because it works for you doesn't mean it works.
Same goes the other way. It's not because it doesn't work for you that it should go away.
That technology has its uses, and Cloudflare is probably aware that there are still some false positive, and probably is working on it as we write.
The decision is for the website owner to take, taking into consideration the advantages of filtering out a majority of bots and the disadvantages of loosing some legitimate traffic because of false positives. If you get Cloudflare challenge, chances are that he chosed that the former vastly outclass the later.
Now there are some self-hosted alternatives, like Anubis, but business clients prefer SaaS like Cloudflare to having to maintain their own software. Once again it is their choices and liberty to do so.
lmao imagine shilling for corporate Cloudflare like this. Also false positive vs false negative are fundamentally not equal.
Cloudflare is probably aware that there are still some false positive, and probably is working on it as we write.
The main issue with Cloudflare is that it's mostly bullshit. It does not report any stats to the admins on how many users were rejected or any false positive rates and happily put's everyone under "evil bot" umbrella. So people from low trust score environments like Linux or IPs from poorer countries are under significant disadvantage and left without a voice.
I'm literally a security dev working with Cloudflare anti-bot myself (not by choice). It's a useful tool for corporate but a really fucking bad one for the health of the web, much worse than any LLM agent or crawler, period.
I suspect a lot of it comes down to your ISP. Like the original commentor I also frequently can't pass CloudFlare turnstile when on Wifi, although refreshing the page a few times usually gets me through. Worst case on my phone's hotspot I can much more consistently pass. It's super annoying and combined with their recent DNS outage has totally ruined any respect I had for CloudFlare.
Profitez des vidéos et de la musique que vous aimez, mettez en ligne des contenus originaux, et partagez-les avec vos amis, vos proches et le monde entier.
Perplexity argues that a platform’s inability to differentiate between helpful AI assistants and harmful bots causes misclassification of legitimate web traffic.
So, I assume Perplexity uses appropriate identifiable user-agent headers, to allow hosters to decide whether to serve them one way or another?
No, as per the article, their argumentation is that they are not web crawlers generating an index, they are user-action-triggered agents working live for the user.
You could probably route all requests to your site from them, back at themselves, so they DDoS themselves, and on top off it, cost them more because their endpoint needs to process things via their LLM.
The amount of people just reacting to the headline in the comments on these kinds of articles is always surprising.
Your browser acts as an agent too, you don’t manually visit every script link, image source and CSS file. Everyone has experienced how annoying it is to have your browser be targeted by Cloudflare.
There’s a pretty major difference between a human user loading a page and having it summarized and a bot that is scraping 1500 pages/second.
Cheering for Cloudflare to be the arbiter of what technologies are allowed is incredibly short sighted. They exist to provide their clients with services, including bot mitigation. But a user initiated operation isn’t the same as a bot.
Which is the point of the article and the article’s title.
It isn’t clear why OP had to alter the headline to bait the anti-ai crowd.
But a user initiated operation isn’t the same as a bot.
Oh fuck off with that AI company propaganda.
The AI companies already overwhelmed sites to get training data and are repeating their shitty scraping practices when users interact with their AI. It's the same fucking thing.
Web crawlers for search engines don't scrape pages every time a user searches like AI does. Both web crawlers and scrapers are bots, and how a human initiates their operation, scheduled or not, doesn't matter as much as the fact that they do things very differently and only one of the two respects robots.txt.
There’s no difference in server load between a user looking at a page and a user using an AI tool to summarize the page.
The AI companies already overwhelmed sites to get training data and are repeating their shitty scraping practices when users interact with their AI. It’s the same fucking thing.
You either didn’t read the article or are deliberately making bad faith arguments. The entire point of the article is that the traffic that they’re referring to is initiated by a user, just like when you type an address into your browser’s address bar.
This traffic, initiated by a user, creates the same server load as that same user loading the page in a browser.
Yes, mass scraping of web pages creates a bunch of server load. This was the case before AI was even a thing.
This situation is like Cloudflare presenting was a captcha in order to load each individual image, css or JavaScript asset into a web browser because bot traffic pretends to be a browser.
I don’t think it’s too hard to understand that a bot pretending to be a browser and a human operated browser are two completely different things and classifying them as the same (and captchaing them) would be a classification error.
This is exactly the same kind of error. Even if you personally believe that users using AI tools should be blocked, not everyone has the same opinion. If Cloudflare can’t distinguish between bot requests and human requests then their customers can’t opt out and allow their users to use AI tools even if they want to.
It's been this from the very beginning. But they don't fit the definition of a protection racket as they're not the ones attacking you if you don't pay up. So they're more like a security company that has no competitors due to the needed investment to operate.
Alles richtig gemacht! Was soll ich sonst schreiben, über den Geburtstagsfilm, den der SWR dem größten Mülheimer Genie der Gegenwart im Auftrag der ARD hat widmen lassen? Angesichts der Unmöglichkeit der gestellten Aufgabe, haben sie dort glücklicherweise kollektiv entschieden, den Künstler das Werk doch lieber selbst anfertigen zu lassen, bevor die Sendeanstalt sich der Peinlichkeit einer weiteren öffentlich-rechtlichen Hagiographie die dann doch nicht mehr als ein Recycling alter Talkshows und Sketche geworden wäre. (ARD, Neu!)
Alles richtig gemacht! Was soll ich sonst schreiben, über den Geburtstagsfilm, den der SWR dem größten Mülheimer Genie der Gegenwart im Auftrag der ARD hat widmen lassen? Angesichts der Unmöglichkeit der gestellten Aufgabe, haben sie dort glücklicher…
It's not particularly new (except to me) or that shiny... but I got a new gfx card today. I bought it mainly for the nv remove background functionality and because it will be much faster at other tasks too! I am only about 5 years behind the times at least, rather than the 10 I normally am....
Provo a chiedere qui dove probabilmente molti di voi sono interessati all'argomento. Siete a conoscenza di buone risorse, in Italiano o in Inglese, per imparare a riportare dati non finanziari secondo la direttiva UE CSRD per il reporting non-finanziario (incluso ambientale) in ambito EEA?
Sono alla ricerca di corsi, tutorial e altri materiali VERAMENTE informativi. Cercando online ho solo trovato una pletora di articoli scritti con ChatGPT che non vanno a parare da nessuna parte.
A team of Cornell computer science researchers has developed a way to “watermark” light in videos, which they can use to detect if video is fake or has been manipulated, another potential tool in the fight against misinformation.
The U.S. Federal Trade Commission sued ticket reseller Key Investment Group for evading purchasing limits to buy up thousands of tickets to live events including Taylor Swift's Eras tour and resell them at a markup, according to a complaint filed in Maryland federal court on Monday.
Disclaimer: The article linked is from a single source with a single perspective. Make sure to cross-check information against multiple sources to get a comprehensive view on the situation.
Disclaimer: The article linked is from a single source with a single perspective. Make sure to cross-check information against multiple sources to get a comprehensive view on the situation.
Disclaimer: The article linked is from a single source with a single perspective. Make sure to cross-check information against multiple sources to get a comprehensive view on the situation.
Dame Rachel de Souza warns it is ‘absolutely a loophole that needs closing’ as new report finds proportion of children who report accessing pornography online has increased over last two years
Syrian media reported on Monday that Israeli forces carried out a raid inside the village of Ain Ziwan in southern Quneitra countryside, searching several civilian homes.
Disclaimer: The article linked is from a single source with a single perspective. Make sure to cross-check information against multiple sources to get a comprehensive view on the situation.
The Norwegian sovereign wealth fund, the largest in the world, said on Monday that it has decided to exclude six Israeli companies from its investment portfolio.
Disclaimer: The article linked is from a single source with a single perspective. Make sure to cross-check information against multiple sources to get a comprehensive view on the situation.
Technos Media is dedicated to bringing you the latest advancements in technology, science, and business. Our platform offers a rich blend of news, reviews, in-depth analyses, and opinion pieces, covering a broad spectrum of topics. We’re committed to keeping you updated on the latest technological wonders, as well as groundbreaking scientific and medical discoveries. Our enthusiastic and knowledgeable editorial team strives to make complex developments accessible and engaging for our readers. Based in the vibrant heart of New York City, we operate at the center of digital media innovation, with a diverse group of skilled editors and writers from the US, Europe, and Asia. Our team works tirelessly to provide timely and insightful content, shaping our understanding of technology, science, and health developments worldwide.
Disclaimer: The article linked is from a single source with a single perspective. Make sure to cross-check information against multiple sources to get a comprehensive view on the situation.
Trudeau started using this vague section to bypass legislating but Carney is doubling down on it. And Carney only won because of Trump's direct attacks on Canada's sovereignty. And even on that front he is failing. He's acting like US bitch and at the same time trying to pass massive surveillance laws akin to US FISA to share the data with the US.
U.S. President Donald Trump has interrupted his talks in Washington with European leaders to call Russian President Vladimir Putin, an EU diplomat told Reuters on Monday.
Disclaimer: The article linked is from a single source with a single perspective. Make sure to cross-check information against multiple sources to get a comprehensive view on the situation.
Japan's first osmotic power plant that uses the difference in salt concentration between seawater and fresh water to generate electricity began operations in early August in a southwestern prefecture.
The details of property developer Amir Abu Abara's lengthy court case can only now be revealed — five years after he illegally cleared land at Barden Ridge in Sydney's south.
Disclaimer: The article linked is from a single source with a single perspective. Make sure to cross-check information against multiple sources to get a comprehensive view on the situation.
Disclaimer: The article linked is from a single source with a single perspective. Make sure to cross-check information against multiple sources to get a comprehensive view on the situation.
Frustrated by rising subscription costs and fragmented content availability, viewers worldwide are returning to piracy at unprecedented levels, reversing years of progress made by affordable streaming services. Recent data from London-based monitoring firm MUSO shows piracy visits skyrocketed from 130 billion in 2020 to 216 billion by 2024, with the industry facing projected losses exceeding $113 billion.
Subscription Fatigue Drives Digital Exodus
The streaming landscape has transformed from Netflix's early promise of "everything in one place" into what critics call "Cable 2.0"—a fractured ecosystem requiring multiple subscriptions. According to The Guardian, the average European household now spends close to €700 annually on three or more video-on-demand subscriptions. With Netflix's standard plan reaching $15.49 monthly and competitors following suit, consumers are increasingly viewing piracy as a rational alternative.
"Piracy is not a pricing issue, it's a service issue," Valve co-founder Gabe Newell observed in 2011—a prediction that appears prophetic as streaming platforms struggle with content fragmentation and rising prices. In Sweden, birthplace of both Spotify and The Pirate Bay, 25% of people surveyed admitted to pirating content in 2024, predominantly driven by those aged 15 to 24.
Content Wars Create Consumer Casualties
The fragmentation crisis has worsened as studios create exclusive content silos. Viewers face scenarios where favorite shows vanish from one platform only to appear on another, or require separate purchases despite existing subscriptions. Even purchased content can become unavailable due to licensing disputes, prompting consumer lawsuits against platforms like Amazon Prime Video.
MUSO data reveals that unlicensed streaming now accounts for 96% of all TV and film piracy, representing a fundamental shift in how content theft occurs. Modern pirates leverage sophisticated tools including AI-driven search engines and encrypted networks that adapt faster than anti-piracy measures can respond.
Industry Scrambles for Solutions
Streaming executives are experimenting with bundled offerings and cracking down on password sharing, but these measures often backfire by further alienating users. According to Antenna research, one-quarter of U.S. streamers are "chronic churners," frequently canceling subscriptions due to cost and frustration.
The resurgence marks a stark reversal from the mid-2010s when convenient, affordable streaming services nearly eliminated piracy. As one industry analyst noted, studios have created "artificial scarcity in a digital world that promised abundance", suggesting that without addressing core affordability and access issues, the piracy revival may continue reshaping entertainment consumption patterns.
Frustrated by rising subscription costs and fragmented content availability, viewers worldwide are returning to piracy at unprecedented levels, reversing years of progress made by affordable streaming services. Recent data from London-based monitoring firm MUSO shows piracy visits skyrocketed from 130 billion in 2020 to 216 billion by 2024, with the industry facing projected losses exceeding $113 billion.
Subscription Fatigue Drives Digital Exodus
The streaming landscape has transformed from Netflix's early promise of "everything in one place" into what critics call "Cable 2.0"—a fractured ecosystem requiring multiple subscriptions. According to The Guardian, the average European household now spends close to €700 annually on three or more video-on-demand subscriptions. With Netflix's standard plan reaching $15.49 monthly and competitors following suit, consumers are increasingly viewing piracy as a rational alternative.
"Piracy is not a pricing issue, it's a service issue," Valve co-founder Gabe Newell observed in 2011—a prediction that appears prophetic as streaming platforms struggle with content fragmentation and rising prices. In Sweden, birthplace of both Spotify and The Pirate Bay, 25% of people surveyed admitted to pirating content in 2024, predominantly driven by those aged 15 to 24.
Content Wars Create Consumer Casualties
The fragmentation crisis has worsened as studios create exclusive content silos. Viewers face scenarios where favorite shows vanish from one platform only to appear on another, or require separate purchases despite existing subscriptions. Even purchased content can become unavailable due to licensing disputes, prompting consumer lawsuits against platforms like Amazon Prime Video.
MUSO data reveals that unlicensed streaming now accounts for 96% of all TV and film piracy, representing a fundamental shift in how content theft occurs. Modern pirates leverage sophisticated tools including AI-driven search engines and encrypted networks that adapt faster than anti-piracy measures can respond.
Industry Scrambles for Solutions
Streaming executives are experimenting with bundled offerings and cracking down on password sharing, but these measures often backfire by further alienating users. According to Antenna research, one-quarter of U.S. streamers are "chronic churners," frequently canceling subscriptions due to cost and frustration.
The resurgence marks a stark reversal from the mid-2010s when convenient, affordable streaming services nearly eliminated piracy. As one industry analyst noted, studios have created "artificial scarcity in a digital world that promised abundance", suggesting that without addressing core affordability and access issues, the piracy revival may continue reshaping entertainment consumption patterns.
Anything but solve the main issue: pirate sites offer a better service, with no stupid licensing problems, having everything on a single app and without geolocking bullshit.
When the pirate alternative it's not just cheaper but also way more convenient, it's no wonder they are losing customers.
That was my reason for cancelling. I can afford the service, but it better let me stream at full resolution through LibreWolf or I’m just gonna download the movies and shows.
I absolutely love rubbing it in the face of people subscribing to streaming services that I get more content without even having to enter in my credit card info.
If three or four people did that and shared their library with friends then they would have a massive catalog between them.
Legally, you are allowed to make backups of the media you own (in the US). If a group of friends got together and shared their media libraries, proving that only one person was accessing the media at a time, then there would be no issue.
I can literally pay for every service available in my area and still not watch everything (not to mention anything) I want. Geo-blocking as well as expired rights made me never want to pay for one of those overpriced services again as there is one option that's way more convenient, and by chance much cheaper too.
There’s just too many shit shows packaged along a few gems to make subscribing to the whole lot worthwhile.
And as a Canadian right now it feels downright patriotic to divert money from US-owned streaming companies who region-lock my purchases so I have to buy terrestrial cable to view them.
No mention of advertising in streaming services now? That was a big factor in making me shut of my subscriptions. The price increases made me cut back but the ads have made me turn them off. I'm not paying to watch commercials.
Yeah I was watching a show recently on Prime and now they are injecting 1min plus long ads like every 10-20 minutes. It's worse than watching cable tv.
There’s an easy solution to this. I pay for Apple Music because I get access to pretty much all the music I want. I can sideload what they don’t have, which isn’t much. They have better audio quality, and aren’t stiffing artists to pay some right wing nutjob science denier like the other streaming platform of note. I pay because I love music and want to support what I love. Why isn’t there a similar service for TV and movies? That’s the solution. Let us pay for what we love and make it easy. Apple figured it out with music. Valve figured it out with games.
I think they don’t want to solve the problem. I think they want to solve a different problem. I think they’re making this a problem so they can push legislation to protect their profits.
Absolutely. But to clarify, Apple Music pays more per stream than Spotify and others. Spotify trends to cut bigger checks to popular musicians because they have more subscribers.
Also — someone feel free to fact check this — I’ve heard that if, say, you put an album out on BandCamp but not streaming, and I buy it and sideload it into both services, and you later add it to both services, Spotify won’t pay you for my streams because I’m streaming the sideloaded copy whereas Apple will match it. I keep the metadata if it’s different but you’d get paid for the streams because it matches it.
Why isn’t there a similar service for TV and movies?
Because the Artists involved don't see the royalties in those industries. They've already been paid and the rights holders want to extract every drop of profit possible. and the sad truth is that splintering streaming worked for a very long time to this end.
Why isn’t there a similar service for TV and movies?
Because Steve Jobs died before he could hypnotize the executives into do it.
Only half kidding, people forget that's how we got all the music labels together, it was Apple iTunes Store which later became streaming.
There were a few weird awkward music stores but Apple did it better and this was early 2000's when their brand was barely recovering from near bankruptcy.
Netflix streaming didn't launch until 2007 and didn't really take off until 2011, the same year Jobs died.
Netflix was the only one that was positioned to do it but they couldn't pull it together because they didn't have the reality distortion field that Jobs had. Netflix had to push into original programming instead to survive and the brand has long since enshittified.
Can't help but wonder if whatever pitch Jobs used to sell record labels couldn't have been reworked to convince the movie studios.
Here’s what’s wild though. At first with music streaming it was largely just American, Western, popular music. I left Spotify for Apple Music because the latter had Japanese music and I was tired of sideloading it into Spotify. Now Spotify has Japanese music too.
The Japanese music market is super weird. Anime is to Japanese music in the 2010s and 2020s what MTV was to western music in the ‘80s and ‘90s. It’s the international hit maker. So anime is bringing western eyes to all this music, not you go in YouTube and a lot of them have “YouTube edition” videos that are like half the video. Because they don’t fully trust us I guess? Sometimes the video is on Apple Music though.
I know Japanese music is more expensive than ours. I mean like the cost of a CD. So when bands would release a Japanese album, they’d add bonus tracks to help increase the value. Western bands do it too. Look up an album you know on Wikipedia and see if there’s a Japanese version with some bonus tracks.
I’m wondering how Apple Music and later Spotify more or less tamed the Japanese music market but TV and movies are so much harder.
“We think there is a fundamental misconception about piracy. Piracy is almost always a service problem and not a pricing problem. If a pirate offers a product anywhere in the world, 24 x 7, purchasable from the convenience of your personal computer, and the legal provider says the product is region-locked, will come to your country 3 months after the US release, and can only be purchased at a brick and mortar store, then the pirate’s service is more valuable.”
GabenN
When what you're selling isn't appealing, then people won't buy it. It's not super complicated.
Realistically, if there was a service that had everything on it that was past the cinema/pay per view stage, and was a reasonable price (say £35 a month, the price of two current streaming services), then I would probably be on that instead of Jellyfin.
And I mean everything back to the dawn of time. Anything you want, TV series, movies, the lot. Original versions, directors cuts, etc. George fucking Lucas, I'm looking at you here.
But there isn't. There never will be. Because they're all in a race to grab as much money as they can, before literal heat death engulfs the whole planet.
They could make a Steam style store for movies to buy individual movies that then stay on your account. Deep deep discounts on some oldie movies intermittently. People would eat that shit up and buy whole movie libraries to keep. Way more movies than they could watch (like people do with a Steam library).
I've got my server more stable now so it works a lot more reliably for Jellyfin. Before now, my wife would wholeheartedly agree with this and insist on paying for Netflix
I recently upgraded my Jellyfin server from GTX 970 to GTX 1660 because I wanted to have HEVC 10bit support for transcoding on the fly (40 Mbps uplink is not ideal), it cost me $60 and I sold my old one for $20 to a buddy with GTX 600ish.
Maybe she doesn't know it, but how she reacts to this situation is my main test of her suitability as my partner. Nothing is more attractive to me than a woman who can just chill for 15 minutes without complaint while I exportfs -a and restart nfsd.
This is the Spotify/Apple Music/etc model and the reason why music piracy is practically dead (yes, I know there are a few sites still going).
These services are doing their best to find ways to push people back to piracy but for now they keep it at bay through competition to provide better service.
If there was a catchall video streaming service where all publishers released and got a cut of their plays it would be game over for piracy. Fortunately that'll never happen.
I don't think I've pirated a game since I started working and actually spending on Steam. Except Borderlands 3, because fuck Epic Games Store, their dumb fucking exclusivity deals, and their shitty launcher - they won't EVER see a penny from me.
I bought it on Steam a year later when it came out, on sale, with DLCs.
When the price was $9 a month I'd forget it even existed and go months without using a particular service I was paying for. But I didn't mind it because sometimes I'd get the urge to go watch some cult classic I hadn't seen in awhile, or guests could use it.
And I could eat that $9 each month because it was the equivalent of skipping one McDonald's burger a month.
Now? I use them one at a time and pirate anything not on that service. And if the cheapest option they have is ad-supported I skip paying entirely and head straight out to sea.
Of course they had it good. Studios didn't understand streaming and so sold long term rights to Netflix for pennies. Now, everyone is cutting out cable, making it impossible for those studios to fund new production on cable and broadcast alone.
New production is going to hit a cliff as streaming isn't going to be able to fund golden era of television any more.
As subscription costs rise and choice diminishes on legal sites, film and TV fans are turning to VPNs and illicit streamers, with Sweden – home of both Spotify and The Pirate Bay – leading the way
Hey, the Supreme Court said it’s cool for LLMs to pirate stuff for their bullshit so it’s free game. I’m not paying for shit. Just like that bullshit article that has been posted, which I won’t open
I am working on a prototype purely organic LLM called myself. In order to be able to interact with prompts that include popular media references I will train the model on TV shows and movies.
Training can be reinforced by opining on said media at work, with friends and online. Feedback from these other models in the form of their own wrong opinions about the media reinforces one’s own model to always be right.
I find this so ironic. For so many years the US has been sort of the anti-piracy centre of the world. Now their just blatantly allowing piracy for the sake of progress.
Hopefully this won't get me too much negative reaction: I'm not a proud pirate. I'd rather not pirate at all. I'm kind of ashamed that it's come to this.
There were a few solid years where I literally did not do it and felt no desire to, back when streaming was new, and there were only a few serious players. I'd love to return to that era, but I know it will never exist again.
So now, I and other members of my family, pay a ridiculous amount of money for a rotating suite of services, trying to do things the right way, and still, there are way too many times when we can't find anything we want to watch on any of those services and/or the thing we wanted to watch is not available on any of those services.
Finally broke down and just said fuck it. I tried to support this mess as best I could in hopes it would get better, but fully knowing it wouldn't. When it definitely did not get better I said no more.
you could account share? most services banned account sharing last year afaik.
and anyway; how is it a bad thing to reject predatory services?
piracy wad literally always a service problem. they solved it by netflix monopoly providing a great service people wanted and piracy was practically dead for an entire decade - then they all got greedy and we're all headed back to piracy.
the solution here seems pretty obvious, and i think even the execs are aware of it.
It literally is. They solved music piracy by adopting all in one licensing with services like Spotify. If they released a subscription platform that included all movies and tv shows for a reasonable price I'd instantly sub. Instead we get the current enshittification mess.
I was recently travelling and tried to sign up for Netflix and couldn't. Because I didn't have a phone number in the country I was in. And they block vpns.
You can see where this is headed. Mullvad needs the money more than Netflix anyway.
I received an email from them last week, telling me my current plan is no longer available and I either switch to a more premium one or a lower ad-driven tier.
That’s atrocious and will be the final nail in my subscription. I’m already running a jellyfin powered infrastructure and I will cancel my Netflix. Screw them, Black Mirror loving MFs.
I've been using this setup for a bit. Super simple and hassle free. I enjoy self hosting but in terms of simplicity, you really cannot beat Stremio combined with a debrid sub.
Better to use a general foss redirect extension than slowly accumulate various service-specific ones. Libredirect natively does xitter -> xcancel, bluesky -> skyview, fandom -> breeze, reddit -> redlib, and like 50 more. All with tons of prefilled mirrors and uptests ...
It's a great place to find those alternative frontends. If you're annoyed by some enshittified platform, good chance you can just enable a redirect in there.
Possibly with custom hardware. Cartridges are essentially glorified SD cards. However, they would need a dump of the decryption key, which is not possible to obtain yet. This is most likely fake.
Two ways: undisclosed software exploit, or decapping and using an electron microscope. The former would be a world first as AFAIK no hacking group has claimed to be able to do that yet, and the latter is very expensive and extremely hard.
As there is (prove me if I'm wrong) no current way to dump and decrypt Switch2 games this is a highly probable hoax. I wouldn't consider this to be true and be extermely careful.
Bullshit, this is a Scam. There is no know way to dump ROMs or get the decryption key. How are they archiving this without a Hack for the Switch 2? The only way would be if they had their own hack and their own emulator, both developed in 2 Month.
The United Nations Security Council began to debate on Monday a resolution drafted by France to extend the UN peacekeeping force in south Lebanon for a year with the ultimate aim of withdrawing it.
Disclaimer: The article linked is from a single source with a single perspective. Make sure to cross-check information against multiple sources to get a comprehensive view on the situation.
::: spoiler Disable JavaScript to Access. 1. Open Chrome Settings: Click the three-dot menu (Customize and control Google Chrome) in the top-right corner and select "Settings". 2. Navigate to Site Settings: Go to "Privacy and security" and then click on "Site settings". 3. Find JavaScript Settings: Scroll down to the "Content" section and click on "JavaScript". 4. Disable JavaScript: Toggle the switch to "Don't allow sites to use JavaScript". :::
Disclaimer: The article linked is from a single source with a single perspective. Make sure to cross-check information against multiple sources to get a comprehensive view on the situation.
Israeli media has reported that a senior official in the Israel National Cyber Directorate was arrested in Las Vegas on suspicion of online paedophilia.
If you've read about the sudden appearance of age verification across the internet in the UK and thought it would never happen in the U.S., take note: many politicians want the same or even more strict laws.
kittenzrulz123
in reply to Davriellelouna • • •null
in reply to Davriellelouna • • •Sturgist
in reply to null • • •💁u
Here, you dropped this!
Wispy2891
in reply to Davriellelouna • • •WolfLink
in reply to Davriellelouna • • •pyre
in reply to WolfLink • • •int32
in reply to pyre • • •pressanykeynow
in reply to int32 • • •int32
in reply to pressanykeynow • • •pressanykeynow
in reply to int32 • • •int32
in reply to pressanykeynow • • •then make all links to your website link to that snapshot, and turn your server off.
pressanykeynow
in reply to int32 • • •int32
in reply to pressanykeynow • • •turmoil
in reply to int32 • • •int32
in reply to turmoil • • •sorry archive.org, I promise I'll donate ❤️
Block Cloudflare MITM Attack – Get this Extension for 🦊 Firefox (en-US)
addons.mozilla.orgoppy1984
in reply to pyre • • •ubergeek
in reply to oppy1984 • • •JcbAzPx
in reply to ubergeek • • •oppy1984
in reply to ubergeek • • •I get the centralization concerns, but I would think that's on the consumer since there are other options. As for the fascist content, as another commenter said, they could risk their safe harbor if they started stated regulating content that they weren't legally required to regulate.
Just my thoughts.
TheGrandNagus
in reply to Davriellelouna • • •sunbeam60
in reply to TheGrandNagus • • •They’re not. They’re using this as an excuse to become paid gatekeepers of the internet as we know it. All that’s happening is that Cloudflare is using this to menuever into position where they can say “nice traffic you’ve got there - would be a shame if something happened to it”.
AI companies are crap.
What Cloudflare is doing here is also crap.
And we’re cheering it on.
DreamlandLividity
in reply to TheGrandNagus • • •floquant
in reply to Davriellelouna • • •BigFig
in reply to floquant • • •Tollana1234567
in reply to BigFig • • •☂️-
in reply to Tollana1234567 • • •scarabic
in reply to BigFig • • •panda_abyss
in reply to Davriellelouna • • •I actually agree with them
This feels like cloudflare trying to collect rent from both sides instead of doing what’s best for the website owners.
There is a problem with AI crawlers, but these technologies are essentially doing a search, fetching a several pages, scanning/summarizing them, then presenting the findings to the user.
I don’t really think that’s wrong, it’s just a faster version of rummaging through the SEO shit you do when you Google something.
(I’ve never used perplexity, I do use Kagi’s ki assistant for similar search. It runs 3 searches and scans the top results and then provides citations)
drspod
in reply to panda_abyss • • •panda_abyss
in reply to drspod • • •AstralPath
in reply to panda_abyss • • •Pennomi
in reply to AstralPath • • •Tollana1234567
in reply to Pennomi • • •kopasz7
in reply to panda_abyss • • •Search engines been going relatively fine for decades now. But the crawlers from AI companies basically DDOS hosts in comparison, sending so many requests in such a short interval. Crawling dynamic links as well that are expensive to render compared to a static page, ignoring the robots.txt entirely, or even using it discover unlinked pages.
Servers have finite resources, especially self hosted sites, while AI companies have disproportinately more at their disposal, easily grinding other systems to a halt by overwhelming them with requests.
Tollana1234567
in reply to kopasz7 • • •Ekybio
in reply to Davriellelouna • • •panda_abyss
in reply to Ekybio • • •Cloudflare runs as a CDN/cache/gateway service in front of a ton of websites. Their service is to help protect against DDOS and malicious traffic.
A few weeks ago cloudflare announced they were going to block AI crawling (good, in my opinion). However they also added a paid service that these AI crawlers can use, so it actually becomes a revenue source for them.
This is a response to that from Perplexity who run an AI search company. I don’t actually know how their service works, but they were specifically called out in the announcement and Cloudflare accused them of “stealth scraping” and ignoring robots.txt and other things.
very_well_lost
in reply to panda_abyss • • •I think it's also worth pointing out that all of the big AI companies are currently burning through cash at an absolutely astonishing rate, and none of them are anywhere close to being profitable. So pay-walling the data they use is probably gonna be pretty painful for their already-tortured bottom line (good).
Tollana1234567
in reply to very_well_lost • • •BetaDoggo_
in reply to Ekybio • • •Perplexity (an "AI search engine" company with 500 million in funding) can't bypass cloudflare's anti-bot checks. For each search Perplexity scrapes the top results and summarizes them for the user. Cloudflare intentionally blocks perplexity's scrapers because they ignore robots.txt and mimic real users to get around cloudflare's blocking features. Perplexity argues that their scraping is acceptable because it's user initiated.
Personally I think cloudflare is in the right here. The scraped sites get 0 revenue from Perplexity searches (unless the user decides to go through the sources section and click the links) and Perplexity's scraping is unnecessarily traffic intensive since they don't cache the scraped data.
lividweasel
in reply to BetaDoggo_ • • •That seems almost maliciously stupid. We need to train a new model. Hey, where’d the data go? Oh well, let’s just go scrape it all again. Wait, did we already scrape this site? No idea, let’s scrape it again just to be sure.
rdri
in reply to lividweasel • • •ubergeek
in reply to rdri • • •I think it boils down to "consent" and "remuneration".
I run a website, that I do not consent to being accessed for LLMs. However, should LLMs use my content, I should be compensated for such use.
So, these LLM startups ignore both consent, and the idea of remuneration.
Most of these concepts have already been figured out for the purpose of law, if we consider websites much akin to real estate: Then, the typical trespass laws, compensatory usage, and hell, even eminent domain if needed ie, a city government can "take over" the boosted post feature to make sure alerts get pushed as widely and quickly as possible.
rdri
in reply to ubergeek • • •That all sounds very vague to me, and I don't expect it to be captured properly by law any time soon. Being accessed for LLM? What does it mean for you and how is it different from being accessed by a user? Imagine you host a weather forecast. If that information is public, what kind of compensation do you expect from anyone or anything who accesses that data?
Is it okay for a person to access your site? Is it okay for a script written by that person to fetch data every day automatically? Would it be okay for a user to dump a page of your site with a headless browser? Would it be okay to let an LLM take a look at it to extract info required by a user? Have you heard about changedetection.io project? If some of these sound unfair to you, you might want to put a DRM on your data or something.
Would you expect a compensation from me after reading your comment?
ubergeek
in reply to rdri • • •It already has been captured, properly in law, in most places. We can use the US as an example: Both intellectual property and real property have laws already that cover these very items.
Well, does a user burn up gigawatts of power, to access my site every time? That's a huge different.
Depends on the terms of service I set for that service.
Sure!
Sure! As long as it doesn't cause problems for me, the creator and hoster of said content.
See above. Both power usage and causing problems for me.
No. I said, I do not want my content and services to be used by and for LLMs.
I have now. And should a user want to use that service, that service, which charges 8.99/month for it needs to pay me a portion of that, or risk having their service blocked.
There no need to use it, as I already provide RSS feeds for my content. Use the RSS feed, if you want updates.
Or, I can just block them, via a service like Cloud Flare. Which I do.
None. Unless you're wanting to access if via an LLM. Then I want compensation for the profit driven access to my content.
rdri
in reply to ubergeek • • •And it causes a lot of trouble to many people and pains me specifically. Information should not be gated or owned in a way that would make it illegal for anyone to access it under proper conditions. License expiration causing digital work to die out, DRM causing software to break, idiotic license owners not providing appropriate service, etc.
Doing a GET request doesn't do that.
What kind of problems that would be?
?? How? And what?
You have to agree that at one point "be used by LLM" would not be different from "be used by a user".
It's self-hosted and free.
How does that prohibit usage and processing of your info? That sounds like "I won't be providing any comments on Lemmy website, if you want my opinion you can mail me at a@b.com"
That will never block all of them. Your info will be used without your consent and you will not feel troubled from it. So you might not feel troubled if more things do the same.
What if I use my local hosted LLM? Anyway, the point is, selling text can't work well, and you're going to spend much more resources on collecting and summarizing data about how your text was used and how others benefited from it, in order to get compensation, than it worths.
Also, it might be the case that some information is actually worthless when compared to a service provided by things like LLM, even though they use that worthless information in the process.
I'm all for killing off LLMs, btw. Concerns of site makers who think they are being damaged by things like Perplexity are nothing compared to what LLMs do to the world. Maybe laws should instead make it illegal to waste energy. Before energy becomes the main currency.
ubergeek
in reply to rdri • • •Then you don't believe content creators should have any control over their own works?
The "proper conditions" are deemed by the content creator, not the consumers.
Not at all. It consumes at most, a watt.
Increasing my hosting bill, to accommodate the senseless traffic being sent my way?
Outages for my site, making my content unavailable for legitimate users?
Not at all. LLMs are not users.
If you want, or they charge for the hosted version. If they want to use a paid for version, then they can divert some of that revenue to me, the creator, because without creators, they would have no product.
That's a apples and oranges comparison, and you know it.
Perplexity seems to be troubled by it.
If selling text can't work well, then why do LLM products insist on using my text, to sell it?
LLMs are a net negative, as far as costs go. They consume far more in resources than they provide in benefit. If my information was worthless without an LLM, it's worthless with an LLM, therefore, LLMs don't need to access it. Periodt.
The bottom line? Content creators get the first say in how their content is used, and consumed. You are not entitled to their labor, for free, and without condition.
rdri
in reply to ubergeek • • •ubergeek
in reply to rdri • • •Its a good thing I don't just block Perplexity, but all of the LLMs.
And I wont comment on the rest of this, but lets consider another form of property: Real estate.
You own a plot of land. Should others be able to use it, however they feel, whenever they feel like? Or should you have a say in how it gets used?
If you feel like you should have exclusive say in how real estate you own is used and when and by whom, why is intellectual property any different? There must be value in using it, so what's wrong with revenues generated by that use being shared (At least) with the creator?
Last I checked, I'm not seeing rev shares from any of these LLMs that have certainly used my code and other content to train?
kreskin
in reply to Davriellelouna • • •Dr. Moose
in reply to kreskin • • •tempest
in reply to Dr. Moose • • •Yeah and the worst part is it doesn't fucking work for the one thing it's supposed to do.
The only thing it does is stop the stupidest low effort scrapers and forces the good ones to use a browser.
5gruel
in reply to kreskin • • •Recaptcha v2 does way more than check if the box was checked.
stackoverflow.com/a/27299487
How does Google reCAPTCHA v2 work behind the scenes?
Stack Overflowkreskin
in reply to 5gruel • • •Glitchvid
in reply to Davriellelouna • • •GamingChairModel
in reply to Glitchvid • • •Encrypt-Keeper
in reply to GamingChairModel • • •Demdaru
in reply to Encrypt-Keeper • • •gian
in reply to Demdaru • • •GamingChairModel
in reply to Encrypt-Keeper • • •And my point is that defining "unauthorized" to include visitors using unauthorized tools/methods to access a publicly visible resource would be a policy disaster.
If I put a banner on my site that says "by visiting my site you agree not to modify the scripts or ads displayed on the site," does that make my visit with an ad blocker "unauthorized" under the CFAA? I think the answer should obviously be "no," and that the way to define "authorization" is whether the website puts up some kind of login/authentication mechanism to block or allow specific users, not to put a simple request to the visiting public to please respect the rules of the site.
To me, a robots.txt is more like a friendly request to unauthenticated visitors than it is a technical implementation of some kind of authentication mechanism.
Scraping isn't hacking. I agree with the Third Circuit and the EFF: If the website owner makes a resource available to visitors without authentication, then accessing those resources isn't a crime, even if the website owner didn't intend for site visitors to use that specific method.
United States v. Andrew Auernheimer
Electronic Frontier FoundationGlitchvid
in reply to GamingChairModel • • •When sites put challenges like Anubis or other measures to authenticate that the viewer isn't a robot, and scrapers then employ measures to thwart that authentication (via spoofing or other means) I think that's a reasonable violation of the CFAA in spirit — especially since these mass scraping activities are getting attention for the damage they are causing to site operators (another factor in the CFAA, and one that would promote this to felony activity.)
The fact is these laws are already on the books, we may as well utilize them to shut down this objectively harmful activity AI scrapers are doing.
tomalley8342
in reply to Glitchvid • • •Glitchvid
in reply to tomalley8342 • • •Do you think DoS/DDoS activities should be criminal?
If you're a site operator and the mass AI scraping is genuinely causing operational problems (not hard to imagine, I've seen what it does to my hosted repositories pages) should there be recourse? Especially if you're actively trying to prevent that activity (revoking consent in cookies, authorization captchas).
In general I think the idea of "your right to swing your fists ends at my face" applies reasonably well here — these AI scraping companies are giving lots of admins bloody noses and need to be held accountable.
I really am amenable to arguments wrt the right to an open web, but look at how many sites are hiding behind CF and other portals, or outright becoming hostile to any scraping at all; we're already seeing the rapid death of the ideal because of these malicious scrapers, and we should be using all available recourse to stop this bleeding.
tomalley8342
in reply to Glitchvid • • •ubergeek
in reply to tomalley8342 • • •tomalley8342
in reply to ubergeek • • •As someone who registered this account on this platform in response to Reddit's API restrictions, it would be hypocritical of me to accept such a belief.
ubergeek
in reply to tomalley8342 • • •tomalley8342
in reply to ubergeek • • •I can see that things are the way things are. Accepting it is a different matter.
To me, the "access" that I am referring to (the interface with which you gain access to a service) and that "access" (your behavior once you have gained access to a service) are different topics. The same distinction can be made with the concern over DoS attacks mentioned earlier in the thread. The user's behavior of overwhelming a site's traffic is the root concern, not the interface that the user is connecting with.
finitebanjo
in reply to GamingChairModel • • •GamingChairModel
in reply to finitebanjo • • •Yeah, fuck everything about that. If I'm a site visitor I should be able to do what I want with the data you send me. If I bypass your ads, or use your words to write a newspaper article that you don't like, tough shit. Publishing information is choosing not to control what happens to the information after it leaves your control.
Don't like it? Make me sign an NDA. And even then, violating an NDA isn't a crime, much less a felony punishable by years of prison time.
Interpreting the CFAA to cover scraping is absurd and draconian.
finitebanjo
in reply to GamingChairModel • • •GamingChairModel
in reply to finitebanjo • • •finitebanjo
in reply to GamingChairModel • • •GamingChairModel
in reply to finitebanjo • • •So yeah, I stand by my statement that anyone thinks this is a crime, or should be a crime, has a poor understanding of either the technology or the law. In this case, even mentioning Alphabet suing for damages means that you don't know the difference between criminal law and civil law.
That's not a crime, and again reveals gaps in your knowledge on this topic.
finitebanjo
in reply to GamingChairModel • • •GamingChairModel
in reply to finitebanjo • • •Who said anything about DDoS? I'm using ad blockers and saving/caching/archiving websites with a single computer, and not causing damage. I'm just using the website in a way the owner doesn't like. That's not a crime, nor should it be.
finitebanjo
in reply to GamingChairModel • • •We did
GamingChairModel
in reply to finitebanjo • • •finitebanjo
in reply to GamingChairModel • • •You appear to have misread
YOU caused google lost ad revenue
GOOGLE's Crawlers have crippled sites
poopkins
in reply to Davriellelouna • • •I've developed my own agent for assisting me with researching a topic I'm passionate about, and I ran into the exact same barrier: Cloudflare intercepts my request and is clearly checking if I'm a human using a web browser. (For my network requests, I've defined my own user agent.)
So I use that as a signal that the website doesn't want automated tools scraping their data. That's fine with me: my agent just tells me that there might be interesting content on the site and gives me a deep link. I can extract the data and carry on my research on my own.
I completely understand where Perplexity is coming from, but at scale, implementations like ~~this~~ Perplexity's are awful for the web.
(Edited for clarity)
IphtashuFitz
in reply to poopkins • • •I hate to break it to you but not only does Cloudflare do this sort of thing, but so does Akamai, AWS, and virtually every other CDN provider out there. And far from being awful, it’s actually protecting the web.
We use Akamai where I work, and they inform us in real time when a request comes from a bot, and they further classify it as one of a dozen or so bots (search engine crawlers, analytics bots, advertising bots, social networks, AI bots, etc). It also informs us if it’s somebody impersonating a well known bot like Google, etc. So we can easily allow search engines to crawl our site while blocking AI bots, bots impersonating Google, and so on.
poopkins
in reply to IphtashuFitz • • •What I meant with "things like this are awful for the web," I meant that automation through AI is awful for the web. It takes away from the original content creators without any attribution and hits their bottom line.
My story was supposed to be one about responsible AI, but somehow I screwed that up in my summary.
sylver_dragon
in reply to Davriellelouna • • •snooggums
in reply to sylver_dragon • • •flux
in reply to snooggums • • •This is not about training data, though.
Personally I think that claim is a decent one: user-initiated request should not be subject to robot limitations, and are not the source of DDOS attack to web sites.
I think the solution is quite clear, though: either make use of the user identity to walz through the blocks, or even make use of the user browser to do it. Once a captcha appears, let the user solve it.
Though technically making all this happen flawlessly is quite a big task.
snooggums
in reply to flux • • •They are one of the sources!
The AI scraping when a user enters a prompt is DDOSing sites in addition to the scraping for training data that is DDOSing sites. These shitty companies are repeatedly slamming the same sites over and over again in the least efficient way because they are not using the scraped data from training when they process a user prompt that does a web search.
Scraping once extensively and scraping a bit less but far more frequently have similar impacts.
flux
in reply to snooggums • • •When user enters a prompt, the backend may retrieve a handful a pages to serve that prompt. It won't retrieve all the pages of a site. Hardly different from a user using a search engine and clicking 5 topmost links into tabs. If that is not a DoS attack, then an agent doing the same isn't a DDoS attack.
Constructing the training material in the first place is a different matter, but if you're asking about fresh events or new APIs, the training data just doesn't cut it. The training, and subsequenctly the material retrieval, has been done a long time ago.
Amberskin
in reply to Davriellelouna • • •Uh, are they admitting they are trying to circumvent technological protections setup to restrict access to a system?
Isn’t that a literal computer crime?
utopiah
in reply to Amberskin • • •Deflated0ne
in reply to utopiah • • •iamdefinitelyoverthirteen
in reply to utopiah • • •dinckel
in reply to Amberskin • • •Silicon
in reply to dinckel • • •Dr. Moose
in reply to Davriellelouna • • •It's insane that anyone would side with Cloudflare here. To this day I cant visit many websites like nexusmods just because I run Firefox on Linux. The Cloudflare turnstile just refreshes infinitely and has been for months now.
Cloudflare is the biggest cancer on the web, fucking burn it.
baronofclubs
in reply to Dr. Moose • • •omg ur a hacker
Did you mean Edge on Windows? 'Cause if so, welcome in!
dodos
in reply to Dr. Moose • • •Yeller_king
in reply to dodos • • •Dr. Moose
in reply to dodos • • •"Wrong with my setup" - thats not how internet works.
I'm based in south east asia and often work on the road so IP rating probably is the final crutch in my fingerprint score.
Either way this should be no way acceptible.
JcbAzPx
in reply to Dr. Moose • • •jaemo
in reply to dodos • • •Thirded. All three (Linux, FF, nexus)
ZERO ISSUES.
Dremor
in reply to Dr. Moose • • •Linux and Firefox here. No problem at all with Cloudflare, despite having more or less as much privacy preserving add-on as possible. I even spoof my user agent to the latest Firefox ESR on Linux.
Something's may be wrong with your setup.
Dr. Moose
in reply to Dremor • • •Dremor
in reply to Dr. Moose • • •Same goes the other way. It's not because it doesn't work for you that it should go away.
That technology has its uses, and Cloudflare is probably aware that there are still some false positive, and probably is working on it as we write.
The decision is for the website owner to take, taking into consideration the advantages of filtering out a majority of bots and the disadvantages of loosing some legitimate traffic because of false positives. If you get Cloudflare challenge, chances are that he chosed that the former vastly outclass the later.
Now there are some self-hosted alternatives, like Anubis, but business clients prefer SaaS like Cloudflare to having to maintain their own software. Once again it is their choices and liberty to do so.
Dr. Moose
in reply to Dremor • • •lmao imagine shilling for corporate Cloudflare like this. Also false positive vs false negative are fundamentally not equal.
The main issue with Cloudflare is that it's mostly bullshit. It does not report any stats to the admins on how many users were rejected or any false positive rates and happily put's everyone under "evil bot" umbrella. So people from low trust score environments like Linux or IPs from poorer countries are under significant disadvantage and left without a voice.
I'm literally a security dev working with Cloudflare anti-bot myself (not by choice). It's a useful tool for corporate but a really fucking bad one for the health of the web, much worse than any LLM agent or crawler, period.
Laser
in reply to Dr. Moose • • •Linux user here, Cloudflare hasn't blocked access to a single page for me unless I use a VPN, which then can trigger it.
Dremor
in reply to Dr. Moose • • •COASTER1921
in reply to Dremor • • •I suspect a lot of it comes down to your ISP. Like the original commentor I also frequently can't pass CloudFlare turnstile when on Wifi, although refreshing the page a few times usually gets me through. Worst case on my phone's hotspot I can much more consistently pass. It's super annoying and combined with their recent DNS outage has totally ruined any respect I had for CloudFlare.
Interesting video on the subject: youtu.be/SasXJwyKkMI
- YouTube
youtu.beCatDogL0ver
in reply to Dr. Moose • • •It happened to me before until I did a Google search. It was my VPN web protection. It was too " over protective".
Check your security settings, antivirus and VPN
Kissaki
in reply to Davriellelouna • • •So, I assume Perplexity uses appropriate identifiable user-agent headers, to allow hosters to decide whether to serve them one way or another?
Dr. Moose
in reply to Kissaki • • •ubergeek
in reply to Kissaki • • •Kissaki
in reply to ubergeek • • •ubergeek
in reply to Kissaki • • •Except, it's not a live user hitting 10 sights all the same time, trying to crawl the entire site... Live users cannot do that.
That said, if my robots.txt forbids them from hitting my site, as a proxy, they obey that, right?
lime!
in reply to Kissaki • • •seraphine
in reply to lime! • • •lime!
in reply to seraphine • • •i really wish we wouldn't do those. feels too reddity.
but thanks.
seraphine
in reply to lime! • • •lime!
in reply to seraphine • • •Kokesh
in reply to Davriellelouna • • •ubergeek
in reply to Kokesh • • •FauxLiving
in reply to Davriellelouna • • •The amount of people just reacting to the headline in the comments on these kinds of articles is always surprising.
Your browser acts as an agent too, you don’t manually visit every script link, image source and CSS file. Everyone has experienced how annoying it is to have your browser be targeted by Cloudflare.
There’s a pretty major difference between a human user loading a page and having it summarized and a bot that is scraping 1500 pages/second.
Cheering for Cloudflare to be the arbiter of what technologies are allowed is incredibly short sighted. They exist to provide their clients with services, including bot mitigation. But a user initiated operation isn’t the same as a bot.
Which is the point of the article and the article’s title.
It isn’t clear why OP had to alter the headline to bait the anti-ai crowd.
ubergeek
in reply to FauxLiving • • •Except, they don't. It's a toggle, available to users, and by default, allows Perplexity's scraping.
snooggums
in reply to FauxLiving • • •Oh fuck off with that AI company propaganda.
The AI companies already overwhelmed sites to get training data and are repeating their shitty scraping practices when users interact with their AI. It's the same fucking thing.
Web crawlers for search engines don't scrape pages every time a user searches like AI does. Both web crawlers and scrapers are bots, and how a human initiates their operation, scheduled or not, doesn't matter as much as the fact that they do things very differently and only one of the two respects robots.txt.
FauxLiving
in reply to snooggums • • •There’s no difference in server load between a user looking at a page and a user using an AI tool to summarize the page.
You either didn’t read the article or are deliberately making bad faith arguments. The entire point of the article is that the traffic that they’re referring to is initiated by a user, just like when you type an address into your browser’s address bar.
This traffic, initiated by a user, creates the same server load as that same user loading the page in a browser.
Yes, mass scraping of web pages creates a bunch of server load. This was the case before AI was even a thing.
This situation is like Cloudflare presenting was a captcha in order to load each individual image, css or JavaScript asset into a web browser because bot traffic pretends to be a browser.
I don’t think it’s too hard to understand that a bot pretending to be a browser and a human operated browser are two completely different things and classifying them as the same (and captchaing them) would be a classification error.
This is exactly the same kind of error. Even if you personally believe that users using AI tools should be blocked, not everyone has the same opinion. If Cloudflare can’t distinguish between bot requests and human requests then their customers can’t opt out and allow their users to use AI tools even if they want to.
ubergeek
in reply to FauxLiving • • •There is, in scale.
Electricd
in reply to Davriellelouna • • •tempest
in reply to Electricd • • •Laser
in reply to tempest • • •Electricd
in reply to tempest • • •sandwich.make(bathing_in_bismuth)
in reply to Electricd • • •iamdefinitelyoverthirteen
in reply to sandwich.make(bathing_in_bismuth) • • •EDIT: It was supposed to say "loops", but I'm keeping it.
Electricd
in reply to Davriellelouna • • •They do have a point though. It would be great to let per-prompt searches go through, but not mass scrapping
I believe a lot of websites don't want both though
threeganzi
in reply to Electricd • • •Electricd
in reply to threeganzi • • •I assume their script does some search engine stuff like query google or bing and then "scrap" the links they go on
Some selenium stuff
Jimmycrackcrack
in reply to Davriellelouna • • •tarknassus
in reply to Davriellelouna • • •