Salta al contenuto principale


The Italian instance Friendica "Poliverso" apologizes to the whole fediverse! (long post)


Last night, between 23.00 on 23 June and 1.00 AM on 24, the Poliverse instance "exploded" a thousand messages around the fediverse!

** We have to apologize to a lot of people: **
• we apologize to anyone who was ** woken up by our notifications **;
• we apologize to those who have cisted their ** federated timeline invaded by messages ** coming from Poliverso;
• We apologize to the ** administrators and instance moderators who know us ** who probably wondered who hacked our server;
• We apologize to the ** administrators who didn't know us ** and who noticed us in the worst way
• We apologize most of all ** to some of our users ** who, through our fault, have ** UNKnowingly ** behaved as a spammer ...

Unfortunately, we could not foresee this collapse and we did everything possible to limit the damage: unfortunately we are sure that our instance or some of our users have ended up in some blacklist.

** If so, we ask you to read this message, in which we explained what happened. **

PS: if any Friendica expert should realize that some considerations are wrong or should be better defined, please let us have your own considerations in the comments!

What happened to the Poliverso server?

The update


The new Friendica release, the 2022.06 called “Giant Rhubarb” is a seemingly useless release, because nothing changes for users! Yet it has greatly improved compatibility with other software in the fediverse: there has in fact been immense work by the developers and there have been many changes; in such cases it is possible that disservices arise and, although some instances have already installed an exploratory version, it is clear that some problems can only emerge during the production phase.

Precisely for this reason we decided to proceed with the installation only several days after the release, to understand what the problems would be for users and notify them in time; and yesterday 23 June at 22.00, the Poliverso server was updated.

The problem


At first it didn't seem like there were any problems and, in fact, after half an hour we stopped monitoring. But at 11.00 something happened ...

1467567


Admin 1
22.56 (the developers of friendica) have changed something: some users get a lot of notifications about old things .. then everything is fixed
22.56 (I'm seeing a lot of queue in processes)

Admin 2
23.00 So far I am not having this type of problem. Actually, I am not having any problems
23.01 To be more precise, I still can't see anything different than before ... 😅
23.04 Right now, after a few tens of minutes of pause, he's posting stuff on Twitter
23.07 Holy shit! my accounts are flooding the universe with messages

Admin 1
23.14 in what sense? : D

Admin 2
23.15 in what sense? The system is shitting out all the messages posted by the RSS Feed. It is a disaster. I'm emptying the ocean with a spoon 🤣
23.15 This time we are banned from all the galaxies of the fediverse

Admin 1
23.16 I noticed a lot of RSS stuff, but I said to myself "boh"

Admin 2
23.16 They are the queues of those messages

In short, yesterday there was a real disaster for which the Poliverse instance will pay the consequences for a long time as in reputational terms. However, the management of the problem was absolutely timely both from the point of view of resolution (at 1:00 AM almost all the messages had been deleted) and from the point of view of communication.

https://poliverso.org/display/0477a01e-4062-b4e8-2a73-375308279474

1467569

The reasons for the problem


However, the problem was not systemic or due to problems of administration of the instance; the problem was due to a series of factors that multiplied its impact:

  • Friendica upgrade bugs exist: the "system" upgrade all together, some integration problems with other activitypub instances; but all this causes a "re-reading" of the notifications from the RSS FEED; in this way, users who also repost some posts from the feed get** a new repost (😱!)** of the feed.
  • this system (republishing from feed) is not very common among Friendica users; so the problem was not noticed too much on the forums
  • inside Friendica there are no valid administration or moderation tools (an ancestral and well-known Friendica problem: https://github.com/friendica/friendica/issues/8724), which would allow us to delete messages in bulk or to immediately stop the republishing service from feed or to temporarily suspend the "unwittingly spammy" users. So we either manually deleted and deactivated everything or had users manually delete their messages - this was an incredible waste of time!

The reputational damage


All of this prevented us from predicting the problem and then amplified it. However, we can assure you that the management of the application was professional and within a couple of hours from the alert, we have erased all traces of the problem!

The fact remains that some users who followed the accounts in question probably wondered "what's going on?"; others have probably defollowed, silenced or banned the accounts of innocent and unaware users ...

The same thing will have happened for administrators: some of them, to protect their users from spam, will have silenced or banned users; or they will have silenced or banned the instance!

Fortunately, those in the Central European Time Zone were probably asleep at the time and were not online on Friendica, Mastodon, Misskey, Pleroma or Hubzilla

Final thoughts


Of course during those hours we felt [a furious rage] (https://poliverso.org/display/0477a01e-1062-b4db-9d28-f53436523782) towards the Friendica developers who caused us such a problem...

https://poliverso.org/display/0477a01e-1062-b4db-9d28-f53436523782

...but we clearly don't get mad at them. Moreover, it is thanks to them that Friendica exists; it is thanks to them that there is what we believe is the software of the fediverse that allows the most complete social experience today.

Next time, however, we recommend that you report the problem well and above all, express at least a little regret to the Friendica instance administrators and other administrators who had to manage what seemed to all intents and purposes a massive spam activity!

But above all, we send a message to all instance administrators and users who are reading us: if you have defollowed, silenced or banned some Poliverse users, think again! They had no fault, other than to use the RSS feed reposting system: give them a second chance!

in reply to Poliverso

This is why the admins should stop banning left and right. Let the users decide. So what if I see lots of posts form you instance!? I, personally can block or ignore, but not as an instance admin to do that for so many users. This is why I keep on insisting that admins should block users/instances for all of their users. It is a very bad practice. In any case, I hope your instance didn't get to be very damaged.

don't like this

in reply to Tio

You're right: blocking an entire room is something that needs to be done only in very special cases, but unfortunately yesterday and the day before yesterday there were some episodes of real spam from some Pleroma servers. these servers have begun to annoy a lot of random users and many administrators have decided to practice blocking the entire instance. I hope that what happened yesterday to the Poliverse server was not considered a malicious attack, by all those who had been negatively affected by that spam attack that took place yesterday.
in reply to Poliverso

Yeah but I think admins should chill a bit. Else they will ruin the fediverse. How many users you have?

don't like this

in reply to Tio

We have 250 users, although only about a hundred are active. And among them are 12 service accounts, which are the ones I manage
in reply to Poliverso

I see....we have over 600 and over 50 accounts that auto-post from RSS sources...I didn't seem to get any heads-up as if we were spamming...but well that's gone now anyway....

Democracy doesn't like this.

in reply to Tio

I understand, but you don't necessarily get a warning ...

In fact, the additional problem is that if a mastodon, or pleroma or misskey user reports a Friendica content as if it were spam, the administrator would not even notice it, because there is no moderation interface in Friendica ...

And this can expose Friendica instances to ban.

But that's another problem 😅

PS: However, I take this opportunity to tell you that I really like your instance and I really like how you manage it but also all the fantastic posts you wrote on friendica
in reply to Poliverso

Yeah I wish there were better tools for the admin...I get you. And thank you for the nice words!
in reply to Tio

I mean, users can decide to move to an instance with less moderation.
in reply to Ephera

People keep on saying that.....same with those who keep on saying: if you do not like google, use bing. Don't like Facebook? Use Twitter. And they are blind to the point that's being made.

don't like this

in reply to Tio

I really don't see those as the same thing. You can't talk to the same people on Twitter that you know from Facebook. With the Fediverse, you generally can.

A user on https://mastodon.technology can migrate to https://fosstodon.org and will generally find the same people + whoever may have been silenced by more aggressive moderation.
Same thing the other way around. Most users don't care about seeing everything, and may just find it more convenient when trolls and assholes are already filtered out for them, so may migrate from Fosstodon to mastodon.technology.

No, this is not a perfect system on an individual user basis, since most users won't really know what they're not seeing.
But I do think, in the grand scheme of things, people will bundle up around instances that offer a good balance between interesting content and trolls/assholes.
in reply to Ephera

But my point is that this practice is harmful for the fediverse as a whole. Non transparent, decided for thousands of users, shallow decisions in many instances. To move from one federated platform to another is not as easy as you make it sound. I know, I have done that and I also manage a few instances. So this is a non-solution.

The solution is to, for one, make all instance-level blocking of users or other instances transparent and notify all users when any future such decision is taken. And second, try not to panic. It is fine. Words, no swords, we are dealing with. Users have a lot of control over that they see. Teach them to exercise that. This is a far better approach.

Democracy doesn't like this.

Signor Amministratore reshared this.

in reply to Tio

@Tio@Ephera A very nice example of how one can be transparent is the public moderation register in Lemmy.

The measures of the administrators of the instance and of the moderators of each community are made evident. This is a very useful thing to understand how an instance is administered and understand if the moderators practice abuse or support a balanced moderation aimed at maintaining a plural and peaceful environment.

https://lemmy.ml/modlog
Unknown parent

Tio
Yes this is a decent start and I wonder too if it is automated. It should. Plus, if there is a new block, should create a notification for all users by default I would say.

Democracy doesn't like this.

Unknown parent

Tio
Probably should be better than this, but for sure an instance needs to be very open about this practice.
Unknown parent

Liwott
@BobIsMyManager@Tio@Ephera
Alternatively, there could be a public modlog with a RSS feed (granted, not all fedi software allow to follow RSS feed :) )
Unknown parent

Informapirata
We must make a premise: there are different types of instances in the fediverse and some of these, especially mastodon, are of a strongly generalist nature, apolitical and dedicated to an audience that does not want to see disturbing content (Not Safe for Work NSFW).

These instances are forced to silence and sometimes block users who post content of this type without the content warning tag.

This can be considered excessive, but it is the only way to keep an environment free of disturbing elements such as overly fiery political speech, explicit content, foul language.

All this concerns above all the generalist instances.

As for the others, it becomes more rare to practice BAN and silencing.

Unfortunately, however, there are two types of drastic measures: those towards users of your own instance and those towards users of other instances.

In my opinion, the latter should undergo a periodic review, a sort of amnesty that occasionally cleans up the sentences imposed or at least reconsiders some users who at the beginning of their life in the fediverse behaved badly because they were used to the toxic environment of Facebook or of Twitter.

In fact, the problem with these users is that they risk being invisible for life from some instances.
Questa voce è stata modificata (1 anno fa)
in reply to Liwott

Lemmy uber alles... 😁
https://lemmy.ml/modlog
Questa voce è stata modificata (1 anno fa)
in reply to Informapirata

@Informapirata
Yes I probably had it in mind when saying that (I mean I did not use the word "modlog" for free :) ). All it misses to fit @Tio 's ideal is a RSS-like feature (that could be an actual RSS feed)