Skip to main content
I’ve thought about this when @ar.al🌻 shared their massive #Mastodon worker activity increase related to the #TwitterMigration. This migration barely started, relative to the total number of Twitter accounts, and the #Fediverse wasn’t particularly under-provisioned.

It’s just that activity increase isn’t linear with the total number of users/instances. And as long as we were a marginal item, existing hosting solutions could stay ahead of the curve.

The Twitter migration isn’t the real deal, it’s a relatively small stress test, but its effects already are hard to handle for most existing servers.

I believe we may be at a turning point where no #Fediverse project will be able to claim to be “light” or “fast” anymore, just by virtue of the compounded network load.

Yellow Flag

2022-11-09 09:57:34


It will be interesting to watch live how Fediverse scales up in reality.

Thing is: Fediverse’s communication between instances is roughly quadratic in the number of instances, and it’s getting closer to quadratic as the instances are becoming more interconnected due to increased user numbers.

An additional issue: the effort of running an instance (e.g. storage space) isn’t linear in the number of instance users but rather roughly linear in the number of total Fediverse users across all instances.

In other words: if at some point the majority of Twitter users migrates to Mastodon or other Fediverse software, spreading the load across a huge number of instances won’t make it all too much easier to handle. On the other hand, the traffic required to keep all these instances in sync might become completely unrealistic.

Interesting times…
#fediverse #Mastodon #twittermigration @Aral Balkan
Ed Chivers
Ed Chivers  
•
I was wondering about this yesterday - I am a newbie here but I've noticed that the instance I'm on gets pretty slow during the North American day. With all these instances talking to each other, it feels like linear user growth is going to ramp up traffic by some kind of square (or exponential?) factor. I don't know how the system is designed, whether it's many-to-many or whether some sort of hub-and-spoke design might help?

But, as I said, total newbie, just thinking out loud.
Hypolite Petovan
Hypolite Petovan  
•
@Ed Chivers Like @https://infosec.exchange/users/WPalant mentioned in their posts I linked, it seems to be quadratic to the number of instances and the efforts to run a single server is linear with the total number of Fediverse users which sounds reasonably accurate to me.

It is many to many, even accounting for the server blocks, and I'm not sure how a hub-and-spoke model would work given the underlying protocol design decisions.
@Ed Chivers
Ed Chivers
Ed Chivers  
•
I read the post (ok, skimmed it) and you're right, hub and spoke would probably fall foul of some of the design philosophy on here. I was thinking in terms of how content distribution networks work, but of course there's no "central" place we're all pulling content from. It's definitely beyond my knowledge/expertise but it's an interesting problem.
Hypolite Petovan
Hypolite Petovan  
•
@Ed Chivers What most interests me is how the technical decisions enabled a completely different social media culture, and how much changing the technical details would affect said culture.
@Ed Chivers
Darnell Clayton :verified:
Darnell Clayton :verified:  
•
One possible solution is to aggressively advocate for most hosting companies to take Federated software more seriously.

Many are nervous about hosting social networks due to copyright issues, but the appeal of new customers might make it reasonable for them to consider.
Becky
Becky  
Would simply spinning up new instances solve this problem? I understand, however, that a lot of people would be stressed to start their own instance...
Hypolite Petovan
Hypolite Petovan  
•
@Becky Unfortunately not, like explained in the shared post, activity increases non-linearly with the number of individual instances, so spinning up more instances actually increases the potential load for everybody!
@Becky
Becky
Becky  
Ah, I've actually read it now 😓

I hope we can figure it out
Hypolite Petovan
Hypolite Petovan  
•
@Becky After reading this: https://anarchism.space/@iarna/109310657675344110, I believe the "figuring out" will likely not be needed for long after Google or Facebook embrace, extend and finally extinguish #ActivityPub as they've done it before for email, XMPP, public APIs, etc...
#ActivityPub @Becky
Yellow Flag
Yellow Flag  
•
@firefly_lightning No, that’s exactly what is going to make things worse as mentioned in my post. More instances means more communication between instances. And this grows quadratically. It will reach the point where no connection will be wide enough to handle the traffic way too soon.

@hypolite
@Becky @Hypolite Petovan
clacke: exhausted pixie dream boy 🇸🇪🇭🇰💙💛
clacke: exhausted pixie dream boy 🇸🇪🇭🇰💙💛  
•
> the effort of running an instance (e.g. storage space) isn’t linear in the number of instance users but rather roughly linear in the number of total Fediverse users across all instances.

If I'm a single-user instance and follow 1000 people, it doesn't matter to me if the rest of fedi is 100 k people or 1 M people, I'll have the same traffic ... unless they all start following me.

I guess a bigger network means followers, but I don't see how it's superlinear. Especially my load doesn't change if 1 person or 1000 people follow me from mastodon.online. And certainly not my storage space.

@Hypolite Petovan @ar.al🌻 @Yellow Flag
@Yellow Flag @Aral Balkan @Hypolite Petovan
Hypolite Petovan
Hypolite Petovan  
•
@a Claes unto himself 🇸🇪🇭🇰💙💛 Your node interacts with any remote node from which people follow you. Refreshing contact information is among the more frequent request we perform, and this happens whether people follow you or you follow them. We also store the profile picture and the banner image for each contact we know about, no matter how, so it will add up.
@clacke: exhausted pixie dream boy 🇸🇪🇭🇰💙💛
Hypolite Petovan
Hypolite Petovan  
•
@a Claes unto himself 🇸🇪🇭🇰💙💛 Case in point, my own single-user node is aware of 12,406 remote instances, and I have only 1,000 contacts myself. But other people like and reply to my posts, so they are probed as well. So my node maintains records for 216,392 contacts.

Most of these contacts would be shared in larger instances, but single-user instances need to keep tabs of remote contacts independently, increasing the network traffic.
@clacke: exhausted pixie dream boy 🇸🇪🇭🇰💙💛
Hypolite Petovan
Hypolite Petovan  
•
@Andy H3 @ar.al🌻 @Yellow Flag It entirely depends on the density of the social graph, but the maximum values are scary.
@Yellow Flag @Aral Balkan @Andy H3
⇧