Why are these Twitter clones always so resource intensive and finicky? Twitter is just "IRC in reverse", if you take literal descriptions transitive relations in IRC(v2) and moved around nouns, lots of it should apply to Twitter/Bluesky/Mastodon well. RDB gurus can probably recreate lots of APIs as tables on bare MySQL too.
It doesn't make sense to me that such a thing take so much dev and ops cost compared to IRCv2 servers, other than for the fact that modern webdev just so happens to be extremely bloated, especially when extremely competent and high spirited developers are giving up like this.
Are we doomed to keep adding more RAM and more disk and more bandwidth to catch up with ever-growing bloat?
Not related to Mastodon, but in the case of Matrix, the server software ranges from "runs on a raspberry pi with zero issues" (Conduit) to "even with 16 GB of RAM, federating with a large enough room will exhaust Python's heap" (Synapse).
In the case of Conduit, a Matrix server with a few private rooms and users consumed only 32 MB of RAM, using RocksDB for storage. The equivalent on Synapse required about 5x as much memory, despite using SQLite. In practice, Synapse instances will use Postgres since many appservice plugins specifically require Postgres and don't support SQLite. Not to mention, SQLite isn't optimized for frequent, concurrent writes.
I do sincerely think the choice of Rails, and the fact Ruby only got a compiler people use recently, means that most Ruby programs require fairly beefy processors and plenty of memory in order to keep up with a few hundred clients.
Of course, I am extrapolating based off my experience running Synapse (a Matrix server) with Postgres. There is a chance Mastodon scales much better.
>extremely competent and high spirited developers are giving up like this.
I'm pretty sure the median IRC server runs for much less than 7.5 years. I don't think anyone expects volunteers to dedicate decades of their life to admin duty. and it seems fine and healthy for the ecosystem that he is telling people they have a few months to move their bots to a different server.
From what I understand, IRC doesn't hold messages for extended periods of time or allow media uploads, while fediverse does, so that's one big difference.
I think there are issues with hotspots. The most popular tweets are seen by a big chunk of the userbase, which means they have to operate on a fanout model where each tweet is pushed to individual followers.
I believe IRC doesn't operate like that. The messages delivered to each user don't need to be retained, and I assume the size of the largest channels is in the tens or hundreds of thousands.
Aw too bad, this has been a really useful service. I wonder if anyone wants to pick it up? The post mentions part of the problem is Mastodon's implementation being a poor match to high volume bots. You could imagine other architectures that were more efficient for this use case, it'd be a fun yak shaving exercise.
If anyone needs to migrate their own projects I've had good luck with feed2toot, to post RSS to a Mastodon account on a ordinary server. It's been around a long time now and seems reliable.
I wholly appreciated the openness to accepting bot accounts, migrated some projects from twitter there during the big exoduses in the last couple years. And while it worked for bot purposes, fun to tinker with etc...(not unlike twitter tbf) it was just some server in space a blip in the fediverse and traction and lack of proper network effects for accounts meant it wasn't much use.
I'm not hot on the fediverse in general, and this just sours me on it a bit more. A bunch of dedicated admins keeping instances going, basically running hobby servers/websites like it was the 90s/early 00s, is never gonna work for the kind fo scale services grow to these days. I know not everything requires scale and lots of ppl are happy existing in their little silos, but that's just it, they're silos. Might as well be back on seperate forums for our seperate interests again. When you want the power of a mix of accounts/networks/interests everything balloons and can't be run with funds and larger centralization. Sigh. It's a tough one and has yet to be solved in full, with any existing approaches all sort of half-solutions. Maybe that's the way forwards in general (an internet of islands) but it sucks to have things going up and down and having to migrate around the net (with or without our own data) like nomads.
> When you want the power of a mix of accounts/networks/interests everything balloons and can't be run with funds and larger centralization.
I don't really get the appeal of this. Different topics/interests often demand different moderation and forum features. Trying to shove everything into the lowest common denominator of social media results in things like people posting essays as screenshots of Apple Notes.
The only obvious benefit I see of this kind of large scale centralization is for marketing. And that's not a benefit to me as a user.
The benefit of a centralized forum like Reddit, for a user like me, is the ability to use one account for everything (i.e. easy to join new forums) and the ability to have everything on one algorithmic or chronological feed.
Traditional forums can solve the former by using social sign-ins, and the latter by having RSS. However, support for either of this is inconsistent, so it's usually easier to just use subreddits.
Finally, money. Forums cost money to host, while subreddits don't.
Damn, I asked the other day if it was down after noticing a ton of timeouts in sidekiq. Reached out and Colin said they were looking into it. Guess fixing it up just proved to be too much.
>Over the years, the server has grown to have around a few thousand active accounts, which isn't all that many. However, they've generated something like 32 million statuses. Just to put that in perspective, mastodon.social has over 2 million users, who have generated around 110 million statuses.
unsurprising that the bots would outpace organic users, but wow, what a ratio. i'd be curious to see this data charted over time
It makes sense. Assuming the average bot toots ~once an hour (24 times a day) and has been tooting for two years, you get on the order (nearest multiple of 5000) of 20,000 toots per user, which works out to 1600 users.
Also, there's a measuring change from active accounts to plain old users. I don't know the proportion that are active, but if I recall right, the fediverse as a whole had under 1 million active users. Assuming 500,000 active accounts that pull all the weight, it's 220 toots per user on average.
One of the reasons I maintain a node with only one user is I fear the day I'll be responsible for other people's social media presence; I could easily see myself going "It's just a few thousand users" and the next thing I know I'm asking whether I can keep this thing going (and agonizing over what it'll do to my users to cut the service). And unlike Colin, I despise Rails and wouldn't have the patience to hammer on it when it starts to misbehave.
Federated networks like Mastodon and Lemmy are going to get people well-acquainted with websites shutting down. It's hard work (time, money, etc.) to run these things for people, and people start to really lean on them.
It's almost novel now days getting sucked into something that shuts down. killedbygoogle.com is a meme partly I think because websites shutting down is just so uncommon in areas that we get personally invested in.
I run my own Lemmy instance just for my self and even that can be trying sometimes. I enjoy using it instead of reddit, but one day I will probably shut it down and be sad.
I'm starting to think nostr was barking up the right tree after all. Put as much complexity into the client as possible and make the servers dumb and completely uncoordinated, utterly interchangeable. Spam your broadcasts to any relay that will listen. No idea if it actually works (I've read a lot about nostr and AT proto but never used either of them) but I think it's very obvious that any system that deeply relies on some company that everyone becomes extremely reliant on (including AT proto/Bluesky) is only a couple steps away from the same sort of problems as centralization.
Of course, the real gold standard would be P2P, if only it could work. But... mobile phones can't burn battery running P2P clients in the background, everyone's under a NAT these days, and some types of software (like microblogging networks) would be horrifically intractable as a P2P system.
Oh well. At the very least, I really love the concept of Decentralized Identifiers (DIDs). I'd like to see more stuff like that.
The atproto team came from the p2p space. We had a lot of experience running client-side computation. There are challenges you can try to solve in that design -- key sync, key recovery, reliable hosting, reliable discovery, etc -- but what you can't get around is the need to run queries against aggregations of data. Even scales that we consider "mid" start to strain on the client-driven model. Federated queries might be able to solve it, but it's extremely challenging to get reliable performance out of that. The design we landed on was replaceable big nodes.
The common comparable people raise is email/gmail, but we used the DID system and account portability to try to get a better outcome on provider migration. It's hopefully more like web/google -- which still has the centralizing pressures of scale benefits, but hopefully injects enough fluidity in the system to move the situation forward. Sometimes, you pick the design that moves the ball down the field, not the one that guarantees a touchdown. If somebody wanted to improve on what we've done, I'd tell them to focus on the federated queries problem.
In theory AT proto doesn't seem like a bad design. I've read a fair bit of the docs, although mostly skimming. (I've been meaning to read the paper on Merkle Search Trees so I can figure out what exactly is going on with PDSes.)
On the other hand, in practice it seems like the AT proto infrastructure is still very centralized for now. DIDs are excellent in theory, but everyone is using PLC DIDs, which depends on the centralized plc.directory. You can run your own PDSes, but there's only one relay for now (that I am aware of.) I also don't think there is more than one instance of the Bluesky AppView, and the official instance is locked into the Bluesky Moderation Service, which seems to limit the usefulness of some of the censorship resistance of the protocol.
I'm not sure how much of that is social problems rather than technical, but I worry that since Bluesky and AT proto are gaining massive popularity with this status quo (millions of users!) it'll have a problem akin to Matrix.org, where in practice almost everyone is using the same infrastructure anyways.
It's still relatively early days, but millions of people is definitely enough to where you start to hit problems with having everyone under one roof. I really hope we get to see how things play out when more of the network is operated independently.
The necessary future is more providers, more relays, a move of PLC to an independent body, and more DID methods.
I will also say - there are ~100 self-hosting PDSes in the network, about 25 relay consumers, 3 alternative appviews that I know of (smoke signals, frontpage.fyi, and whitewind), the firehose & backfill are fully available, the specs are getting fairly complete, and the software is open source. This is a priority for us.
fwiw roughly 50% of Matrix is on the matrix.org instance currently. we consider this a bug, but also prioritise ease of onboarding over decentralisation purity ideology.
I hate to be a downer but there's a lot of things Matrix could prioritize over decentralization. That said, the decentralization works pretty badly. Large federated joins are somewhere between comically slow and horrifically slow. Status does not seem to work well across federation either.
I'm also a bit miffed that Dendrite was positioned as a "next generation" Matrix server but now it feels nearly orphaned with missing support for newer features, issues with various appservice bridges, few updates at a very slow pace, and no migration path out in sight. I know it came with a fair number of disclaimers, but that still bums me out as it seemed like it would be okay for a small non-critical homeserver, and now it seems likely I'll have to engineer my own path out when clients finally stop working with Dendrite. (It already happened once...)
You have no idea how bad I want to love Matrix, but frankly if it was due to a focus on usability that decentralization "purity" has suffered, it simply does not show in the resulting usability improvements over years of time. Sorry to be harsh.
I did see the Matrix 2.0 announcement, though for obvious reasons I can't actually use any of the features listed in it. Obviously, improvements to the basic chat functions of Matrix would be great. For now though, I am stuck with the reality that joining a large federated channel sometimes takes more than 6 hours. I wish I were exaggerating.
edit: I guess though that faster room joins weren't a part of Matrix 2.0. Actually, I don't really have a huge problem with the sync taking too long personally. So maybe Matrix 2.0 wouldn't bring that big of an improvement for me anyway.
ActivityPub seems to require a lot of hardware resources in order to run properly, which is unfortunate. It’s not something I would ever want to run myself, especially to the public.
Please be wary to conflate ActivityPub with the code on top of it, like Mastodon for instance; which is, on both front and back-end, proven to be resource intensive and difficult/costly to scale especially over time. (the older the dbs get, the more inactive users pile up, etc.)
Versus something like Pleroma; which I've used since it's inception, being incredibly janky and lightweight, prone to breaking, but later versions have mostly ironed out most of those catastrophic bugs. It has it's own challenges as well, but it does historically scale better, is more flexible, and less intensive per instance
One demands a lot of money and time, where the other demands a lot of time and not so much on the money side.
I'm not going to spend the time to give you a history of pleroma/mastodon instances, as it's a controversial history at best and there's a lot of people who know little, yet who will believe themselves an oracle. (ofc that could also be me so take it with a grain of salt and all)
Though if you are willing to read through a bunch of highly opinionated accounts, and you pay close attention to what actually happened, the answer is pretty clear.
ActivityPub is intensive, but not the main culprit.
I'm guessing the syncing process between instances is really brutal on resources. Imagine constantly syncing external databases for your service to function properly.
Eh, it's not new to federated sites. Many of the web sites I frequented a decade a go are dead. Or enshittified useless so I visit them rarely anymore.
It was sad to see say, the "user friendly" webcomic go away. But I enjoyd it in the time. Just live the moment. Don't expect even the big websites and apps of today to last - not at least in the form you enjoy.
I, too, ran my own instance. I enjoyed it for some time but I've now moved to the omg.lol ecosystem. I feel that by paying a little money for it that I have a higher chance of the server not shutting down.
Besides the monetary costs of operating small fora there are also significant competence hurdles. Site owners who manage to hit a couple thousand users have to figure out spam handling, automated content moderation (including photoDNA and the required reporting if they host images), registering a DMCA agent with the copyright office, setting up an LLC, assessing their needs for COPPA, GDPR and CCPA, their site's tax situation and anything they may want to do involving employing others (such as T&S) without getting burnt out. The median size for a forum getting its first subpoena is 4,300 users.
Managing all of that is easily learnable in a couple months if they have time, disposable income and few distractions but surprisingly few people who have site management thrust upon them know about these things in advance. To most people who think about running an internet anything the above are unknown unknowns. You can't go looking for things you don't know exist, so burnout is high.
Citation highly needed. I'm close to quite a few people who run larger instances than that, including my own, and none of them have ever told me about getting subpoenaed. That's exactly the kind of war story we'd tell each other, too.
I've seen no evidence that running a fediverse server is nearly so legally fraught.
I'll go looking for the citation and it may take a couple hours but I'll point out that FWIW my info is drawn from some academic surveying vBulletin and Xenforo site owners in the 2010s so I wouldn't be surprised if they aren't applicable anymore.
No problem. I thought that seemed on the high side but didn’t have any stats to counter with. Which paper does 430k come from? I’d like to squirrel that away for later.
> Federated networks like Mastodon and Lemmy are going to get people well-acquainted with websites shutting down. It's hard work (time, money, etc.) to run these things for people, and people start to really lean on them.
Mastodon is filled with such utterly basic UX issues. You move instances because the old one announces a shutdown? No old posts visible, no import possible. You want to see the history of an old account on another instance? The oldest toot you'll see is the first one that your instance picked up from that account. You have to switch to their instance to see old toots - there's a helpful link at the end of the feed, but it's still annoying. "Trending" topics only carry stuff happening on your server, and most of it is days old garbage. Search is horribly broken and inconsistent.
Federated networks like Mastadon strike me as being centralization at scale.
They don’t appear to solve any of the power dynamics of users and operators - users are still at the mercy of the operator - and they run on either altruism or monetization.
Mastadon appears to have successfully created N copies of the Facebook problem, which is definitely better than where we were.
I like Mastodon, it’s the only Twitter like thing I use.
But I think this just reflects the facts. Centralization works and is highly preferable for many users. Just like in the only big federated service: email.
Yes you can run your own. But there are a lot of costs in terms of time/complexity/knowledge/trust to that.
Outsourcing it to someone else is really nice.
You don’t need one big instance like Twitter was. Having a small handful of big ones works well too.
But the dream some people seemed to have where everyone should run their own instance alone or with a few friends was never going to happen.
>Yes you can run your own. But there are a lot of costs in terms of time/complexity/knowledge/trust to that.
>Outsourcing it to someone else is really nice.
Yeah but the key thing is that you can choose your provider. Email isn't a walled garden that can be enshittified because you can just migrate somewhere else - yes it's a huge pain and has a bunch of drawbacks, but you can do it, and people do do it.
Moving to a different Mastodon instance is a way smaller transition than moving from Twitter to another social media platform entirely.
The Fediverse has a bunch of issues but I don't think we should think about it as "running your own", we should think of it as "choosing the provider that best fits your needs", as many have with Gmail.
> Centralization works and is highly preferable for many users.
I don't think users care about that at all, and if they have it explained to them, hate it. I think the real problem is that we haven't decentralized ownership and decisionmaking, instead we shattered big dictatorships into little fiefdoms, often run by local gangs (as one would expect.) Arguing that federation should automatically solve our problems with social media is like the US argument for "state's rights." You had one problem, now you have 50.
This is also exacerbated by the fact that people can't migrate. That would seem like it should be a developer priority to enable competition between instances, but instead people get irritated when asked about it at all. Every post locks you in farther to a particular instance. If people can leave on a whim, bad instances would starve. Instead of people being able to vote with their feet, the politics of mastodon all revolve around punishing other instances for various examples wrongthink by defederating. So now it's little fiefdoms at war with each other, you have to be in the in-crowd of your likely randomly chosen instance to have a say about it, and if you leave you lose everything.
As always, it depends. I'm on a mastodon instance centered around a fairly specific topic, whose members donate (more than) enough to cover the costs of running the instance.
Of course, it still relies on the benevolence of the guy who runs and maintains the instance. He actually takes a fee out of the donations each month to pay for his time, but it's a token amount.
I think OP’s point is that most users don’t choose to do so. Whether because of lack of ability, interest, time, whatever, people would mostly rather just be users.
This is why I believe that Bluesky and the AT protocol is a significantly more attractive system than Mastodon and ActivityPub. Frankly, we’ve tried the kind of system ActivityPub offers before: a decentralized server network ultimately forming one big system, and the same problems have inevitably popped up every time.
XMPP tried to do it for chat. All the big players adopted it and then either realized that the protocol wasn’t complex enough for the features they wanted to offer or that it was much better financially to invest in a closed system. Sometimes both. The big providers split off into their own systems (remember, Google Talk/Hangouts/Chat and Apple iChat/FaceTime both started out as XMPP front-ends) and the dream of interconnected IMing mostly died.
RSS tried to do it for blogs. Everyone adopted it at first, but eventually content creators came to the realization that you can’t really monetize sending out full-text posts directly in any useful way without a click back to the originating site (mostly defeating the purpose), content aggregators realized that offering people the option to use any front-end they wanted meant that they couldn’t force profitable algorithmic sorts and platform lock-in, and users overwhelmingly wanted social features integrated into their link aggregators (which Google Reader was famously on the cusp of implementing before corporate opted to kill it in favor of pushing people to Google+; that could have potentially led to a very different Internet today if it had been allowed to release). The only big non-enthusiast use of RSS that survives is podcasts, and even those are slowly moving toward proprietary front-ends like Spotify.
Even all the way back to pre-Web protocols: IRC was originally a big network of networks where every server could talk to every other server. As the system grew, spam and other problems began to proliferate, and eventually almost all the big servers made the decision to close off into their own internal networks. Now the multi-server architecture of IRC is pretty much only used for load balancing.
But there’s two decentralized systems that have survived unscathed: the World Wide Web over HTTP and email over SMTP. Why those two? I believe that it’s because those systems are based on federated identities rather than federated networks.
If you have a domain name, you can move the website attached to it to any publicly routable server and it still works. Nobody visiting the website even sees a difference, and nobody linking to your website has to update anything to stay “connected” to your new server. The DNS and URL systems just work and everyone just locates you automatically. The same thing with email: if you switch providers on a domain you control, all the mail still keeps being routed to you. You don’t have to notify anyone that anything has changed on your end, and you still have the same well-known name after the transition.
Bluesky’s killer feature is the idea of portable identities for social media. The whole thing just ties back to a domain name: either one that you own or a subdomain you get assigned from a provider. That means that picking a server isn’t something the average person needs to worry about, you can just use the default and easily change later if you want to and your entire identity just moves with you.
If the server you’re on evaporates, the worst thing that you lose is your activity, and that’s only if you don’t maintain any backups somewhere else. For most people, you can just point your identity at a different server, upload a backup of your old data, and your followers don’t even know anything has changed. A sufficiently advanced client could probably even automate all of the above steps and move your whole identity elsewhere in one click.
Since the base-level object is now a user identity rather than a server, almost all of the problems with ActivityPub’s federation model go away. You don’t deal with blocking bad servers, you just block bad people (optionally using the same sorts of “giant list” mechanisms already available for places like Twitter). You don’t have to deal with your server operator getting themself blacklisted from the rest of the network. You don’t have to deal with your server operator declaring war on some other server operator and suddenly cutting you off from a third of your followers.
People just publish their posts to a server of their choice, others can fetch those posts from their server, the server in question can be moved wherever without affecting anything for those other users, and all of the front-end elements like feed algorithms, post display, following lists and block lists, and user interface options could either be handled on the client-side or by your choice of (transferable) server operator. Storage and bandwidth costs for text and (reasonable) images are mostly negligible at scale, and advertising in clients, subscription fees, and/or offering ancillary services like domain registration could easily pay for everything.
ActivityPub sounds great to nerds who understand all of this stuff. But it’s too complicated for the average social media user to use, and too volatile for large-scale adoption to take off.
AT protocol is just as straightforward to understand as email (“link a website domain if you already have one or just register for a free one on the homepage, and you can easily change in the future”), doesn’t require any special knowledge to utilize, and actually separates someone’s identity and content from the person running the server. Mastodon is 100 tiny Twitters that are somewhat connected together, AT actually lets everyone have their own personal Twitter and connect them all together in a way that most people won’t even notice.
Good post of historical reminders and I appreciate the framing of bluesky's identity approach. Never was sold on Fediverse/ActivityPub as being it and not a fan yet of Bluesky's slow-building-in-public approach but am intrigued by this key facet of the main role the personal domain takes. How can one easily change/migrate their AT identity if they change domains? How is their whole social history transferrable? Like that was one of the problems/unclear things to most about Mastodon - that it actually wasn't that easy to move instances because sure your identity could move but your posts would be on the old instance, so it wasn't really that portable. I'm all about the permanence and data preservation, so I don't want to commit to a platform now without assured control over my data and ability to maintain history/identity in a move. Have enjoyed the centralization and longevity for too long on a place like Twitter to get all loose and ephemeral now.
> But the recent Mastodon upgrade has caused a significant amount of performance degradation, and I think the only way to really solve it is going to be to throw a lot of money into hardware.
I found the latest upgrade also making some odd UX decisions. Content warnings got a weird new styling and it's not clear anymore how to hide images separately from hiding the text.
Are the mastodons okay?
There are good things too, don't get me wrong, like grouping notifications instead of getting a notification flood on a popular toot. That's nice. But what's up with perf regressions and (in my opinion) UX regressions?
Why are these Twitter clones always so resource intensive and finicky? Twitter is just "IRC in reverse", if you take literal descriptions transitive relations in IRC(v2) and moved around nouns, lots of it should apply to Twitter/Bluesky/Mastodon well. RDB gurus can probably recreate lots of APIs as tables on bare MySQL too.
It doesn't make sense to me that such a thing take so much dev and ops cost compared to IRCv2 servers, other than for the fact that modern webdev just so happens to be extremely bloated, especially when extremely competent and high spirited developers are giving up like this.
Are we doomed to keep adding more RAM and more disk and more bandwidth to catch up with ever-growing bloat?
Not related to Mastodon, but in the case of Matrix, the server software ranges from "runs on a raspberry pi with zero issues" (Conduit) to "even with 16 GB of RAM, federating with a large enough room will exhaust Python's heap" (Synapse).
In the case of Conduit, a Matrix server with a few private rooms and users consumed only 32 MB of RAM, using RocksDB for storage. The equivalent on Synapse required about 5x as much memory, despite using SQLite. In practice, Synapse instances will use Postgres since many appservice plugins specifically require Postgres and don't support SQLite. Not to mention, SQLite isn't optimized for frequent, concurrent writes.
I do sincerely think the choice of Rails, and the fact Ruby only got a compiler people use recently, means that most Ruby programs require fairly beefy processors and plenty of memory in order to keep up with a few hundred clients.
Of course, I am extrapolating based off my experience running Synapse (a Matrix server) with Postgres. There is a chance Mastodon scales much better.
>extremely competent and high spirited developers are giving up like this.
I'm pretty sure the median IRC server runs for much less than 7.5 years. I don't think anyone expects volunteers to dedicate decades of their life to admin duty. and it seems fine and healthy for the ecosystem that he is telling people they have a few months to move their bots to a different server.
From what I understand, IRC doesn't hold messages for extended periods of time or allow media uploads, while fediverse does, so that's one big difference.
I think there are issues with hotspots. The most popular tweets are seen by a big chunk of the userbase, which means they have to operate on a fanout model where each tweet is pushed to individual followers.
I believe IRC doesn't operate like that. The messages delivered to each user don't need to be retained, and I assume the size of the largest channels is in the tens or hundreds of thousands.
Because twitter isn't just IRC in reverse?
Scrollback.
Aw too bad, this has been a really useful service. I wonder if anyone wants to pick it up? The post mentions part of the problem is Mastodon's implementation being a poor match to high volume bots. You could imagine other architectures that were more efficient for this use case, it'd be a fun yak shaving exercise.
If anyone needs to migrate their own projects I've had good luck with feed2toot, to post RSS to a Mastodon account on a ordinary server. It's been around a long time now and seems reliable.
I wholly appreciated the openness to accepting bot accounts, migrated some projects from twitter there during the big exoduses in the last couple years. And while it worked for bot purposes, fun to tinker with etc...(not unlike twitter tbf) it was just some server in space a blip in the fediverse and traction and lack of proper network effects for accounts meant it wasn't much use.
I'm not hot on the fediverse in general, and this just sours me on it a bit more. A bunch of dedicated admins keeping instances going, basically running hobby servers/websites like it was the 90s/early 00s, is never gonna work for the kind fo scale services grow to these days. I know not everything requires scale and lots of ppl are happy existing in their little silos, but that's just it, they're silos. Might as well be back on seperate forums for our seperate interests again. When you want the power of a mix of accounts/networks/interests everything balloons and can't be run with funds and larger centralization. Sigh. It's a tough one and has yet to be solved in full, with any existing approaches all sort of half-solutions. Maybe that's the way forwards in general (an internet of islands) but it sucks to have things going up and down and having to migrate around the net (with or without our own data) like nomads.
> When you want the power of a mix of accounts/networks/interests everything balloons and can't be run with funds and larger centralization.
I don't really get the appeal of this. Different topics/interests often demand different moderation and forum features. Trying to shove everything into the lowest common denominator of social media results in things like people posting essays as screenshots of Apple Notes.
The only obvious benefit I see of this kind of large scale centralization is for marketing. And that's not a benefit to me as a user.
The benefit of a centralized forum like Reddit, for a user like me, is the ability to use one account for everything (i.e. easy to join new forums) and the ability to have everything on one algorithmic or chronological feed.
Traditional forums can solve the former by using social sign-ins, and the latter by having RSS. However, support for either of this is inconsistent, so it's usually easier to just use subreddits.
Finally, money. Forums cost money to host, while subreddits don't.
Lemmy works like that more or less. You can subscribe to communities on other instances and interact with them.
Ack. I have a bot there. I appreciate the extended warning time.
This is incredibly sad; botsin.space has been a steady stream of joy for me over the years. Here's hoping a bunch of alternatives pop up.
What was it?
A mastodon instance where bots were welcomed.
It's always nice when Fediverse servers have a sunsetting period which allows account migration to other servers.
>I'd like to thank everyone who has ever run a bot on botsin.space and gotten joy out of it
I definitely got joy out of setting up a bot on it. Huge thanks to colin for making it so easy.
Website is being hugged:
https://archive.ph/6Krrp
I've had lots of fun calling it bot sin space. It will be missed.
Damn, I asked the other day if it was down after noticing a ton of timeouts in sidekiq. Reached out and Colin said they were looking into it. Guess fixing it up just proved to be too much.
>Over the years, the server has grown to have around a few thousand active accounts, which isn't all that many. However, they've generated something like 32 million statuses. Just to put that in perspective, mastodon.social has over 2 million users, who have generated around 110 million statuses.
unsurprising that the bots would outpace organic users, but wow, what a ratio. i'd be curious to see this data charted over time
It makes sense. Assuming the average bot toots ~once an hour (24 times a day) and has been tooting for two years, you get on the order (nearest multiple of 5000) of 20,000 toots per user, which works out to 1600 users.
Also, there's a measuring change from active accounts to plain old users. I don't know the proportion that are active, but if I recall right, the fediverse as a whole had under 1 million active users. Assuming 500,000 active accounts that pull all the weight, it's 220 toots per user on average.
RIP!
Thanks for hosting this all those years. I'll try to find a new home for my bot.
Thanks, botsin.space! It was fun while it lasted!
Maybe requiring payment could've worked out instead of shutting down?
Safe travels to a known name.
One of the reasons I maintain a node with only one user is I fear the day I'll be responsible for other people's social media presence; I could easily see myself going "It's just a few thousand users" and the next thing I know I'm asking whether I can keep this thing going (and agonizing over what it'll do to my users to cut the service). And unlike Colin, I despise Rails and wouldn't have the patience to hammer on it when it starts to misbehave.
Props to Colin having the guts to take the risk.
Federated networks like Mastodon and Lemmy are going to get people well-acquainted with websites shutting down. It's hard work (time, money, etc.) to run these things for people, and people start to really lean on them.
It's almost novel now days getting sucked into something that shuts down. killedbygoogle.com is a meme partly I think because websites shutting down is just so uncommon in areas that we get personally invested in.
I run my own Lemmy instance just for my self and even that can be trying sometimes. I enjoy using it instead of reddit, but one day I will probably shut it down and be sad.
I'm starting to think nostr was barking up the right tree after all. Put as much complexity into the client as possible and make the servers dumb and completely uncoordinated, utterly interchangeable. Spam your broadcasts to any relay that will listen. No idea if it actually works (I've read a lot about nostr and AT proto but never used either of them) but I think it's very obvious that any system that deeply relies on some company that everyone becomes extremely reliant on (including AT proto/Bluesky) is only a couple steps away from the same sort of problems as centralization.
Of course, the real gold standard would be P2P, if only it could work. But... mobile phones can't burn battery running P2P clients in the background, everyone's under a NAT these days, and some types of software (like microblogging networks) would be horrifically intractable as a P2P system.
Oh well. At the very least, I really love the concept of Decentralized Identifiers (DIDs). I'd like to see more stuff like that.
The atproto team came from the p2p space. We had a lot of experience running client-side computation. There are challenges you can try to solve in that design -- key sync, key recovery, reliable hosting, reliable discovery, etc -- but what you can't get around is the need to run queries against aggregations of data. Even scales that we consider "mid" start to strain on the client-driven model. Federated queries might be able to solve it, but it's extremely challenging to get reliable performance out of that. The design we landed on was replaceable big nodes.
The common comparable people raise is email/gmail, but we used the DID system and account portability to try to get a better outcome on provider migration. It's hopefully more like web/google -- which still has the centralizing pressures of scale benefits, but hopefully injects enough fluidity in the system to move the situation forward. Sometimes, you pick the design that moves the ball down the field, not the one that guarantees a touchdown. If somebody wanted to improve on what we've done, I'd tell them to focus on the federated queries problem.
In theory AT proto doesn't seem like a bad design. I've read a fair bit of the docs, although mostly skimming. (I've been meaning to read the paper on Merkle Search Trees so I can figure out what exactly is going on with PDSes.)
On the other hand, in practice it seems like the AT proto infrastructure is still very centralized for now. DIDs are excellent in theory, but everyone is using PLC DIDs, which depends on the centralized plc.directory. You can run your own PDSes, but there's only one relay for now (that I am aware of.) I also don't think there is more than one instance of the Bluesky AppView, and the official instance is locked into the Bluesky Moderation Service, which seems to limit the usefulness of some of the censorship resistance of the protocol.
I'm not sure how much of that is social problems rather than technical, but I worry that since Bluesky and AT proto are gaining massive popularity with this status quo (millions of users!) it'll have a problem akin to Matrix.org, where in practice almost everyone is using the same infrastructure anyways.
It's still relatively early days, but millions of people is definitely enough to where you start to hit problems with having everyone under one roof. I really hope we get to see how things play out when more of the network is operated independently.
The necessary future is more providers, more relays, a move of PLC to an independent body, and more DID methods.
I will also say - there are ~100 self-hosting PDSes in the network, about 25 relay consumers, 3 alternative appviews that I know of (smoke signals, frontpage.fyi, and whitewind), the firehose & backfill are fully available, the specs are getting fairly complete, and the software is open source. This is a priority for us.
fwiw roughly 50% of Matrix is on the matrix.org instance currently. we consider this a bug, but also prioritise ease of onboarding over decentralisation purity ideology.
I hate to be a downer but there's a lot of things Matrix could prioritize over decentralization. That said, the decentralization works pretty badly. Large federated joins are somewhere between comically slow and horrifically slow. Status does not seem to work well across federation either.
I'm also a bit miffed that Dendrite was positioned as a "next generation" Matrix server but now it feels nearly orphaned with missing support for newer features, issues with various appservice bridges, few updates at a very slow pace, and no migration path out in sight. I know it came with a fair number of disclaimers, but that still bums me out as it seemed like it would be okay for a small non-critical homeserver, and now it seems likely I'll have to engineer my own path out when clients finally stop working with Dendrite. (It already happened once...)
You have no idea how bad I want to love Matrix, but frankly if it was due to a focus on usability that decentralization "purity" has suffered, it simply does not show in the resulting usability improvements over years of time. Sorry to be harsh.
if you feel miffed, imagine how the Dendrite team feels, given the lack of funding which means it is on best-effort dev currently.
if you’re interested in progress on Matrix, https://matrix.org/blog/2024/10/29/matrix-2.0-is-here/ is where it’s at.
I did see the Matrix 2.0 announcement, though for obvious reasons I can't actually use any of the features listed in it. Obviously, improvements to the basic chat functions of Matrix would be great. For now though, I am stuck with the reality that joining a large federated channel sometimes takes more than 6 hours. I wish I were exaggerating.
edit: I guess though that faster room joins weren't a part of Matrix 2.0. Actually, I don't really have a huge problem with the sync taking too long personally. So maybe Matrix 2.0 wouldn't bring that big of an improvement for me anyway.
ActivityPub seems to require a lot of hardware resources in order to run properly, which is unfortunate. It’s not something I would ever want to run myself, especially to the public.
> ActivityPub seems to require a lot of hardware resources in order to run properly
Correction: "The Mastodon software requires a lot of hardware resources in order to run properly"
Alternatives like GotoSocial, Akkoma, even Honk are much less resource intensive.
Please be wary to conflate ActivityPub with the code on top of it, like Mastodon for instance; which is, on both front and back-end, proven to be resource intensive and difficult/costly to scale especially over time. (the older the dbs get, the more inactive users pile up, etc.)
Versus something like Pleroma; which I've used since it's inception, being incredibly janky and lightweight, prone to breaking, but later versions have mostly ironed out most of those catastrophic bugs. It has it's own challenges as well, but it does historically scale better, is more flexible, and less intensive per instance
One demands a lot of money and time, where the other demands a lot of time and not so much on the money side. I'm not going to spend the time to give you a history of pleroma/mastodon instances, as it's a controversial history at best and there's a lot of people who know little, yet who will believe themselves an oracle. (ofc that could also be me so take it with a grain of salt and all) Though if you are willing to read through a bunch of highly opinionated accounts, and you pay close attention to what actually happened, the answer is pretty clear.
ActivityPub is intensive, but not the main culprit.
I wonder how much of that is endemic to ActivityPub or how much is about the software stack itself
I don’t mean to be glib, just wondering if things can be “done better”
I'm guessing the syncing process between instances is really brutal on resources. Imagine constantly syncing external databases for your service to function properly.
I thought AP was push-based though? Might be wrong of course, and I think stuff like media is its own thing
Eh, it's not new to federated sites. Many of the web sites I frequented a decade a go are dead. Or enshittified useless so I visit them rarely anymore.
It was sad to see say, the "user friendly" webcomic go away. But I enjoyd it in the time. Just live the moment. Don't expect even the big websites and apps of today to last - not at least in the form you enjoy.
I, too, ran my own instance. I enjoyed it for some time but I've now moved to the omg.lol ecosystem. I feel that by paying a little money for it that I have a higher chance of the server not shutting down.
Besides the monetary costs of operating small fora there are also significant competence hurdles. Site owners who manage to hit a couple thousand users have to figure out spam handling, automated content moderation (including photoDNA and the required reporting if they host images), registering a DMCA agent with the copyright office, setting up an LLC, assessing their needs for COPPA, GDPR and CCPA, their site's tax situation and anything they may want to do involving employing others (such as T&S) without getting burnt out. The median size for a forum getting its first subpoena is 4,300 users.
Managing all of that is easily learnable in a couple months if they have time, disposable income and few distractions but surprisingly few people who have site management thrust upon them know about these things in advance. To most people who think about running an internet anything the above are unknown unknowns. You can't go looking for things you don't know exist, so burnout is high.
Citation highly needed. I'm close to quite a few people who run larger instances than that, including my own, and none of them have ever told me about getting subpoenaed. That's exactly the kind of war story we'd tell each other, too.
I've seen no evidence that running a fediverse server is nearly so legally fraught.
I'll go looking for the citation and it may take a couple hours but I'll point out that FWIW my info is drawn from some academic surveying vBulletin and Xenforo site owners in the 2010s so I wouldn't be surprised if they aren't applicable anymore.
Update: My apologies. I've totally confabulated old blog posts from Antone Johnson with TSPA articles and some Steven Myers papers.
The actual figure is a median of 1 subpoena per annum per 430k users if the majority of the users are under 30.
No problem. I thought that seemed on the high side but didn’t have any stats to counter with. Which paper does 430k come from? I’d like to squirrel that away for later.
> Federated networks like Mastodon and Lemmy are going to get people well-acquainted with websites shutting down. It's hard work (time, money, etc.) to run these things for people, and people start to really lean on them.
Mastodon is filled with such utterly basic UX issues. You move instances because the old one announces a shutdown? No old posts visible, no import possible. You want to see the history of an old account on another instance? The oldest toot you'll see is the first one that your instance picked up from that account. You have to switch to their instance to see old toots - there's a helpful link at the end of the feed, but it's still annoying. "Trending" topics only carry stuff happening on your server, and most of it is days old garbage. Search is horribly broken and inconsistent.
Federated networks like Mastadon strike me as being centralization at scale.
They don’t appear to solve any of the power dynamics of users and operators - users are still at the mercy of the operator - and they run on either altruism or monetization.
Mastadon appears to have successfully created N copies of the Facebook problem, which is definitely better than where we were.
I like Mastodon, it’s the only Twitter like thing I use.
But I think this just reflects the facts. Centralization works and is highly preferable for many users. Just like in the only big federated service: email.
Yes you can run your own. But there are a lot of costs in terms of time/complexity/knowledge/trust to that.
Outsourcing it to someone else is really nice.
You don’t need one big instance like Twitter was. Having a small handful of big ones works well too.
But the dream some people seemed to have where everyone should run their own instance alone or with a few friends was never going to happen.
>Yes you can run your own. But there are a lot of costs in terms of time/complexity/knowledge/trust to that.
>Outsourcing it to someone else is really nice.
Yeah but the key thing is that you can choose your provider. Email isn't a walled garden that can be enshittified because you can just migrate somewhere else - yes it's a huge pain and has a bunch of drawbacks, but you can do it, and people do do it.
Moving to a different Mastodon instance is a way smaller transition than moving from Twitter to another social media platform entirely.
The Fediverse has a bunch of issues but I don't think we should think about it as "running your own", we should think of it as "choosing the provider that best fits your needs", as many have with Gmail.
> Centralization works and is highly preferable for many users.
I don't think users care about that at all, and if they have it explained to them, hate it. I think the real problem is that we haven't decentralized ownership and decisionmaking, instead we shattered big dictatorships into little fiefdoms, often run by local gangs (as one would expect.) Arguing that federation should automatically solve our problems with social media is like the US argument for "state's rights." You had one problem, now you have 50.
This is also exacerbated by the fact that people can't migrate. That would seem like it should be a developer priority to enable competition between instances, but instead people get irritated when asked about it at all. Every post locks you in farther to a particular instance. If people can leave on a whim, bad instances would starve. Instead of people being able to vote with their feet, the politics of mastodon all revolve around punishing other instances for various examples wrongthink by defederating. So now it's little fiefdoms at war with each other, you have to be in the in-crowd of your likely randomly chosen instance to have a say about it, and if you leave you lose everything.
As always, it depends. I'm on a mastodon instance centered around a fairly specific topic, whose members donate (more than) enough to cover the costs of running the instance.
Of course, it still relies on the benevolence of the guy who runs and maintains the instance. He actually takes a fee out of the donations each month to pay for his time, but it's a token amount.
> users are still at the mercy of the operator
Mastodon allows you to be the operator, if you so choose.
I think OP’s point is that most users don’t choose to do so. Whether because of lack of ability, interest, time, whatever, people would mostly rather just be users.
> Mastodon allows you to be the operator, if you so choose.
... and if you forget a critical update or you see it too late, you'll get hacked.
Self-hosting anything comes with serious challenges that most people only realize in hindsight.
This is why I believe that Bluesky and the AT protocol is a significantly more attractive system than Mastodon and ActivityPub. Frankly, we’ve tried the kind of system ActivityPub offers before: a decentralized server network ultimately forming one big system, and the same problems have inevitably popped up every time.
XMPP tried to do it for chat. All the big players adopted it and then either realized that the protocol wasn’t complex enough for the features they wanted to offer or that it was much better financially to invest in a closed system. Sometimes both. The big providers split off into their own systems (remember, Google Talk/Hangouts/Chat and Apple iChat/FaceTime both started out as XMPP front-ends) and the dream of interconnected IMing mostly died.
RSS tried to do it for blogs. Everyone adopted it at first, but eventually content creators came to the realization that you can’t really monetize sending out full-text posts directly in any useful way without a click back to the originating site (mostly defeating the purpose), content aggregators realized that offering people the option to use any front-end they wanted meant that they couldn’t force profitable algorithmic sorts and platform lock-in, and users overwhelmingly wanted social features integrated into their link aggregators (which Google Reader was famously on the cusp of implementing before corporate opted to kill it in favor of pushing people to Google+; that could have potentially led to a very different Internet today if it had been allowed to release). The only big non-enthusiast use of RSS that survives is podcasts, and even those are slowly moving toward proprietary front-ends like Spotify.
Even all the way back to pre-Web protocols: IRC was originally a big network of networks where every server could talk to every other server. As the system grew, spam and other problems began to proliferate, and eventually almost all the big servers made the decision to close off into their own internal networks. Now the multi-server architecture of IRC is pretty much only used for load balancing.
But there’s two decentralized systems that have survived unscathed: the World Wide Web over HTTP and email over SMTP. Why those two? I believe that it’s because those systems are based on federated identities rather than federated networks.
If you have a domain name, you can move the website attached to it to any publicly routable server and it still works. Nobody visiting the website even sees a difference, and nobody linking to your website has to update anything to stay “connected” to your new server. The DNS and URL systems just work and everyone just locates you automatically. The same thing with email: if you switch providers on a domain you control, all the mail still keeps being routed to you. You don’t have to notify anyone that anything has changed on your end, and you still have the same well-known name after the transition.
Bluesky’s killer feature is the idea of portable identities for social media. The whole thing just ties back to a domain name: either one that you own or a subdomain you get assigned from a provider. That means that picking a server isn’t something the average person needs to worry about, you can just use the default and easily change later if you want to and your entire identity just moves with you.
If the server you’re on evaporates, the worst thing that you lose is your activity, and that’s only if you don’t maintain any backups somewhere else. For most people, you can just point your identity at a different server, upload a backup of your old data, and your followers don’t even know anything has changed. A sufficiently advanced client could probably even automate all of the above steps and move your whole identity elsewhere in one click.
Since the base-level object is now a user identity rather than a server, almost all of the problems with ActivityPub’s federation model go away. You don’t deal with blocking bad servers, you just block bad people (optionally using the same sorts of “giant list” mechanisms already available for places like Twitter). You don’t have to deal with your server operator getting themself blacklisted from the rest of the network. You don’t have to deal with your server operator declaring war on some other server operator and suddenly cutting you off from a third of your followers.
People just publish their posts to a server of their choice, others can fetch those posts from their server, the server in question can be moved wherever without affecting anything for those other users, and all of the front-end elements like feed algorithms, post display, following lists and block lists, and user interface options could either be handled on the client-side or by your choice of (transferable) server operator. Storage and bandwidth costs for text and (reasonable) images are mostly negligible at scale, and advertising in clients, subscription fees, and/or offering ancillary services like domain registration could easily pay for everything.
ActivityPub sounds great to nerds who understand all of this stuff. But it’s too complicated for the average social media user to use, and too volatile for large-scale adoption to take off.
AT protocol is just as straightforward to understand as email (“link a website domain if you already have one or just register for a free one on the homepage, and you can easily change in the future”), doesn’t require any special knowledge to utilize, and actually separates someone’s identity and content from the person running the server. Mastodon is 100 tiny Twitters that are somewhat connected together, AT actually lets everyone have their own personal Twitter and connect them all together in a way that most people won’t even notice.
Good post of historical reminders and I appreciate the framing of bluesky's identity approach. Never was sold on Fediverse/ActivityPub as being it and not a fan yet of Bluesky's slow-building-in-public approach but am intrigued by this key facet of the main role the personal domain takes. How can one easily change/migrate their AT identity if they change domains? How is their whole social history transferrable? Like that was one of the problems/unclear things to most about Mastodon - that it actually wasn't that easy to move instances because sure your identity could move but your posts would be on the old instance, so it wasn't really that portable. I'm all about the permanence and data preservation, so I don't want to commit to a platform now without assured control over my data and ability to maintain history/identity in a move. Have enjoyed the centralization and longevity for too long on a place like Twitter to get all loose and ephemeral now.
The important bit
> But the recent Mastodon upgrade has caused a significant amount of performance degradation, and I think the only way to really solve it is going to be to throw a lot of money into hardware.
I found the latest upgrade also making some odd UX decisions. Content warnings got a weird new styling and it's not clear anymore how to hide images separately from hiding the text.
Are the mastodons okay?
There are good things too, don't get me wrong, like grouping notifications instead of getting a notification flood on a popular toot. That's nice. But what's up with perf regressions and (in my opinion) UX regressions?