@Felix Bohmann so many questions, would have been good to number them.

Hello helpers. on my friendica instance (2023-05) i see posts from other instances coming in that were written 12-24 hours in the past.

That is a long delay.

what mechanism causes this?

There is 3 possible reasons I think.
1. Your server is not reachable well, maybe timeouts or so, and hence the posts can not be received and are re-transmitted after some hours
2. or your server actively gets them from remote.
3. Some of your contacts interacted with the post, which caused you to receive the interaction and the the post came with it.

my server seems to be doing fine cpu-wise, sometimes db queries take a bit long (i will move files from db to local storage next, i guess that is is the problem with the slow queries), but the queues are not large etc

Try mysqltuner.pl: https://github.com/major/MySQLTuner-perl#downloadinstallation

I am not sure about the file system storage being faster, if your server is new there should not be so much in the db yet.

Disable avatar caching, that really uses many resources and was a common suggestion the last months.

or is that just the federation code on the external (mostly mastodon) servers not keeping up with delivery?

No, very unlikely.

also, as i understand, my instance sees all public posts from a remote instance once someone on my instance follows a person over there. how is this going to work once meta starts spewing out gigabits of content per second?

That could happen. I would expect that meta and others would not turn this on for every user, it will be opt-in and maybe just on separate systems from the current Facebook/Instagram.

Such public posts can be discarded after a short time, currently the settings usually will be some days, but in such cases you can probably set that to hours, just receive them and delete them again after some time. It would cost a lot of database time.