Generally not, proactive blocking or silencing is reserved for instances that host illicit content like child porn or hate speech. Inter-moderator communication happens between your instance admin/democracy (or whatever structure your instance uses) and other instances.
Traditional tools like muting a user, silencing an instance (if they're spamming up the Federated Timeline on other instances, like Humblr.social does) and blocking (if the mods are total shit) can still be used. Small instances are the bulk of the content, so you aren't about to block them. Large instances like Mastodon.social are generally blocked tho (if your admin/democracy cares about a vibrant Fediverse).
Honestly, I think this is the wrong way of looking at things. The Fediverse isn't really like a single social network that's shattered amongst many instances ... it's more like a bunch of individual, separate hangouts, in the vein of classic forums and IRC chatrooms and fandom websites, which have the ability to talk to each other and share stuff if they want.
It's not so much about "blocking" mastodon.social as not choosing to link up with it - because your instance is a specific community, and actual human social relations thrive when they can choose who they're hanging out with. So you choose to talk to other smaller, more diverse communities that you can actually get to know a good chunk of the people in them, instead of just drinking straight from the firehose that is a big instance.
Federating with another instance is basically two groups of people deciding to hang out together, effectively merging. Federating with mastodon.social means, effectively, merging your community with mastodon.social ... which means submerging it beneath the mass of mastodon.social's much greater activity.
A vibrant Mastodon isn't a more-obtuse recreation of the Twitter experience. Twitter is already Twitter. It's about fostering all the things that Twitter isn't - personal and Dunbar-scaled and locally moderated and diverse.
I think this is inherently a sociological problem than a technological one. When there are enough people in an organically grown group, the community will become unsustainable and then 1) it disintegrates into multiple groups until it becomes stable again or 2) it transforms into a hierarchial, authoritative society where a small group of people makes most of the decisions.
Also, I think Mastodon’s design deliberately steers instances away from becoming large ones; it rather tries to be a nurturer for various small communities and subcultures. It doesn’t view size as a good thing, and that’s totally okay. Nowadays, I think we need less monoculture and more diversity/creativity in our society.
It'll probably end up "solved" in a similar way to email span being "solved" - if global spam starts to become a problem, you'll have big blocklists of spammy instances and general suspicion towards federating with any new instance that doesn't look sufficiently trustworthy.
Malicious actors definitely try to spam and data mine the fediverse, but their accounts get disabled by the local instance admin, and when they try evading that by spinning up a few instances on their own domains to hide their spam or follow bots behind, those instances generally get blocked by instances that see said spam toots or spam followers.
Traditional tools like muting a user, silencing an instance (if they're spamming up the Federated Timeline on other instances, like Humblr.social does) and blocking (if the mods are total shit) can still be used. Small instances are the bulk of the content, so you aren't about to block them. Large instances like Mastodon.social are generally blocked tho (if your admin/democracy cares about a vibrant Fediverse).