You are right, multiple desktop environments are not an issue -- unless you are toolkit author; the different corner cases in different Wayland compositors can be issue, but only for toolkit authors. We are talking about such a layer in the stack, where Apple or Microsoft won't even let you play. Anyone, who uses the multiple DEs as an excuse is firmly put into doesn't know what he is talking about box.

Multiple package managers are not an issue either. In this day and age, you should be using Flatpak. That would also allow you to target very specific ABIs in a cross-distribution way, thus removing the fragmentation excuse.