It's not the ISPs themselves, necessarily. You have a box that you own and that sits in someone else's network, where their security policies and practices apply. Those policies and practices are not going to be the same as yours. Perhaps more importantly, other people have physical access to the hardware. That includes employees of the ISP, but perhaps also 3rd parties (think multi-tenant datacenter, for example). That's why you should treat that box as being in a hostile environment.