The whole population?

Freedom is freedom for minority rights and opinions to not be quashed by the majority. Thus the term "the tyranny of the majority"

A majority can "not care" about systematic racial injustice and other inequities. The consent of the majority doesn't make the system "free".