"How can we be most effective with our giving" is a good thought. It's what I thought EA was, like, two weeks ago.
"The movement can literally compute the optimal pattern of giving, such that to be most effective with our giving means donating money to people to ride around on bicycles thinking about how to prevent future AGIs from killing us" is not a good thought, but it turns out to be a description of what EA is like; some of this is on their own website (another thing I've learned: beware any movement that calls its documentation "sequences").
"The moral urgency of giving is such that a reasonable person should seek to maximize their income" is a bad thought. It leads to dark places. "Earning to give" is a reasonable idea: it is probably more effective for people to excel at their jobs and donate the proceeds than for them to phone bank or canvass. But Rationalists can't seem to help falling into these asymptote traps, so that "donating a portion of ones income" becomes "pursue billions in wealth, and clear a path for other billionaires, to enable concentrated giving". That's a demented idea.
The problems here aren't abstract. They just lead to Scott Aaronson withholding moral judgement on Sam Bankman-Fried, strictly because of the logic of the preceding paragraph.
Rationalists brought us Roko’s Basilisk after all. A lot of the leaders don’t seem to have basic sanity checks in their brains that stop their ideas from getting wildly out of control, which can lead to stupid shit like the Basilisk on a small scale but end up justifying fraud on a much larger scale. It’s like fundamentalist religion
> Rationalists brought us Roko’s Basilisk after all.
This reminds me of the Zen story about two monks and a woman (one helped her cross a river, and the other couldn't stop thinking about her for the whole day).
The "Roko's Basilisk" was an edgy comment that was written on a rationalist web forum, and then deleted by the admin. A decade later, some people outside the rationalist community still keep discussing it, which is the only reason why anyone cares.
> A lot of the leaders don’t seem to have basic sanity checks in their brains that stop their ideas from getting wildly out of control
Only in this story, the admin was the one who deleted that comment, not the one who wrote it. Still gets blamed for it.
"The movement can literally compute the optimal pattern of giving, such that to be most effective with our giving means donating money to people to ride around on bicycles thinking about how to prevent future AGIs from killing us" is not a good thought, but it turns out to be a description of what EA is like; some of this is on their own website (another thing I've learned: beware any movement that calls its documentation "sequences").
"The moral urgency of giving is such that a reasonable person should seek to maximize their income" is a bad thought. It leads to dark places. "Earning to give" is a reasonable idea: it is probably more effective for people to excel at their jobs and donate the proceeds than for them to phone bank or canvass. But Rationalists can't seem to help falling into these asymptote traps, so that "donating a portion of ones income" becomes "pursue billions in wealth, and clear a path for other billionaires, to enable concentrated giving". That's a demented idea.
The problems here aren't abstract. They just lead to Scott Aaronson withholding moral judgement on Sam Bankman-Fried, strictly because of the logic of the preceding paragraph.