Matt McAlister's List: Less Wrong 8 - The Craft and the Community

• Behind every exciting, dramatic failure, there is a more important story about a larger and less dramatic failure that made the first failure possible.
• If every trace of religion was magically eliminated from the world tomorrow, then - however much improved the lives of many people would be - we would not even have come close to solving the larger failures of sanity that made religion possible in the first place.
• We have good cause to spend some of our efforts on trying to eliminate religion directly, because it is a direct problem.  But religion also serves the function of an asphyxiated canary in a coal mine - religion is a sign, a symptom, of larger problems that don't go away just because someone loses their religion.
• Consider this thought experiment - what could you teach people that is not directly about religion, which is true and useful as a general method of rationality, which would cause them to lose their religions?  In fact - imagine that we're going to go and survey all your students five years later, and see how many of them have lost their religions compared to a control group; if you make the slightest move at fighting religion directly, you will invalidate the experiment.  You may not make a single mention of religion or any religious belief in your classroom, you may not even hint at it in any obvious way.  All your examples must center about real-world cases that have nothing to do with religion.
• If you can't fight religion directly, what do you teach that raises the general waterline of sanity to the point that religion goes underwater?

• We now know this person is not applying any technical, explicit understanding of...

   
   
  • ...what constitutes evidence and why;
   
  • ...Occam's Razor;
   
  • ...how the above two rules derive from the lawful and causal operation of minds as mapping engines, and do not switch off when you talk about tooth fairies;
   
  • ...how to tell the difference between a real answer and a curiosity-stopper;
   
  • ...how to rethink matters for themselves instead of just repeating things they heard;
   
  • ...certain general trends of science over the last three thousand years;
   
  • ...the difficult arts of actually updating on new evidence and relinquishing old beliefs;
   
  • ...epistemology 101;
   
  • ...self-honesty 201;
• That's what the dead canary, religion, is telling us: that the general sanity waterline is currently really ridiculously low.  Even in the highest halls of science.

5 more annotations...

• Even if you limit yourself to people who can derive Bayes's Theorem - which is going to eliminate, what, 98% of the above personnel? - that's still not a whole lot of rationality.  I mean, it's a pretty basic theorem.
• Sometimes I see a hint, an echo, of what I think should be possible, when I read the writings of folks like Robyn Dawes, Daniel Gilbert, Tooby & Cosmides.  A few very rare and very senior researchers in psychological sciences, who visibly care a lot about rationality - to the point, I suspect, of making their colleagues feel uncomfortable, because it's not cool to care that much.  I can see that they've found a rhythm, a unity that begins to pervade their arguments -
• We practice our skills, we do, in the ad-hoc ways we taught ourselves; but that practice probably doesn't compare to the training regimen an Olympic runner goes through, or maybe even an ordinary professional tennis player.
• The chief obstacle to doing this the way it really should be done, is the difficulty of testing the results of rationality training programs, so you can have evidence-based training methods. 
• There are experiments done now and again on debiasing interventions for particular biases, but it tends to be something like, "Make the students practice this for an hour, then test them two weeks later."  Not, "Run half the signups through version A of the three-month summer training program, and half through version B, and survey them five years later."
• Daniel Burfoot brilliantly suggests that this is why intelligence seems to be such a big factor in rationality - that when you're improvising everything ad-hoc with very little training or systematic practice, intelligence ends up being the most important factor in what's left.
• Why do most "rationalists" just seem like ordinary people, perhaps of moderately above-average intelligence, with one more hobbyhorse to ride?
• There are other arts, I think, that a mature rationality training program would not neglect to teach, which would make me stronger and happier and more effective - if I could just go through a standardized training program using the cream of teaching methods experimentally demonstrated to be effective.  But the kind of tremendous, focused effort that I put into creating my single sub-art of rationality from scratch - my life doesn't have room for more than one of those.
• Is it more important to hit people than to think?
• Now - they got somewhere with that aspiration, unlike a thousand other aspirations of awesomeness that failed, because they could tell when they had hit someone; and the schools competed against each other regularly in realistic contests with clearly-defined winners.
• But before even that - there was first the aspiration, the wish to become stronger, a sense that more was possible.
• an adopted mode of conversational attire with few or no real consequences

10 more annotations...

• The art, the dojo, and the sensei are seen as sacred.
• If your teacher takes you aside and teaches you a special move and you practice it for 20 years, you have a large emotional investment in it, and you'll want to discard any incoming evidence against the move.
• a martial art can't be learned from a book, so they have to trust the teacher
• Deference to famous historical masters. 
• the problem is that we live in poverty but continue to act as though we live in luxury, as though we can safely afford to believe whatever we’re told
• things went downhill to the point that we have videos on Youtube of supposed Nth-dan black belts being pounded into the ground by someone with real fighting experience.

4 more annotations...

• But there is no statistically discernible difference between the many schools of psychotherapy.  There is no discernible gain from years of expertise.

   

  And there's also no discernible difference between seeing a psychotherapist and spending the same amount of time talking to a randomly selected college professor from another field.  It's just talking to anyone that helps you get better, apparently.
• So far as I can discern, this was the way you picked up prestige in the field - not by discovering an amazing new technique whose effectiveness could be experimentally verified and adopted by all; but, rather, by splitting off your own "school", supported by your charisma as founder, and by the good stories you told about all the reasons your techniques should work.
• The act of creating a new measurement creates new science; if it's a good measurement, you get good science.

1 more annotation...

• I suggest stratifying verification methods into 3 levels of usefulness:

   
   
  • Reputational
   
  • Experimental
   
  • Organizational
• But suppose you wanted to put happier people in positions of power - pay happy people to train other people to be happier, or employ the happiest at a hedge fund?  Then you're going to need some test that's harder to game than just asking someone "How happy are you?"
• This question of verification methods good enough to build organizations, is a huge problem at all levels of modern human society.
• I consider it drop-dead obvious that the task of verifying acquired skills and hence the power to grant degrees should be separated from the institutions that do the teaching, but let's not go into that

• Reputational, experimental, organizational:

   
   
  • Something the masters and schools can do to keep it real (realistically real);
   
  • Something you can do to measure each of a hundred students;
   
  • Something you could use as a test even if people have an incentive to game it.

3 more annotations...

• But none of those donors posted their agreement to the mailing list.  Not one.

   

   

  So far as any of those donors knew, they were alone.  And when they tuned in the next day, they discovered not thanks, but arguments for why they shouldn't have donated.  The criticisms, the justifications for not donating - only those were displayed proudly in the open.
• the Dark Side may require non-obvious skills

• The respected leader speaks, and there comes a chorus of pure agreement: if there are any who harbor inward doubts, they keep them to themselves.  So all the individual members of the audience see this atmosphere of pure agreement, and they feel more confident in the ideas presented - even if they, personally, harbored inward doubts, why, everyone else seems to agree with it.

   

  ("Pluralistic ignorance" is the standard label for this.)

• Therefore, as a group, the Light Side will always be divided and weak.  Atheists, libertarians, technophiles, nerds, science-fiction fans, scientists, or even non-fundamentalist religions, will never be capable of acting with the fanatic unity that animates radical Islam.  Technological advantage can only go so far; your tools can be copied or stolen, and used against you.  In the end the Light Side will always lose in any group conflict, and the future inevitably belongs to the Dark.

   

  I think that one's reaction to this prospect says a lot about their attitude towards "rationality".
• I have a problem with the idea that the Dark Side, thanks to their pluralistic ignorance and affective death spirals, will always win because they are better coordinated than us.
• You would think, perhaps, that real rationalists ought to be more coordinated?  Surely all that unreason must have its disadvantages?  That mode can't be optimal, can it?
• "It is dangerous to be half a rationalist."
• I can think of ways to sabotage someone's intelligence by selectively teaching them certain methods of rationality
• Yes, a group which can't tolerate disagreement is not rational.  But if you tolerate only disagreement - if you tolerate disagreement but not agreement - then you also are not rational.  You're only willing to hear some honest thoughts, but not others.  You are a dangerous half-a-rationalist.
• Doing worse with more knowledge means you are doing something very wrong.  You should always be able to at least implement the same strategy you would use if you are ignorant, and preferably do better.  You definitely should not do worse.  If you find yourself regretting your "rationality" then you should reconsider what is rational.
• politically opinionated students with more knowledge of the issues reacted less to incongruent evidence, because they had more ammunition with which to counter-argue only incongruent evidence.
• If I were setting forth to systematically train rationalists, there would be lessons on how to disagree and lessons on how to agree, lessons intended to make the trainee more comfortable with dissent, and lessons intended to make them more comfortable with conformity.
• But why is it not okay to practice that, while it is okay to practice disagreeing with everyone else in the crowd?  Are you never going to have to agree with the majority?
• We signal our superior intelligence and our membership in the nonconformist community by inventing clever objections to others' arguments.  Perhaps that is why the atheist/libertarian/technophile/sf-fan/Silicon-Valley/programmer/early-adopter crowd stays marginalized, losing battles with less nonconformist factions in larger society.  No, we're not losing because we're so superior, we're losing because our exclusively individualist traditions sabotage our ability to cooperate.

  • Crucial
• The best informal definition I've ever heard of rationality is "That which can be destroyed by the truth should be." 
• "That which can be destroyed by the truth should be," and "That which the truth nourishes should thrive."
• I think that there are parts of life where we should learn to applaud strong emotional language, eloquence, and poetry.
• You need both sides of it - the willingness to turn away from counterproductive causes, and the willingness to praise productive ones; the strength to be unswayed by ungrounded appeals, and the strength to be swayed by grounded ones.
• People simply announced their pledges - not with grand drama and pride, just simple announcements - and that encouraged others to do the same.  Those who had nothing to give, stayed silent; those who had objections, chose some later or earlier time to voice them.  That's probably about the way things should be in a sane human community - taking into account that people often have trouble getting as motivated as they wish they were, and can be helped by social encouragement to overcome this weakness of will.

  • #useful

17 more annotations...

• "A good 'atheistic hymn' is simply a song about anything worth singing about that doesn't happen to be religious."
• If you suddenly deleted religion from the world, the largest gap left would not be anything of ideals or morals; it would be the church, the community.  Among those who now stay religious without quite really believing in God - how many are just sticking to it from wanting to stay with their neighbors at the church, and their family and friends?  How many would convert to atheism, if all those others deconverted, and that were the price of staying in the community and keeping its respect?
• a church is ostensibly about worship, and a workplace is ostensibly about the commercial purpose of the organization.  Neither has been carefully optimized to serve as a community.
• That the cost of supporting a church and a pastor is expensive, compared to the number of different communities who could time-share the same building for their gatherings;
• If you were doing it over from scratch... then you might have a big building that could be used for the occasional wedding, but it would be time-shared for different communities meeting at different times on the weekend, and it would also have a nice large video display that could be used for speakers giving presentations, lecturers teaching something, or maybe even showing movies
• to the extent that churches provide community, they're trying to do it without actually admitting that this nearly all of what people get out of it.  Same thing with the corporations whose workplaces are friendly enough to serve as communities; it's still something of an accidental function.
• anything that people have in common, especially any goal they have in common, tends to want to define a community.  Why not take advantage of that?
• Should the Earth last so long, I would like to see, as the form of rationalist communities, taskforces focused on all the work that needs doing to fix up this world.
• Tasks like that need communities anyway, and this Earth has plenty of work that needs doing, so there's no point in waste.  We have so much that needs doing - let the energy that was once wasted into the void of religious institutions, find an outlet there.  And let purposes admirable without need for delusion, fill any void in the community structure left by deleting religion and its illusionary higher purposes.
• Strong communities built around worthwhile purposes:  That would be the shape I would like to see for the post-religious age
• as long as you've got a building with a nice large high-resolution screen anyway, I wouldn't mind challenging the idea that all post-adulthood learning has to take place in distant expensive university campuses with teachers who would rather be doing something else

9 more annotations...

• You can't capture all the value you create
• if both atheists and anti-Prohibitionists are willing to step back a bit and say a bit about the general, abstract principle of confronting a discomforting truth that interferes with a fine righteous tirade, then both atheism and marijuana legalization pick up some of the benefit from both efforts.

• A certain person who was taking over as the president of a certain organization once pointed out that the organization had not enjoyed much luck with its message of "This is the best thing you can do", as compared to e.g. the X-Prize Foundation's tremendous success conveying to rich individuals of "Here is a cool thing you can do."

   

  This is one of those insights where you blink incredulously and then grasp how much sense it makes.  The human brain can't grasp large stakes and people are not anything remotely like expected utility maximizers, and we are generally altruistic akrasics.  Saying, "This is the best thing" doesn't add much motivation beyond "This is a cool thing".  It just establishes a much higher burden of proof.  And invites invidious motivation-sapping comparison to all other good things you know (perhaps threatening to diminish moral satisfaction already purchased).
• Dollars come out of different mental accounts, cost different amounts of willpower (the true limiting resource) under different circumstances,
• people want to spread their donations around as an act of mental accounting to minimize the regret if a single cause fails
• There are, of course, limits to this principle of benign tolerance.
• If someone has a project to help stray puppies get warm homes, then it's probably best to regard them as trying to exploit bugs in human psychology for their personal gain, rather than a worthy sub-task of the great common Neo-Enlightenment project of human progress.
• e may need to learn to say, by habit and in nearly all forums, "Here is a cool rationalist project", not, "Mine alone is the highest-return in expected utilons per marginal dollar project."
• if we could all identify as elements of the Common Project of human progress, the Neo-Enlightenment, there would be a substantially higher probability of finding ten of us in any given city.

7 more annotations...

• But by and large, the answer to the question "How do large institutions survive?" is "They don't!"  The vast majority of large modern-day institutions - some of them extremely vital to the functioning of our complex civilization - simply fail to exist in the first place.
• Science is not a good emotional fit for people spending their own money.
• In fact very few things are, with the individuals we have now.  It seems to me that this is key to understanding how the world works the way it does - why so many individual interests are poorly protected - why 200 million adult Americans have such tremendous trouble supervising the 535 members of Congress, for example.
• You rarely see a scientific project making a direct bid for some portion of society's resource flow; rather, it first gets allocated to Science, and then scientists fight over who actually gets it.
• But it's still an interesting point that Science manages to survive not because it is in our collective individual interest to see Science get done, but rather, because Science has fastened itself as a parasite onto the few forms of large organization that can exist in our world.
• It seems to me that modern humanity manages to put forth very little in the way of coordinated effort to serve collective individual interests.  It's just too non-ancestral a problem when you scale to more than 50 people.  There are only big taxers, big traders, supermemes, occasional individuals of great power; and a few other organizations, like Science, that can fasten parasitically onto them.

4 more annotations...

• This resource balance principle implies that - over a very wide range of approximately rational systems, including even the interior of a self-modifying mind - there will exist some common currency of expected utilons, by which everything worth doing can be measured.
• Every time I spend money I feel like I'm losing hit points.  That's the problem with having a unified quantity describing your net worth:  Seeing that number go down is not a pleasant feeling
• There is this very, very old puzzle/observation in economics about the lawyer who spends an hour volunteering at the soup kitchen, instead of working an extra hour and donating the money to hire someone to work for five hours at the soup kitchen.
• There's this thing called "Ricardo's Law of Comparative Advantage".  There's this idea called "professional specialization".  There's this notion of "economies of scale".  There's this concept of "gains from trade".  The whole reason why we have money is to realize the tremendous gains possible from each of us doing what we do best.
• This is what grownups do.  This is what you do when you want something to actually get done.  You use money to employ full-time specialists.
• But "volunteering" just one hour of legal work, constantly delayed, spread across three weeks in casual minutes between other jobs?  This is not the way something gets done when anyone actually cares about it, or to state it near-equivalently, when money is involved.
• personally involved.  This is a mere reflection of a mind that doesn't understand the post-hunter-gatherer concept of a market economy.  The act of donating money is not the momentary act of writing the check, it is the act of every hour you spent to earn the money to write that check - just as though you worked at the charity itself in your professional capacity, at maximum, grownup efficiency.
• If the lawyer needs to work an hour at the soup kitchen to keep himself motivated and remind himself why he's doing what he's doing, that's fine.  But he should also be donating some of the hours he worked at the office, because that is the power of professional specialization and it is how grownups really get things done.  One might consider the check as buying the right to volunteer at the soup kitchen, or validating the time spent at the soup kitchen.  I may post more about this later.
• Yes, frugality is a virtue.  Yes, spending money hurts.  But in the end, if you are never willing to spend any units of caring, it means you don't care.

7 more annotations...