This is a link to a question asked on the EA Forum by Aryeh Englander. (Please post responses / discussion there.)

Does the following seem like a reasonable brief summary of the key disagreements regarding AI risk?

Among those experts (AI researchers, economists, careful knowledgeable thinkers in general) who appear to be familiar with the arguments:

• Seems to be broad (but not universal?) agreement that:
• Superintelligent AI (in some form, perhaps distributed rather than single-agent) is possible and will probably be created one day
• By default there is at least a decent chance that the AI will not b

Reply to: Meta-Honesty: Firming Up Honesty Around Its Edge-Cases

Eliezer Yudkowsky, listing advantages of a "wizard's oath" ethical code of "Don't say things that are literally false", writes—

Repeatedly asking yourself of every sentence you say aloud to another person, "Is this statement actually and literally true?", helps you build a skill for navigating out of your internal smog of not-quite-truths.

I mean, that's one hypothesis about the psychological effects of adopting the wizard's code.

A potential problem with this is that human natural language contains a lot of ambiguity. Words can

(Cross-posted from Facebook.)

0: Tl;dr.

• A problem with the obvious-seeming "wizard's code of honesty" aka "never say things that are false" is that it draws on high verbal intelligence and unusually permissive social embeddings. I.e., you can't always say "Fine" to "How are you?" This has always made me feel very uncomfortable about the privilege implicit in recommending that anyone else be more honest.
• Genuinely consistent Glomarization (i.e., consistently saying "I cannot confirm or deny" whether or not there's anything to concea

CFAR recently launched its 2019 fundraiser, and to coincide with that, we wanted to give folks a chance to ask us about our mission, plans, and strategy. Ask any questions you like; we’ll respond to as many as we can from 10am PST on 12/20 until 10am PST the following day (12/21).

Topics that may be interesting include (but are not limited to):

• Why we think there should be a CFAR;
• Whether we should change our name to be less general;
• How running mainline CFAR workshops does/doesn't relate to running "AI Risk for Computer Scientist" type workshops. Why we both do a lot of rec

This is a cross post from http://250bpm.com/blog:128.

Introduction

In the past I've reviewed Eliezer Yudkowsky's "Inadequate Equilibria" book. My main complaint was that while it explains the problem of suboptimal Nash equilibria very well, it doesn't propose any solutions. Instead, it says that we should be aware of such coordination failures and we should expect ourselves to fare better than the official institutions in such cases. What Yudkowsky is saying (if I understand him correctly) is that given that the treatment of short bowel syndrome in babies is stuck in an inadequate eq... (Read more)

The recent adversarial collaboration on spiritual experiences on Slate Star Codex includes this paragraph:

It was also discovered that people in the United States, Australia, the United Kingdom, and Scandinavia do not tend to share their spiritual experiences with others. Hood et al. wonder if this is why such spiritual experiences are thought to be uncommon (as fewer people in these societies might have heard reports of others’ spiritual experiences).

This naturally lead me to wonder, what spiritual experiences have LessWrong readers have that they are willing to share, since the readers... (Read more)

An antimeme is a meme with the following three characteristics:

• Learning it threatens the egos and identities of adherants to the mainstream of a culture^[1].
• Learning the meme renders mainstream knowledge in the field unimportant by broadening the problem space of a knowledge domain, usually by increasing the dimensionality.
• Mainstream wisdom considers detailed knowledge of the antimeme irrelevant, unimportant or low priority. Mainstream culture may just ignore the antimeme altogether instead.

I call these "antimemes" because they exhibit behavior opposite that of regular memes. The typical

To celebrate all the possibilities of humanity during these holidays, have a possible calendar of the year 12020 of the human era (link to full calendar here).

Minor fact: in the Gregorian calendar, the days of the week cycle exactly every 400 years, so the non-time-travellers among you can use this for 2020 as well...

(previous holiday specials can be found here and here)

In the past 3-4 years, I went through a prolonged and painful life crisis in which I systematically deconstructed my existing worldview and slowly moved away from Evangelical Christianity into something Rationalist or Rationalist-adjacent. In the past 4 months, I've started hanging around the Berkeley Rationality community and am now dating someone embedded therein. At this point my partner is still my main connection to the specific values and practices of the community, and given that my worldview is currently being fleshed-out, she has an outsized influence on what my future beliefs and val

In my short-form, I write:

[...] This is way more obvious and way more clear in Inadequate Equilibria. Take a problem, a question and deconstruct it completely. It was concise and to the point, I think it's one of the best things Eliezer has written; I cannot recommend it enough.

Just finished Inadequate Equilibria. Now, I'm reading:

• The Big Picture from Sean Carroll (which seems a really, really good companion to The Sequences.) I'm at chapter 17/50, and I'm really enjoying it so far; it's an ambitious book though!
• In fiction I picked up UNSONG from Scott Alexander; I a

LessWrong is currently reviewing the posts from 2018, and I'm trying to figure out how voting should happen. The new hotness that all your friends are talking about is quadratic voting, and after thinking about it for a few hours, it seems like a pretty good solution to me.

I'm writing this post primarily for people who know more about this stuff to show me where the plan will fail terribly for LW, to suggest UI improvements, or to suggest an alternative plan. If nothing serious is raised that changes my mind in the next 7 days, we'll build a straightforward UI and do it.

I'

Most models of agency (in game theory, decision theory, etc) implicitly assume that the agent is separate from the environment - there is a “Cartesian boundary” between agent and environment. The embedded agency sequence goes through a long list of theoretical/conceptual problems which arise when an agent is instead embedded in its environment. Some examples:

• No defined/input output channels over which to optimize
• Agent might accidentally self-modify, e.g. drop a rock on its head
• Agent might intentionally self-modify, e.g. change its own source code
• Hard to define hypotheticals which

An introduction to a recent paper by myself and Ryan Carey. Cross-posting from Medium.

For some intellectual tasks, it’s easy to define success but hard to evaluate decisions as they’re happening. For example, we can easily tell which Go player has won, but it can be hard to know the quality of a move until the game is almost over. AI works well for these kinds of tasks, because we can simply define success and get an AI system to pursue it as best it can.

For other tasks, it’s hard to define success, but relatively easy to judge solutions when we see them, for example, doing a backflip. Getti

Forgive me if some of this is repetitive, I can’t remember what I’ve written in which draft and what’s actually been published, much less tell what’s actually novel. Eventually there will be a polished master post describing my overall note taking method and leaving out most of how it was developed, but it also feels useful to discuss the journey.

When I started taking notes in Roam (a workflowy/wiki hybrid), I would:

1. Create a page for the book (called a Source page), with some information like author and subject (example)
2. Record every claim the book made on that Source page
3. Tag each claim so

Quick context: Epistemic spot checks started as a process in which I did quick investigations a few of a book’s early claims to see if it was trustworthy before continuing to read it, in order to avoid wasting time on books that would teach me wrong things. Epistemic spot checks worked well enough for catching obvious flaws (*cou*Carol Dweck*ugh*), but have a number of problems. They emphasize a trust/don’t trust binary over model building, and provability over importance. They don’t handle “severely flawed but deeply insightful” well at all. So I started trying to create something better. 

Be... (Read more)

Continuation of:  No Individual Particles
Followup to:  The Generalized Anti-Zombie Principle

Suppose I take two atoms of helium-4 in a balloon, and swap their locations via teleportation.  I don't move them through the intervening space; I just click my fingers and cause them to swap places.  Afterward, the balloon looks just the same, but two of the helium atoms have exchanged positions.

Now, did that scenario seem to make sense?  Can you imagine it happening?

If you looked at that and said, "The operation of swapping two helium-4 atoms produces an identical configuratio... (Read more)