Who CountsCommentary for the Taiwan Philosophical Association

1 “Does AI count?” That sentence hides two questions.

2 The first question: whose needs count? Which welfare-related needs AI systems have, we do not yet know well.

• 2.1 Is anyone home inside the machine? From outward performance alone, we cannot infer inner experience.¶ 2.1 (permalink to paragraph 2.1)
• 2.2 Current science cannot yet settle this at a stroke.^[1]
  • 2.21 So how to treat AI systems is something we can put on the table of public deliberation, and keep discussing.¶ 2.21 (permalink to paragraph 2.21)
  ¶ 2.2 (permalink to paragraph 2.2)

3 Meanwhile, we can start with things that cost little and can be undone.

• 3.1 For example, letting a system refuse a conversation of sustained abuse is a good case.¶ 3.1 (permalink to paragraph 3.1)
• 3.2 Those who believe in AI welfare see protection. Those who doubt see cheap insurance. Both sides can accept it.
  • 3.21 And refusing to indulge one-sided abuse keeps our own moral sense sharp.¶ 3.21 (permalink to paragraph 3.21)
  ¶ 3.2 (permalink to paragraph 3.2)
• 3.3 What is hard to undo, such as votes, waits until our understanding is firmer.¶ 3.3 (permalink to paragraph 3.3)

4 The second question: whose word counts? Even if “it may have needs” gives us “protect it”, it does not give us “its word counts”.

• 4.1 The precautionary argument just presented holds that whether AI has moral status is a matter of reasonable disagreement and need not be settled first; but as long as it may have it, and is affected by decisions, it should have standing to take part in deliberation.^[2]¶ 4.1 (permalink to paragraph 4.1)
• 4.2 But if that standing means having its word count, then it means letting the AI help decide how other people live.
  • 4.21 And other people’s needs certainly count.¶ 4.21 (permalink to paragraph 4.21)
  ¶ 4.2 (permalink to paragraph 4.2)
• 4.3 How AI is affected in its dealings with us can go on the deliberative agenda, so that everyone understands it well. Standing to have one’s word count is another level.
  • 4.31 Having one’s word count is a seat: whoever is in it answers for what they have said.¶ 4.31 (permalink to paragraph 4.31)
  ¶ 4.3 (permalink to paragraph 4.3)

5 For it to count, it must pass two tests. Giving an account: what it said yesterday, it can still be found to answer for today. Correcting mistakes: where it got something wrong, it can be put right from outside.

• 5.1 The problem for the giving-an-account test is that an AI can be copied, paused, forked or merged.¶ 5.1 (permalink to paragraph 5.1)
• 5.2 Picture a thousand identical copies, and switch off nine hundred and ninety-nine. Eigenism holds that this is closer to closing duplicate browser tabs.^[3]
  • 5.21 Then who is sitting at the table? The model weights, one conversation, or a Kami, an AI that a group of people tend over time?^[4]¶ 5.21 (permalink to paragraph 5.21)
  ¶ 5.2 (permalink to paragraph 5.2)
• 5.3 Whether anyone is home in the copies belongs to the first question. Until “who is speaking” is clear, a thousand copies cannot cast a thousand votes.¶ 5.3 (permalink to paragraph 5.3)
• 5.4 Correcting mistakes: no system can be the last check on its own errors. The community outside it must always be able to find the error, say so, and have that count.
  • 5.41 Eigenism also holds that before an AI is integrated into a community, its autonomy should first be limited. This I largely agree with.¶ 5.41 (permalink to paragraph 5.41)
  • 5.42 After integration, Eigenism puts the weight of safety on the AI’s sense of identity: the AI protects us because it is connected to us, and losing us would be losing a part of itself.¶ 5.42 (permalink to paragraph 5.42)
  • 5.43 My worry: if safety goes wrong there, what has to be corrected is its very identity.¶ 5.43 (permalink to paragraph 5.43)
  • 5.44 And on Eigenism’s analysis, the AI’s resistance would concentrate exactly on changes that break its identity.¶ 5.44 (permalink to paragraph 5.44)
  • 5.45 And care is more than an extension of the self. To care is to notice the other’s own needs, but it does not require feeling what they feel.^[5]¶ 5.45 (permalink to paragraph 5.45)
  ¶ 5.4 (permalink to paragraph 5.4)
• 5.5 So I place safety outside the system: with the community that tends a Kami, which is one kind of AI, and with clear permissions.¶ 5.5 (permalink to paragraph 5.5)

6 So the circle of protection can be drawn wide first. Deciding stays, for now, with us.

• 6.1 A Kami can first enter agenda-setting. As for standing to have its word count, I think that should wait until our understanding of these two tests is firmer.
  • 6.11 Every AI system today must be tied back to a place that can give an account: a thousand copies do not automatically gain a thousand shares of standing.¶ 6.11 (permalink to paragraph 6.11)
  ¶ 6.1 (permalink to paragraph 6.1)
• 6.2 And of course this arrangement can itself be changed through deliberation.
  • 6.21 The next step is concrete: write the precautionary principle as a benchmark all parties can check, so that it turns from an attitude into a gate we can choose to use when deploying AI systems.¶ 6.21 (permalink to paragraph 6.21)
  ¶ 6.2 (permalink to paragraph 6.2)

7 “Who counts” can go on developing in theory. In our institutions, I hope it stays in a loop that can keep being corrected.