A clinic posts hours, yet patients show up to a locked door. A responsive clinic apologises, posts why it happened, updates hours, and texts people next time. The fix becomes part of the system.
Competent action creates new information. The person who was served — or who was failed — now knows something about the system that the system does not yet know about itself. Refusing to receive that knowledge is the fastest path to repeated failure.
In Moral Boundaries, Joan Tronto defines responsiveness as "observing that response and making judgements about it — whether the care given was sufficient, successful, or complete." She adds a point that is easy to overlook: the person cared for need not be the one who responds — others in the care setting can assess. Those best positioned to judge whether care worked are often not the engineers who built it but the people who live closest to its effects. Aimee van Wynsberghe read the same element into design a decade ago: responsiveness is what "encourages an active stance of the care-receiver rather than a more passive, vulnerable one" (2013, p. 418). For Civic AI, the public question is simple: when a system harms someone, who can contest the result, how fast, and what changes because they spoke?
Definition
- People closest to harm define harm. They hold the author's pen for evaluations and have a real say in whether repair counted.
- A right to reply must change something. A reply that cannot trigger correction, rollback, or compensation is theatre. (The reply is what Albert Hirschman called voice; Pack 5's portability is his exit, and each disciplines an institution only when the other is real.)
- Shared memory matters. Post-incident learnings become tests; tests prevent repeats.
- Time is part of the service. A fast wrong-then-right beats a slow maybe. Reversible defaults matter.
- Care labour is labour. Eval writing, appeals, moderation, and facilitation should be compensated and visible. Participatory-AI research has already named the trap — participation is not a design fix unless the work is paid and the participants hold the pen (Birhane et al., 2022).
Why it matters
Pack 3 checks whether the process ran as promised. Pack 4 checks whether the care actually landed. That is why responsiveness begins with community judgement, not with an engineering pipeline.
Community-authored evaluations, appeals, and repair logs are the public commitment. Some teams later feed those signals into routing or model updates — sometimes called Reinforcement Learning from Community Feedback (RLCF) — but that implementation choice is secondary. The community authors the yardstick first.
Responsiveness is not an add-on. It is the phase that keeps the other three from becoming control. Care without the care-receiver's own assessment is a projection of the carer's preferences onto a passive recipient. Institutional care has caused enormous harm in precisely this way — care that was actually control. The residential schools that separated Indigenous children from their families were, by many official accounts, expressions of care. The psychiatric institutions that confined people judged deviant often understood that confinement as therapeutic. A system that is attentive, responsible, and competent, but has no mechanism for the people it serves to say "this is not working for me," can drift, undetected, from care into domination. The person receiving care must be able to say that sentence — and the sentence must change the system.
The brake as a care mechanism
Feedback pipelines can change future training runs, but they cannot interrupt a wrong decision in the moment when interruption is what care requires. That takes a brake: the mechanism by which the care-receiver — or the care-giver on their behalf — can say "this time, the automated model is wrong; stop, let me override, or route this to a human now." For a nurse, that means overriding a non-alert and calling the doctor without justifying it to a software interface; for a resident, a two-minute action that triggers human review of their case; for a language community, a way to flag a pattern and force a policy review, not merely lodge individual complaints. What makes the brake functional is not sophistication but accessibility and consequentiality: reachable without a maze, and it must actually stop something.
An honest design cannot hide the tension: a brake that is easy to reach is also easy to lean on, and a ward where every alert is overridden has not solved alarm fatigue — it has automated it. The answer is not a harder brake but a legible one: log every pull, make it attributable — this running record is the override ledger — and surface over-override patterns back into community-authored evaluations as a signal of their own.
What it looks like in practice
- Community-authored evaluations. Affected communities co-write the tests for harm and successful repair.
- Shared eval registry. Use a public, Wikipedia-like registry for tests (such as Weval): anyone can draft, civil society partners review, and labs adopt or explain.
- Clear appeals. Answer urgent cases in 24 hours, standard in 7 days, and complex in 30 days. These timelines are Engagement Contract commitments with consequences for breach, not aspirations. The appeal is the oldest instrument on this page: Danielle Citron's technological due process (2008) asked for notice, hearing, and reasons for automated public decisions, and the contestable-AI literature has since made contestability a design requirement rather than a courtesy.
- Visible repair. Give each incident a page: what happened, whom it affected, what changed, and which new test now guards against repeat harm.
- Correction backlog transparency. Keep a shared queue of open corrections with case age and owner; escalate when backlog depth or oldest-case age exceeds agreed thresholds so repairs keep pace.
From ideas to practice
- Expose the appeal button. Every decision should show a one-click route to challenge it, with the clock visible.
- Accept harm drafts. Let people describe harms in plain language; turn them into community-reviewed tests with partners.
- Triage by severity. Highest-severity cases trigger immediate pause or reversible defaults.
- Fix or explain. Publish the remedy or the reason with next steps, on the clock.
- Memorialise. Turn incidents into tests and link them from the contract changelog.
- Check back. Close the loop with the people who appealed and measure trust-under-loss — the procedural-justice finding, in Tom Tyler's work, that people who lose on the merits still accept a process they judge fair.
Buildable tools
- Appeal API with timers, statuses, and escalation.
- Community feedback pipeline that can feed approved evals into routing, training, or policy signals when appropriate.
- Eval editor that turns plain-language harms into test harnesses.
- Incident tracker for severity, owners, deadlines, and public notes.
- Repair log template for root cause, remedy, and test added. The tracker moves the work; the repair log makes the learning public.
- Correction backlog board that exposes unresolved repairs by age, queue depth, and owner, with reassignment/escalation prompts as cases age.
One case: the flood-bot
- Appeals surge. A language community flags mistranslations in proof rules.
- Local eval. Community partners submit a translation-fidelity eval; the group is compensated from the project's escrow fund, because community knowledge is labour, not free quality assurance. The bot fails the eval; pause triggers; reversible defaults apply.
- System change. Translation uncertainty now routes cases to bilingual human review rather than auto-denial. The eval enters the registry as a permanent release gate: no future update to the bot's documentation rules ships without passing it.
- Fix. Bilingual reviewers update rules; the new test guards future changes.
- Close the loop. Elena gets a text: "We fixed the error; here is your new decision; here is how to see what changed; and here is how to reach a caseworker directly." Trust-under-loss ticks up.
What could go wrong
- Appeal maze. Too many steps. Fix: Single button; auto-escalation if SLA breach; status visible to the appellant at every stage.
- Eval spam. Low-quality tests flood the system, and AI-generated evals can swamp repair channels at the scale already seen in public-comment dockets, drowning signal from the directly affected. Fix: Partner moderation; reputation for contributors; merge/duplicate tools; provenance and accountable standing (not public identity) on feedback channels; per-community rate limits; audit trails that distinguish one affected person from generated proxies. The goal is quality, not credentialing — moderation must not privilege established organisations over community members with direct experience; priority for those directly affected is preserved.
- Blame storms. People, not processes, get blamed. Fix: Blameless incident reviews; focus on mechanism design.
- Weaponised appeals. Adversaries flood appeals or strategically trigger pauses to disrupt service; synthetic appellants can swamp the same repair channels at AI scale. Fix: Require accountable standing (not public identity) for pause triggers and appeal routes; rate-limit by community; audit trails that separate genuine standing from generated proxies; preserve priority access for those directly affected.
- Over-override. The brake gets pulled against correct alerts; alert fatigue is automated rather than solved. Fix: Keep the brake easy to pull but legible, as the brake section above sets out — log and attribute every pull.
- Backlog drift. A handful of urgent fixes can clear quickly while older, lower-priority corrections quietly stack up. Fix: enforce throughput commitments, explicit backlog caps, and public escalation when the oldest open correction or backlog depth exceeds agreed thresholds.
Interfaces
- From Responsibility (Pack 2): who acts is clear; remedies are wired.
- From Competence (Pack 3): observability and guardrails feed responsiveness; incident loops start here.
- To Attentiveness (Pack 1): new needs discovered through response reshape what we notice — the cycle restarts.
- To Solidarity (Pack 5): public repair culture builds cross-group trust.
- To Symbiosis (Pack 6): responsive systems earn the right to stay local.
A closing image: the workshop wall of retired broken parts
Imagine a workshop with a wall full of retired broken parts, each tagged with the story of how it broke, how to avoid future damage, and who fixed it. The wall is not a wall of shame; it is a wall of learning — public, and kept by the community that uses the workshop, not only by the engineers who built the machines. The shop that hides its breaks will repeat them. The shop that builds the wall becomes, over time, the most reliable shop in the district — not because it never breaks anything, but because it has made its failures the foundation of its competence.
Sources for this pack
- Tronto, Moral Boundaries (1993), pp. 106–108 and 127–134 — care-receiving and responsiveness; van Wynsberghe, "Designing Robots for Care" (2013), p. 418 — responsiveness as the care-receiver's active stance.
- Hirschman, Exit, Voice, and Loyalty (1970) — the appeal as voice.
- Citron, "Technological Due Process" (2008); Alfrink et al., "Contestable AI by Design" (2023); Lyons, Velloso, and Miller, "Conceptualising Contestability" (2021) — the appeal as a design requirement.
- Sloane et al., "Participation Is Not a Design Fix for Machine Learning" (2022); Birhane et al., "Power to the People?" (2022) — community-authored evaluation as paid, participatory work.
- Tyler, Why People Obey the Law (1990) — procedural justice, the finding behind trust-under-loss.
- Bainbridge, "Ironies of Automation" (1983); Sendelbach and Funk, "Alarm Fatigue" (2013) — why a brake needs a ledger.
- Douw et al., "Nurses' Worry or Concern" (2015) — nurses' concern can precede a change in vital signs.
- The full trail is on the Sources page.
