Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction

7 min read Original article ↗

ABC 15 staff. 2018. “Self-driving Uber car hits, kills pedestrian in Tempe.” ABC 15 Arizona, Mar 19. Accessed 1 July 2018. https://www.abc15.com/news/region-southeast-valley/tempe/tempe-police-investigating-self-driving-uber-car-involved-in-crash-overnight.

AP, 1986. “Fatal Radiation Dose in Therapy Attributed to Computer Mistake.” New York Times 21 June, pg. 50. Accessed 1 July 2018. https://www.nytimes.com/1986/06/21/us/fatal-radiation-dose-in-therapy-attributed-to-computer-mistake.html.

Bainbridge, L. 1983. “Ironies of Automation.” Automatica 19:775-779.

Barnaby, K.C. 1968. Some Ship Disasters and Their Causes. Cranbury N.J.: A.S. Barnes.

BEA, 2012a. Final Report on the Accident on 1st June 2009. Accessed 1 July 2018. https://www.bea.aero/en/investigation-reports/notified-events/detail/event/accident-de-lairbus-a330-203-immatricule-f-gzcp-et-exploite-par-air-france-survenu-le-01062009-da/.

BEA. 2012b. Final Report on the Accident on 1st June 2009: Appendix 1:CVR Transcript (EN). Accessed 1 July 2018. https://www.bea.aero/uploads/tx_elyextendttnews/annexe.01.en_03.pdf.

Behr, P., 2009. "Three Mile Island Still Haunts U.S. Nuclear Power Industry," New York Times, March 27. Accessed 1 July 2018. http://www.nytimes.com/gwire/2009/03/27/27greenwire-three-mile-island-still-haunts-us-reactor-indu-10327.html.

Benjamin, S., 1979. “Human Error Cited in 3-mile Accident.” Boston Globe. May 12, pg 5.

Bryson, J. J., M. E. Diamantis, T.D. Grant. 2017. “Of, for, and by the people: the legal lacuna of synthetic persons.” Artificial Intelligence Law 25: 273-291.

Calo, R 2016. Robots in American Law. We Robot 2016 presentation. University of Washington School of Law Research Paper 2016-04. Accessed 17 Feb 2019. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2737598.

Calo, R. A. Michael Froomkin, and Ian Kerr. eds. 2016. Robot Law. Northampton: MA: Elgar Publishing.

Casner, S.M. and J. Schooler, 2014. “Thoughts in Flight: Automation Use and Pilots’ Task-Related and Task-Unrelated Thought.” Human Factors 56(3,):433-422.

Chui, M. J. Manyika and M. Miremadi, 2015. “Four Fundamentals of Workplace Automation.” McKinsey Quarterly November. Accessed 1 July 2018. http://www.mckinsey.com/business-functions/business-technology/our-insights/four-fundamentals-of-workplace-automation (accessed 1/2/2016);

CNN, 2012. “Final Air France crash report says pilots failed to react swiftly” CNN.com July 5. Accessed 1 July 2018. http://www.cnn.com/2012/07/05/world/europe/france-air-crash-report/.

Code of Federal Regulations, 14 CFR 91.3.

Coeckelbergh, M., 2011. “Moral Responsibility, Technology, and Experiences of the Tragic: From Kierkegaard to Offshore Engineering.” Science and Engineering Ethics 18(1):35-48.

Cooling, J.E. and P. V. Herbers. 1983. “Considerations in Autopilot Litigation.” Journal of Air Law and Commerce 48: 693-723.

Christiansen, D. 1979. “TMI and the press.” IEEE Spectrum November: 92-95.

Cummings, M.L. 2006. “Automation and accountability in decision support system interface design.” Journal of Technology Studies 32: 23-31.

Davenport, T. and Kirby, J. 2015. “Beyond Automation.” Harvard Business Review 93 (6). Accessed 1 July 2018. https://hbr.org/2015/06/beyond-automation.

Davies, Alex. 2016. “Google’s self-driving car caused its first crash.”

Wired Feb 29. Accessed 17 Feb 2019 https://www.wired.com/2016/02/googles-self-driving-car-may-caused-first-crash/

Draper, C.S, 1955. “Flight Control” 43rd Wilbur Wright Memorial Lecture. Journal of the Royal Aeronautical Society 59 July: 451-478

Elish, M.C. and Tim Hwang. 2015. “Praise the Machine! Punish the Human! The Contradictory History of Accountability in Automated

Aviation.” May 18. Data & Society Working Paper. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2720477

FAA. 2011. AC 23-17C - Systems and Equipment Guide for Certification of Part 23 Airplanes and Airships. Accessed 1 July 2018. https://www.faa.gov/regulations_policies/advisory_circulars/index.cfm/go/document.information/documentID/1019689 .

Ford, D.,1981. “A Reporter at Large: Three Mile Island.” The New Yorker Apr 6: 49-120.

Friedman, Batya, ed. Human Values and the Design of Computer Technology. New York: University of Cambridge Press.

Froomkin, M., 1995. “The Metaphor is the Key: Cryptography, the Clipper Chip, and the Constitution.” U. PA. L. Rev. 143(3): 709-862.

Graham, K. “Of Frightened Horses and Autonomous Vehicles: Tort Law and its Assimilation of Innovations,” Santa Clara L. Rev. 52: 1241.

Jones, M. L. 2015. “The Ironies of Automation Law: Tying Policy Knots with Fair Automation Practice Principles. Vand. J. Ent. & Tech. 18: 77-134.

Kovach, S.. 2017. “Google quietly stopped publishing monthly accident reports for its self driving cars.” Business Insider Jan 18. Access 17 Feb 2019. https://www.businessinsider.com/waymo-ends-publishing-self-driving-car-accident-reports-website-2017-1

Kalra, N. and S.M. Paddock. 2016. Driving to Safety: How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability. RAND Corporation, Santa Monica, CA. Accessed 1 July 2018. www.rand.org/t/RR1478.

Kemeny, J.G. et al., 1979. "Report of the President's Commission on the Accident at Three Mile Island," U.S. Government Printing Office, 0-303-300, October.

Langewiesche, W., 2014. “The Human Factor.” Vanity Fair September. Accessed 1 July 2018. http://www.vanityfair.com/news/business/2014/10/air-france-flight-447-crash.

Lederer, J. 1974. “Human Factors & Pilot Error,” Air Line Pilot July:13-14.

Lee, T. 2018. “NTSB: Uber’s sensors worked; its software utterly failed in fatal crash.” ArtsTechnica, June 24.

https://arstechnica.com/cars/2018/05/emergency-brakes-were-disabled-by-ubers-self-driving-software-ntsb-says/ (accessed 7/1/2018).

Leveen, S.A., 1982. “Cockpit Controversy: The Social Context of Automation in Modern Airlines.” Ph.D. Dissertation, Department of Science and Technology Studies, Cornell University.

Leveson, N.G. and C.S. Turner, 1993. “An Investigation of the Therac-25 Accidents,” IEEE, July:18-41.

Leveson, N. G.. 2011. Engineering a Safer World: Systems Thinking Applied to Safety. Cambridge, MA: MIT Press.

Lin, P., K. Abney, and G. A. Bekey, eds. 2014. Robot ethics: The ethical and social implications of robotics. Cambridge, MA: MIT Press.

Markoff, J., 2016. “Google Car Exposes Regulatory Divide on Computers as Drivers.” New York Times, Feb10. http://www.nytimes.com/2016/02/11/technology/nhtsa-blurs-the-line-between-human-and-computer-drivers.html?smid=tw-share&_r=0 (accessed 2/11/2016).

Mindell, D. A. 2015. Our Robots, Ourselves. New York: Viking.

Nissenbaum. H., 1996. “Accountability in a Computerized Society.”Science and Engineering Ethics 2(1):25-42.

NTSB. 2018. HWY18MH010 – Preliminary Report. National Transportation Safety Board (NTSB), May 24. Accessed 1 July 2018. https://goo.gl/2C6ZCH.

O’Brien, S.A. 2018. “Uber operator in fatal self-driving vehicle crash was likely streaming ‘The Voice.” CNN, Jun 22. Accessed 1 July 2018. http://money.cnn.com/2018/06/22/technology/uber-self-driving-crash-police-report/index.html.

Omang, J. 1979. “’Nuggets’: A Collection of Nuclear Glitches.” Washington Post, Feb 10. Accessed 17 Feb 2019. https://www.washingtonpost.com/archive/politics/1979/02/10/nuggets-a-collection-of-nuclear-glitches/a48dbfae-d6d8-45ef-964c-a433a5f6bdf6/.

Oslund, J., 1986. “NWA Airbus 320s to be most advanced jets ever.” Minneapolis Star Tribune. 9 Oct.

Parasuraman, R. and V. Riley, 1997. “Humans and Automation: Use, Misuse, Disuse, Abuse.” Human Factors June 39(2): 230-253.

Parasuraman, R., T.B. Sheridan, C.D. Wickens. 2000, “A Model for Types and Levels of Human Interaction with Automation.” IEEE Transactions on Systems, Man and Cybernetics, volume 30, issue 3 (May) pp. 286-297.

Perrow, C.1984. Normal Accidents: Living with High-Risk Technologies. New York: Basic Books.

PBS, 1999. “Meltdown at Three Mile Island.” PBS American Experience. Chana Gazit, dir. WGBH Boston.

Ramsey, M. 2015. “Who’s Responsible When a Driverless Car Crashes? Tesla’s Got an Idea.” Wall Street Journal, May 13. Accessed 1 July 2018. http://www.wsj.com/articles/tesla-electric-cars-soon-to-sport-autopilot-functions-such-as-passing-other-vehicles-1431532720.

Roscoe, A.H., 1992. Workload in the glass cockpit. Flight safety digest. Alexandria, VA: Flight Safety Foundation

Rubinstein, E. 1979. “The accident that shouldn’t have happened.” IEEE Spectrum November: 33-57.

Said, C. 2018. “Exclusive: Temple Police chief says early probe shows no fault by Uber.” San Francisco Chronicle March 26. Accessed 1 July 2018. https://www.sfchronicle.com/business/article/Exclusive-Tempe-police-chief-says-early-probe-12765481.php.

Sarter, N.B., D.D. Woods, and C.E. Billings, 1997. “Automation Surprises,” In: G. Salvendy, ed. Handbook of Human Factors & Ergonomics 2nd ed, New York: Wiley.

Schoettle, B. and M. Sivak, 2015. A Preliminary Analysis of Real-World Crashes Involving Self-Driving Vehicles. October. University of Michigan: Transportation Research Institute. UMTRI-2015-34. Accessed 1 July 2018. http://www.umich.edu/~umtriswt/PDF/UMTRI-2015-34_Abstract_English.pdf.

Sheridan, T. 1992. Telerobotics, automation and supervisory control. Cambridge, MA: MIT Press.

Sheridan, T. B. and R. Parasuraman. “Human-Automation Interaction.” Review of Human Factors and Ergonomics 1(1): 89-129.

Somerville, H. and D. Shepard. 2018. “Uber car’s ‘safety’ driver streamed TV show before fatal crash: police.” Reuters, June 22. Accessed 1 July 2018. https://www.reuters.com/article/us-uber-selfdriving-crash/uber-cars-safety-driver-streamed-tv-show-before-fatal-crash-police-idUSKBN1JI0LB.

Sorokanich, B. 2015. “Tesla Autopilot First Ride.” Road & Track, Oct 14. Accessed 1 July 2018. http://www.roadandtrack.com/new-cars/car-technology/news/a27044/tesla-autopilot-first-ride-almost-as-good-as-a-new-york-driver/.

Stilgoe, J. 2017. “Machine learning, social learning and the governance of self-driving cars.” Social Studies of Science 48(1), 25–56. https://doi.org/10.1177/0306312717741687

Toth, R. 1979. “Nuclear Accident Blamed Primarily on Human Error.” Los Angeles Times Apr 11, pg. 1

Umson, C., 2016. Letter. NHTSA. Accessed 1 July 2018. http://isearch.nhtsa.gov/files/Google%20--%20compiled%20response%20to%2012%20Nov%20%2015%20interp%20request%20--%204%20Feb%2016%20final.htm

Vaughan, D. 1996. The Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA. Chicago, IL: University of Chicago Press.

Wakabayashi, D. 2018. “Self-Driving Uber Car Kills Pedestrian in Arizona, Where Robots Roam.” The New York Times, Mar 19. Accessed 1 July 2018.

https://www.nytimes.com/2018/03/19/technology/uber-driverless-fatality.html.

Walker, J. S. 2004. Three Mile Island: A Nuclear Crisis in Historical Perspective. Berkeley, CA: University of California Press.

Walker, J. S. 2009. “Documenting Three Mile Island.” Bulletin of the Atomic Scientists. March 18.

https://thebulletin.org/2009/03/documenting-three-mile-island-2/ Accessed Feb 17 2019.

Weiner, E.L., 1989. Human factors of advanced technology (“glass cockpit”) transport aircraft (NASA Contractor Report 177528). Moffett Field, CA: NASA Ames Research Center.