R A I L

The Journal of Robotics,

Artificial Intelligence & Law

COURT

PRESS

FULL

R A I L

Volume 2, No. 6 | November–December 2019

Editor’s Note: Intellectual Property

Steven A. Meyerowitz

Adapt Your IP Strategy for Arcial Intelligence

Kevin M. Pasquinelli

Biometric Data: Companies Should Act to Migate Risks in the Face of Growing

Regulaons and Increased Risk for Liability

Robert A. Wells, Veronica D. Jackson, and Christopher J. Tully

What “Shall” and “Will” Teach Us About Contract Draing (and Some Thoughts on AI)

Ryan Tanny Kang

Building Trust with a Workforce as It Automates

Mathew Donald

UK Government’s Guide to Using AI in the Public Sector

Lisa Peets, Marn Hansen, Sam Jungyun Choi, and Chance Levian

Everything Is Not Terminator: Is China’s Social Credit System the Future?

John Frank Weaver

RAIL

The Journal of Robotics,

Artificial Intelligence & Law

Volume 2, No. 6 | November–December 2019

385 Editor’s Note: Intellectual Property

Steven A. Meyerowitz

389 AdaptYourIPStrategyforArcialIntelligence

Kevin M. Pasquinelli

415 BiometricData:CompaniesShouldActtoMigateRisksinthe

FaceofGrowingRegulaonsandIncreasedRiskforLiability

Robert A. Wells, Veronica D. Jackson, and Christopher J. Tully

421 What“Shall”and“Will”TeachUsAboutContractDraing(and

Some Thoughts on AI)

Ryan Tanny Kang

433 BuildingTrustwithaWorkforceasItAutomates

Mathew Donald

439 UKGovernment’sGuidetoUsingAIinthePublicSector

  LisaPeets,MarnHansen,SamJungyunChoi,andChanceLevian

445 Everything Is Not Terminator: Is China’s Social Credit System the

Future?

John Frank Weaver

EDITOR-IN-CHIEF

Steven A. Meyerowitz

President, Meyerowitz Communications Inc.

EDITOR

Victoria Prussen Spears

Senior Vice President, Meyerowitz Communications Inc.

BOARD OF EDITORS

Miranda Cole

Partner, Covington & Burling LLP

Kathryn DeBord

Partner & Chief Innovation Ocer, Bryan Cave LLP

Melody Drummond Hansen

Partner, O’Melveny & Myers LLP

Paul B. Keller

Partner, Norton Rose Fulbright US LLP

Garry G. Mathiason

Shareholder, Littler Mendelson P.C.

Elaine D. Solomon

Partner, Blank Rome LLP

Linda J. ayer

Partner, Finnegan, Henderson, Farabow, Garrett & Dunner LLP

Mercedes K. Tunstall

Partner,Pillsbury Winthrop Shaw Pittman LLP

Edward J. Walters

Chief Executive Ocer, Fastcase Inc.

John Frank Weaver

Attorney, McLane Middleton, Professional Association

THE JOURNAL OF ROBOTICS, ARTIFICIAL INTELLIGENCE & LAW (ISSN

2575-5633 (print) /ISSN 2575-5617 (online) at $495.00 annually is published

six times per year by Full Court Press, a Fastcase, Inc., imprint. Copyright

2019 Fastcase, Inc. No part of this journal may be reproduced in any form—by

microlm, xerography, or otherwise—or incorporated into any information

retrieval system without the written permission of the copyright owner. For

customer support, please contact Fastcase, Inc., 711 D St. NW, Suite 200,

Washington, D.C. 20004, 202.999.4777 (phone), 202.521.3462 (fax), or email

customer service at sup[email protected].

Publishing Sta

Publisher: Morgan Morrissette Wright

Journal Designer: Sharon D. Ray

Cover Art Design: Juan Bustamante

Cite this publication as:

e Journal of Robotics, Articial Intelligence & Law (Fastcase)

is publication is sold with the understanding that the publisher is not engaged

in rendering legal, accounting, or other professional services. If legal advice or

other expert assistance is required, the services of a competent professional should

be sought.

A Full Court Press, Fastcase, Inc., Publication

Editorial Oce

711 D St. NW, Suite 200, Washington, D.C. 20004

https://www.fastcase.com/

POSTMASTER: Send address changes to THE JOURNAL OF ROBOTICS,

ARTIFICIAL INTELLIGENCE & LAW, 711 D St. NW, Suite 200, Washington,

D.C. 20004.

Articles and Submissions

Direct editorial inquires and send material for publication to:

Steven A. Meyerowitz, Editor-in-Chief, Meyerowitz Communications Inc.,

26910 Grand Central Parkway, #18R, Floral Park, NY 11005, smeyerowitz@

meyerowitzcommunications.com, 646.539.8300.

Material for publication is welcomed—articles, decisions, or other items of interest

to attorneys and law rms, in-house counsel, corporate compliance ocers,

government agencies and their counsel, senior business executives, scientists,

engineers, and anyone interested in the law governing articial intelligence and

robotics. is publication is designed to be accurate and authoritative, but neither

the publisher nor the authors are rendering legal, accounting, or other professional

services in this publication. If legal or other expert advice is desired, retain the

services of an appropriate professional. e articles and columns reect only the

present considerations and views of the authors and do not necessarily reect

those of the rms or organizations with which they are aliated, any of the former

or present clients of the authors or their rms or organizations, or the editors or

publisher.

QUESTIONS ABOUT THIS PUBLICATION?

For questions about the Editorial Content appearing in these volumes or reprint

permission, please call:

Morgan Morrissette Wright, Publisher, Full Court Press at m[email protected]m

or at 202.999.4878

For questions or Sales and Customer Service:

Customer Service

Available 8am–8pm Eastern Time

866.773.2782 (phone)

sup[email protected] (email)

Sales

202.999.4777 (phone)

[email protected] (email)

ISSN 2575-5633 (print)

ISSN 2575-5617 (online)

Robotics, Articial Intelligence & Law / November–December 2019, Vol. 2, No. 6, pp. 445–451.

ISSN 2575-5633 (print) / ISSN 2575-5617 (online)

Everything Is Not Terminator

Is China’s Social Credit System

the Future?

John Frank Weaver*

In 2014, the central government of the People’s Republic of

China announced a “social credit” system, which will monitor the

behavior of the nation’s population and rank each person based

on his or her social credit, i.e., how well each person behaves as

a citizen. The term social credit dates to 2002, when the central

government was looking at establishing a system to measure the

creditworthiness of its citizens, but ultimately wanted the system

to be compatible with the social services system.

Forty-three

municipalities are currently testing versions of the system, and the

program is scheduled to be operational nationwide by 2020.

track citizens, the municipalities’ systems rely on huge amounts

of personal data from a wide variety of sources, including social

networks, smart phone apps, and video cameras already installed

by the central government. Those cameras are part of “Skynet,” the

Chinese government’s video surveillance system, the public purpose

of which is to track criminal behavior, but which has more than

20million cameras in public spaces across the country.

The end

goal is to rely on sophisticated artificial intelligence (“AI”) to review

all of this data and add and deduct points from citizens based on

how well they engage in lawful behavior.

The Future with AI

Not surprisingly, this has prompted a great deal of criticism.

However, I can’t help but wonder if this is the natural development

of the future with AI.

One theme that I have returned to frequently when writing

and speaking about AI is what laws and regulations should do

in response to fundamental changes in society introduced by

446 e Journal of Robotics, Articial Intelligence & Law [2:445

technological changes.

In particular, the century of legal changes

in the wake of the American Industrial Revolution is a useful

(albeit slow) example of what government can do to spread widely

the benefits of massive technological advancements.

The intro-

duction of the factory system moved unprecedented numbers

of people from rural communities to urban communities and

shifted their work from farming to manufacturing, from work

hours based on the seasons and the sun to work hours based on

the company clock. In response, Congress and state legislatures

enacted legislation that instituted a minimum wage, limited how

many hours a person could work in a week, mandated workplace

and environmental standards, and prohibited children from

working.

Essentially, elected representatives recognized that

the Industrial Revolution had changed America and used laws

and regulations to shape the new country into a version that was

better for their constituents.

If AI is going to introduce as much change as many experts

predict, as much as the Industrial Revolution,

then we should

wonder what that changed world will look like. In the same way

that it would have been hard but not impossible for leaders at the

beginning of the Industrial Revolution to predict how the world

was going to change, it is difficult but not impossible for us to

predict how AI is going to change the world. Systems like China’s

social credit system might be what it looks like.

That should not be too surprising. There are other versions

of it in place already. In the United States, credit-rating agencies

score and rank individuals based on their creditworthiness, a

system that the Chinese central government relied on when it

created its social credit system.

In the United Kingdom, there

are systems in place to score individuals using data from a vari-

ety of sources, like credit score, phone usage, and rent payments;

based on those scores, individuals are ranked by preference for

job applications, access to social services, etc.

Similarly, in

Germany there are systems in place to track data and rank indi-

viduals based on creditworthiness and healthy lifestyle.

China’s

proposal is therefore not unique because of the theory it relies

on, but because of its scope.

That suggests that the world we

live in is already marching toward national governments accept-

ing that AI systems should analyze the vast amounts of personal

data available to them and score citizens based on that analysis.

2019] Is China’s Social Credit System the Future? 447

The Future of AI Does Not Have to be a Bad One

If that is likely the future, does it have to be a bad one? Read

any of the articles in the footnotes of this column and you are likely

to see the terms “dystopian,” “chilling,” and “Black Mirror” tossed

around. That threat is real, and the possibility of a social credit

system arbitrarily hurting already vulnerable populations is strong.

It is all too easy to find stories of people in China who have been

barred from jobs or schools,

prohibited from purchasing certain

train tickets,

and banned from vacationing

due to low social

credit scores with no due process and for seemingly arbitrary or

capricious reasons. However, I do not believe that is the inevitable

result for two reasons.

First, the goal of the Chinese system and copycats in other

nations is to reward good behavior and promote good citizenship,

essentially through gamification.

By assigning points to certain

kinds of behavior, social credit AI systems incentivize that behavior,

possibly making people better citizens, maybe without realizing

they are becoming better citizens.

On its face, that is not neces-

sarily a bad idea. There have been numerous socially engineered

methods of encouraging good behavior—Social Security numbers

that make it harder to escape past bankruptcies and criminal

behavior, civic awards, publication of police logs in newspapers,

etc.—and this is arguably the next evolution of that. If successfully

implemented, the gamification of good citizenship could lead to

improved quality of life and civic engagement. However, the rules

governing the behavior that is rewarded and punished, appeals of

points awarded or deducted, and the transparency of the system

are all vitally important.

Second, it is possible to implement laws and regulations that

properly police a social credit system and provide reassurances to

the public that (a)the right behavior is being rewarded, (b)the AI

is not biased, and (c)there are appropriate safeguards for people

to appeal and confirm their social credit scores. Here are a few

suggestions for government regulations and policies that would

result in a fairer, more beneficial social credit systems:

1. Require Transparency: e source code of the AI devel-

oped to evaluate data and issue social credit scores must

be open to review and sandbox testing by the public.

448 e Journal of Robotics, Articial Intelligence & Law [2:445

All datasets and databases used to train the AI must be

publicly available. All personal data used to determine

actual social credit scores must be anonymized and made

publicly available. Transparency is the rst step toward

reassuring the public that the social credit system is a

public good and is being fairly administered.

2. Establish a Governance Group: Any central government

that decides to develop a social credit system should

also form a governance group that reviews all elements

of that system: how behavior is classied as benecial or

not, how points are awarded, how individuals may appeal

their points, etc. Populated by high-ranking ocials—

cabinet secretaries, key legislative leaders, etc.—and AI

experts, this group should hold public meetings, accept

public comment, and publish reports on its ndings and

recommendations.

3. Require and Promote the Testing and Training of the AI:

e governance group should regularly test the social

credit system’s AI, train it so that the system more ac-

curately rewards the appropriate behavior, and encour-

age third parties to test and train the sandbox version

of the AI. is will both make the AI better and give

public reassurances of the AI strengths and remediated

shortcomings.

4. Require Ex-Post Analysis: Ex-post analysis is a form of

policing AI in which the developers use tools like cryp-

tographic commitments and zero knowledge proofs to

provide auditable evidence that the AI is performing

key analyses correctly. Requiring that social credit AI

uses these benchmarks will give private and government

auditors another method to assess the operation of the

AI.

5. Institute an Appropriate Appeals Process: In addition to the

opaque and seeming arbitrariness of the Chinese social

credit system, its most common complaint appears to

be that there is no easy way to challenge or appeal the

rewarding or deduction of points. Creating an appro-

priate administrative process for the appeal and review

of points will be an important part of creating a social

credit system that fairly promotes better citizenship. e

system must be easy to navigate, administer, and police.

2019] Is China’s Social Credit System the Future? 449

Obviously, the suggestions above are intended as broad ideas

for making a social credit system consistent with the requirements

of due process and the rule of law. But they also demonstrate that

like many other governing inventions, including the development

of regulatory and administrative legal models, the creation of a

social credit system does not automatically mean that due diligence

and responsible government are doomed. If political leaders look

at where technology is trending, the social credit system can help

to form a prosperous AI middle class in the 21st century, like legal

reforms in reaction to the Industrial Revolution created America’s

prosperous industrial middle class in the mid-20th century.

Notes

* John Frank Weaver, an associate at McLane Middleton and a member

of the rm’s privacy and data security practice group, is the “Everything Is

Not Terminator” columnist for e Journal of Robotics, Articial Intelligence&

Law. Mr. Weaver, who may be contacted at john.weaver@mclane.com, has

a diverse technology practice that focuses on information security, data pri-

vacy, and emerging technologies, including articial intelligence, self-driving

vehicles, and drones.

1. Alexandra Ma, “China has started ranking citizens with a creepy

‘social credit’ system—here’s what you can do wrong, and the embarrassing

demeaning ways they can punish you,” Business Insider (October 29, 2018),

https://www.businessinsider.com/china-social-credit-system-punishments-

and-rewards-explained-2018-4.

2. Id.; Rene Raphael & Ling Xi, “Discipline and Punish: e Birth of

China’s Social-Credit System,” e Nation (January 23, 2019), https://www

.thenation.com/article/china-social-credit-system/.

3. Xinmei Shen, “What is ‘Skynet’, China’s massive video surveillance

network,” Abacus News (October 4, 2018), https://www.abacusnews.com/who-

what/skynet-chinas-massive-video-surveillance-network/article/2166938.

20million cameras is a conservative number. Other sources report as many as

200million, “a gure set to triple” by 2020. Megan Palin, “China’s ‘social credit’

system is a real-life ‘Black Mirror Nightmare,” New York Post (September19,

2018), https://nypost.com/2018/09/19/chinas-social-credit-system-is-a-real-

life-black-mirror-nightmare/. I should also point out that when writing about

a national surveillance program called Skynet, the name of this column seems

either a little too on the nose or a bad parody of itself.

4. See Paul Wallis, “Op-Ed: Chinese ‘Social Credit,’ or how AI can

rule your life right now,” Digital Journal (September 18, 2018), http://www

.digitaljournal.com/tech-and-science/technology/op-ed-chinese-social-

450 e Journal of Robotics, Articial Intelligence & Law [2:445

credit-or-how-ai-can-rule-your-life-right-now/article/532386; Karen Hao,

“Is China’s social credit system as Orwellian as it sounds?,” MIT Technology

Review (February 26, 2019), https://www.technologyreview.com/f/613027/

chinas-social-credit-system-isnt-as-orwellian-as-it-sounds/.

5. See Palin, supra note 3 (“If your best friend or dad says something

negative about the government, you’ll lose points too”); Maya Wang, “Chi-

na’s Chilling ‘Social Credit’ Blacklist,” Human Rights Watch (republished

from the Wall Street Journal) (December 12, 2017), https://www.hrw.org/

news/2017/12/12/chinas-chilling-social-credit-blacklist (noting that penalties

assessed due to the social credit system “were exacted in wildly arbitrary and

unaccountable manners”).

6. John Frank Weaver, Robots Are People Too (Praeger Publishing, 2013),

46-48 (“Robots Are People Too”); Brendan McKinnon, “Should Articial

Intelligence Be a Legal Person?,” BC Law Magazine (September 11, 2018),

http://lawmagazine.bc.edu/2018/09/should-artificial-intelligence-be-a-

legal-person/; John Frank Weaver, “We Need to Pass Legislation on Articial

Intelligence Early and Oen,” Slate (September 12, 2014), https://slate.com/

technology/2014/09/we-need-to-pass-articial-intelligence-laws-early-and-

oen.html.

7. Robots Are People Too, supra note 6, at 46-50.

8. Lewis C. Solmon and Michael Tierney, “Education,” in Glenn Porter,

ed., Encyclopedia of American Economic History, vol. 3 (New York: Charles

Scribner’s Sons, 1980), 1015-1016; Arthur M. Johnson, “Economy Since 1914,”

in Glenn Porter, ed., Encyclopedia of American Economic History, vol. 1 (New

York: Charles Scribner’s Sons, 1980), 117 & 127-128; omas K. McCraw,

“Regulatory Agencies.” in Glenn Porter, ed., Encyclopedia of American Eco-

nomic History, vol. 1 (New York: Charles Scribner’s Sons, 1980), 803-804.

9. Klaus Schawb, e Fourth Industrial Revolution (Currency, 2017);

Elizabeth Schulze, “Everything you need to know about the Fourth Industrial

Revolution,” CNBC (January 17, 2019), https://www.cnbc.com/2019/01/16/

fourth-industrial-revolution-explained-davos-2019.html; Bernard

Marr, “e 4th Industrial Revolution is Here—Are You Ready?,” Forbes

(August13, 2018), https://www.forbes.com/sites/bernardmarr/2018/08/13/

the-4th-industrial-revolution-is-here-are-you-ready/#4d44105a628b.

10. Raphael & Xi, supra note 2.

11. Miranda Hall & Duncan McCann, “What’s Your Score?,” New

Economics Foundation (July 10, 2018), https://neweconomics.org/2018/07/

whats-your-score.

12. Cathrin Schaer, “Germany edges toward Chinese-style rat-

ing of citizens,” Handelsblatt Today (February 17, 2018), https://www

.handelsblatt.com/today/politics/big-data-vs-big-brother-germany-edges-

toward-chinese-style-rating-of-citizens/23581140.html?ticket=ST-234038-

NWJa3mpyHbTbfsifGTBh-ap4.

2019] Is China’s Social Credit System the Future? 451

13. Nicole Kobie, “The complicated truth about China’s social

credit system,” Wired (June 7, 2019), https://www.wired.co.uk/article/

china-social-credit-system-explained.

14. Ma, supra note 1.

15. Wang, supra note 5.

16. Karen Chiu, “China’s social credit system is becoming a reality,”

Abacus News (July 10, 2019), https://www.abacusnews.com/digital-life/

chinas-social-credit-system-becoming-reality/article/3017971.

17. Hao, supra note 4; Raphael & Xi, supra note 2; See Miriam A. Cherry,

“e Gamication of Work,” 40 H L. R. 851, 852 (2012).

18. See Cherry, supra note 17, at 852 (e “idea that people could be

working while they play a video game—in some instances without even know-

ing that they are working—is becoming part of our reality.”).

19. See John Frank Weaver, “Evidencing a Lack of Bias: Using Articial

Intelligence to Promote the Rule of Law,” Journal of the World Articial Intel-

ligence Conference (forthcoming). is article discusses points 1-3 in the

general context of AI used by governments.

20. See Deven R. Desai & Joshua A. Kroll, “Trust but Verify: A Guide to

Algorithms and the Law,” 31 H. J. L. & T. 1, 39-41 (2017).