Automated Decision System Task Force
Automated Decision System Task Force
By Lindsay Greyerbiehl
In New York City, how, when, and why automated decision systems (“ADS”)[1] are used is rarely public knowledge. The New York City Council responded to growing concerns about government use of ADS, by enacting Local Law 49 (“LL49”) of 2018.[2] The initial bill text required agencies that use ADS to make their source code[3] public.[4] In response to administration concerns that publicizing source code would violate contract vendors, it was amended. After dropping the source code requirement and adding a mandate for a public task force on ADS the City Council passed the bill in December 2017. On January 17, 2018, after Mayor de Blasio failed to sign the legislation LL49 became law, mandating ADS be implemented fairly and with greater transparency.[5] Mayor de Blasio, subsequently formed the ADS Task Force (“Task Force”) in May 2018, as required by LL49.[6] As the first in the country, the Task Force was the opportunity for New York City to set a national model on how ADS are utilized by city government.[7] The Task Force had 18 months to recommend how city agencies and the public could review publicly-operated ADS.[8]
On January 22, 2018, a group of experts and organizations, including S.T.O.P.’s founder and experts from the AI Now Institute, Center for Democracy and Technology, and the Legal Aid Society, composed a letter to Mayor de Blasio about the Task Force’s formation.[9] Highlighting the need for expertise across multiple disciplines recommendations included S.T.O.P.’s founder and representatives from Data for Black Lives and Picture the Homeless.[10] The letter stressed the need for transparency about Task Force members’ possible conflicts of interest,[11] and appointees with commercial relationships to ADS vendors were asked to disclose such conflicts of interest.[12]
In the early stages, instead of being consulted, the Task Force was presented with plans, language, and meeting agendas that were already set by city officials.[13] One specific example of this is the unofficial, but initially vital role, that the Jain Family Foundation played in the Task Force. The foundation provided “proposed language and policy documents for the Task Force to ratify”.[14] With concerns from Task Force members the relationship with the foundation was dissolved.[15]
On August 17, 2018, the January signatories and additional organizations noted the Task Force’s lack of progress.[16] The letter warned that the Task Force may lose its potential to be an international model for government ADS regulation.[17] Notably, these outside observers come to a consensus on a definition of ADS, the first step in any analysis of ADS regulations, in a matter of weeks. The Task Force was never able to agree on a definition.[18]
Beyond definitional disagreements, the Task Force struggled to carry out even the most cursory evaluation of ADS. One reason Task Force members struggled is that members were deprived of essential information, such as a list of City-operated ADS. Without this information, members could only guess at structural problems and bias impacting the ADS used by government agencies. For this reason, Task Force members and outside advocates recommend the creation of an ongoing permanent independent body to oversee and regulate the implementation of ADS, guaranteeing such an entity sufficient information and independence.[19]
On March 1, 2019, many of the same advocates sent the Task Force a letter denouncing their lack of meaningful public engagement.[20] With only a few page website that went live months after the Task Force was created, nothing else had been done to engage the public.[21] There was nothing preventing the Task Force from engaging with the public, as other task forces had done just that. While New York City may have been the first jurisdiction in the country to propose such an effort, it was quickly falling behind. Without publicly posted agendas and minutes, the letter noted, the Task Force started to mirror the opaque algorithms it had intended to make more transparent.[22] The advocates demanded that the Task Force engage with the public in broad and meaningful ways to get as much input as possible.[23]
On March 27, 2019, the Task Force announced its plan for two public forums both held at the New York School of law,[24] the first of which took place on April 30, 2019.[25] With little advertisement, the forum was sparsely attended. During questioning, Task Force members confirmed that a list of City-operated ADS still had not been provided to the Task Force.[26] Even with experts providing an ADS definition the Task Force was still struggling to decide on a definition for their use. After two hours of presentations by guest speakers and questions by Task Force members, public engagement only lasted about 30 minutes.[27] The public cited concerns that the forum was engaging with community members too late in their process and in an inaccessible way.[28] A month later the second and last public forum was held.[29] Members of the public brought concerns about tasks members ties to private companies that make a profit from ADS, the need for more transparency from the Task Force and concerns about the NYPD’s resistance to oversight in their use of ADS.[30] The public forums highlighted the continued dysfunction and lack of progress of the Task Force.
After the public forums, there was minimal community outreach and there is no public information available about further public meetings such as transcripts or meeting notes. Never able to agree on what constitutes an ADS, little public engagement, and a lack of transparency from the city about what ADS are in use, the Task Force struggled. Ultimately, the Task Force failed to make meaningful recommendations in its final report that was released on November 19, 2019. [31] Instead, the report offered broad ideas about ADS oversight that could have been provided without a Task Force.
In conjunction with the release of the report, Mayor de Blasio issued an Executive Order establishing an Algorithm and Management Policy Officer.[32] The position will be supported by a steering committee, whose power remains unclear.[33] Intended to be a continuation of the Task Force, it’s unclear what powers, if any the AMPO would have to gather City data or secure meaningful change. Specifically, there is no guarantee that the AMPO will gain comprehensive list city-operated ADS, and the budget for the position and its supporting personnel has not been allotted. With the first report due on December 1, 2020, the AMPO’s task is both clear and daunting, but the resources and power remain ambiguous.[34] Additionally, it seems unlikely that a single annual report will be able to keep pace with new technology the city deploys. Sadly, the position may merely placate public concern, rather than holding city agencies accountable.
[1] ADS generally refer to systems that use algorithms to make decisions or predictions without human involvement.
[2] Office of the Mayor, Mayor de Blasio Announces First-In-Nation Task Force To Examine Automated Decision Systems Used By The City, The Official Website of the City of N.Y., May 16, 2018, https://www1.nyc.gov/office-of-the-mayor/news/251-18/mayor-de-blasio-first-in-nation-task-force-examine-automated-decision-systems-used-by.
[3] Source code is the human readable formula of a computer program.
[4] Automated Decisions Used by Agencies: Hearing Before the Committee on Technology N.Y. City Council, 5, 2017, (Statement of James Vacca)
[5] Id.
[6] N.Y Local Law 49 of 2018.
[7] Office of the Mayor, supra.
[8] Local Law 49, supra.
[9] Letter to Mayor de Blasio, NYC Automated Decision Systems Task Force, January 22, 2018.
[10] Id.
[11] Id.
[12] Id.
[13] Diana Budd, New York City’s AI Task Force Stalls, Curbed N.Y., April 16, 2019. https://ny.curbed.com/2019/4/16/18335495/new-york-city-automated-decision-system-task-force-ai
[14] Id.
[15] Id.
[16]Letter to Mayor de Blasio, NYC Automated Decision Systems Task Force, August 17, 2018.
[17] Id.
[18] The definition suggested “’automated decision system’ is any software, system, or process that aims to aid or replace human decision making. Automated decision systems can include analyzing complex datasets to generate scores, predictions, classifications, or some recommended action(s), which are used by agencies to make decisions that impact human welfare.” Id. at 2.
[19] Id.
[20] Letter to Mayor de Blasio, New York City’s Automated Decision Systems Task Force-Public Engagement, March 1, 2019.
[21] Id.
[22] Id.
[23] Id.
[24] N.Y.C. Automated Decision Task Force Announces Spring Public Forums, Press Release, March 27, 2019.
[25] Public Engagement Forum, Automated Decision Systems Task Force, April 30, 2019.
[26] Id. at 6
[27] See Id.
[28] See Id.
[29] Public Engagement Forum, Automated Decision Systems Task Force, May 30, 2019.
[30] Id.
[31] New York City, Automated Decision Systems Task Force Report, November 2019, https://www1.nyc.gov/assets/adstaskforce/downloads/pdf/ADS-Report-11192019.pdf
[32] City of N.Y., Exec. Order No. 50, November 19, 2019, https://www1.nyc.gov/assets/home/downloads/pdf/executive-orders/2019/eo-50.pdf
[33] Id.
[34] Id.