U.S. Senators, Elizabeth Warren, Edward J. Markey, and Richard Blumenthal sent, on 4 October 2021, a letter to four educational technology companies regarding their use of artificial intelligence ('AI') and algorithmic systems to monitor students' online activity. In particular, the letter highlights that the recent expansion of remote learning increased the use of online monitoring software to track student activity, but at the same time studies already found numerous unintended but harmful consequences of student surveillance programs that target vulnerable populations. More specifically, the letter states that AI and algorithmic systems frequently mischaracterise students' activity and flag harmless behaviour as a 'threat'. Moreover, the letter states that school disciplinary measures have a long history of disproportionately targeting students of color, and research has shown that students from minority or marginalised communities are far more likely to be flagged. Furthermore, the letter explains that language processing algorithms are less successful at analysing language of people of color, especially African American dialects and that this raises concerns that digital student surveillance platforms will perpetuate racial and discriminatory biases.

In addition, the letter details that mischaracterisations may have long-lasting and harmful effects on students' mental health due to stigmatisation and differential treatment following a potential false report. Lastly, the letter seeks information regarding the steps each company is taking to ensure the efficacy of its products and to mitigate potential harms on students, including perpetuating discriminatory biases.  

You can read the press release here and the letter here.