It was one of the worst experiences of her life. Worst Experience
Schools are employing dubious AI-powered software to accuse teenagers of wanting to harm themselves and send the police to their homes as a result – with often chaotic and traumatic results. The software, installed on high school students’ school-issued devices, tracks every word they type and analyzes the language for evidence of self-harm.
However, the software can get it wrong by woefully misinterpreting what the students are actually trying to say. A 17-year-old in Neosho, Missouri, was woken up by the police in the middle of the night after a poem she had written years ago triggered the alarm of a software called GoGuardian Beacon, which its maker describes as a way to “safeguard students from physical harm”.
Wellness Check
Internet safety software employed by educational tech companies took off during the COVID-19 shutdowns, leading to widespread surveillance of students in their own homes. Many of these systems are designed to flag keywords or phrases to figure out if a teen is planning to hurt themselves. But as the New York Times reports, we have no idea if they’re at all effective or accurate, since the companies have yet to release any data.
Besides false alarms, schools have reported that the systems have allowed them to intervene in time before they’re at imminent risk at least some of the time. However, the software remains highly invasive and could represent a massive intrusion of privacy. Civil rights groups have criticized the tech, arguing that in most cases, law enforcement shouldn’t be involved.