‘The Least Safe Day’: Rollout of Gun Detecting AI Scanners in Schools Has Been a ‘Cluster,’ Emails Show
August 25, 2022On March 22, Jennifer Dean, the principal of Mallard Creek High School in Charlotte, NC emailed members of her staff to tell them about a new security system being installed. Made by a Massachusetts-based company called Evolv Technology, the system claims to use artificial intelligence to detect guns and other concealed weapons by scanning people as they enter a building.
“I am excited about the new equipment and our scanning process, which will become another layer of safety support for our school,” Dean wrote in an email announcing the scanners’ installation at Mallard Creek High. “We have the unique opportunity, through the use of these scanners, to proactively screen for weapons everyday and to deter people from bringing weapons on our campus.”
Evolv has been especially bullish in marketing its new tech. Its CEO has made multiple media appearances in the wake of the deadly shootings in Uvalde, Tex., and Buffalo, NY, in which he implied that its system could have prevented the tragedies. Schools have been key among its customers, which also include sports stadiums and other large venues across the country.
The principal's email—one of almost 2,000 released to Motherboard as part of a public records request with the school district—aligned with what Evolv Technology says about its security system. On its website, the company describes the technology as an “AI-based weapons screening system” that will allow people to “move quickly through security checkpoints at a seamless pace, knowing they’re well-protected everywhere inside your venue.” The company also repeatedly boasts its “line-free” system requires “no stopping,” emptying of pockets, or removing the contents of bags, and that fewer security staff are needed than with traditional metal detectors.
There is currently no peer-reviewed research showing that AI gun detection is effective at preventing shootings, and Evolv has offered little evidence supporting claims of its system’s effectiveness in meeting these objectives. Schools have also encountered problems with the scanners confusing laptops and other everyday items with guns.
Sign up for Motherboard’s daily newsletter for a regular dose of our original reporting, plus behind-the-scenes content about our biggest stories.
But the documents obtained by Motherboard provide a more detailed look into how Evolv scanners are actually deployed and the problems they actually face. On the ground, the reality of deploying Evolv scanners is very different than marketing materials suggest. Some school administrators are reporting that the scanners have caused “chaos”—failing to detect common handguns at commonly-used sensitivity settings, mistaking everyday school items for deadly weapons, and failing to deliver on the company’s promise of frictionless school security.
“Today was probably the least safe day,” one principal observed the day scanners were deployed at her school, because the machines were triggering false alarms and requiring manual searches on “almost every child as they walked through” monopolizing the attention of safety officers who would otherwise be monitoring the halls and other entrances.
The Charlotte Mecklenburg School District (CMS) purchased the Evolv scanners because it had a gun problem. Twenty-three guns were found among its 180 campuses with more than 140,000 students in the first four months of the school year before the scanners were purchased. The Evolv scanners were one of the measures identified by the district’s Student Safety and Well Being workgroup, according to a January 2 email from the school district’s chief of police Melissa Mangum. The school district quickly agreed to pay $4.9 million for a service subscription to put scanners at its 21 high schools for a four-year period.
The school district publicly boasted that the scanners were a success because the number of guns found on campus dropped to seven in January through June. However, that number had already been dropping. In reality, it was far from such a resounding success story, according to emails obtained by Motherboard through the public records request. Among the findings:
- There is no evidence in the thousands of emails the district ever sought more information on how the scanners actually work, and administrators ignored evidence that they do not detect weapons with perfect reliability. For instance, before the contract was signed, another school district informed CMS they were able to get a Glock pistol through the machines without detection at the sensitivity setting they use daily.
- Marketing and sales staff from Evolv were regularly looped into email threads on how to respond to media inquiries and staff concerns
- Despite promises of “line-free” and seamless scanning, the rollout was, in the words of one principal, “a cluster” (as in short for “clusterfuck”). “Currently, the reality is that ‘weapons of mass instruction’ set off almost every child as they walked through,” the principal said.
- Security experts consulted by the district after the botched rollout warned the school system there is “no magic to any of these” proposed fixes and the easiest thing to do would be to tell students to “get there earlier.”
- The statistical evidence that Evolv’s scanners made schools safer is weak. During the 2021-22 school year, 30 guns were found on the school’s campuses by means other than Evolv scanners, compared to one detected by the scanners. A decline in guns found on campus began months before Evolv scanners were implemented, suggesting other measures such as an anonymous tip system through a smartphone app were more effective.
- After the Uvalde shooting, several concerned parents emailed the school district to ask what is being done to keep kids safe. The chair of the board of education responded, telling parents “A body scanner would not have stopped what happened at the school in Uvalde (or Marjorie Stoneman Douglas HS, or Sandy Hook Elementary).” But in an email template later designed for responding to such inquiries, the district’s communications team did not include this fact and instead listed the body scanners as a preventative measure against mass shootings. Security experts broadly believe body scanners and metal detectors do not prevent mass shootings, a fact Evolv conceded when asked directly.
A spokesperson for Charlotte Mecklenburg School District disputed that the scanners have been expensive security theater. “Staff and students have responded to the installation of the body scanners in an overwhelmingly positive manner,” said CMS spokesperson Cassie Fambro. “Students and staff report feeling safer, not just because of the scanners alone, but because of the multilayered approach to security that includes but is not limited to, locked exterior doors, camera systems, entry systems with camera/speaker, [and] the Say Something Anonymous Reporting System.”
After Motherboard sent Evolv a detailed list of questions for this story, Evolv Chief Marketing Officer Dana Loof responded with a written statement: “As a policy, we do not comment on our customer's implementation of Evolv, nor their security protocols. There are many factors that go into physical security—i.e., keeping weapons out of places they shouldn’t be—including technology, people, and processes, which all must be working together in order to be effective in mitigating a threat. Advanced weapons detection systems are a completely new experience for most schools deploying them and present unique security challenges. To make sure they have every bit of information necessary to create a safer environment, we are completely transparent with our customers and security professionals when it comes to our system—how the technology works, what the limitations may be, and what they need to account for. Our customers test our products and talk to other customers—many of them independently—before making the decision to move forward.”
Both Evolv and CMS declined to discuss details about how the security system works, the sensitivity settings it uses, and what objects can and can’t get through at various settings. Fambro refused to do so out of “an abundance of caution for student safety.” Evolv’s spokesperson went even further, claiming without explanation that merely reporting on the scanners or how they work endangers students.
“Our note to you, as a reporter doing your job: by publicly communicating detailed information on sensitivity settings, protocols and processes puts students and educators at risk and endangers lives,” said Evolv Chief Marketing Officer Dana Loof, in a statement sent to Motherboard.
Overall, the emails and additional reporting by Motherboard show significant growing pains for a company that has quickly expanded into major venues across the country such as NFL stadiums, world-class museums and concert halls, as well as public places such as hospitals and schools. Despite Evolv’s lofty public rhetoric, the emails and accompanying documents reveal a fundamental tension in Evolv’s strategy between security and convenience, one that it is not clear will ever be fully solved or can even get better.
“These emails reveal how deceptive and misleading Evolv is,” was the view of Donald Maye, a security researcher with the firm IPVM who has been actively investigating Evolv’s systems for months. “How I see it is: Evolv’s marketing paints a picture that fundamentally clashes with the reality on the ground.”
On January 24, a purchasing agent for the school district spoke to a student services officer in the Spartanburg, SC school district which has used Evolv scanners since 2019, seeking to find out more about the Evolv scanners. The reference was not exactly a glowing one.
Answering the question of “Do you feel these are effective in identifying firearms and knives, and if applicable potential bombs?,” the purchasing agent recorded that Spartansburg “have not found any weapons with their systems. He states that it is not foolproof, but feels it is enough of a deterrent that it keeps students from attempting to bring them in.”
However, in a test, the school district had to increase the sensitivity setting on the machines “to level D” (“A” being the least sensitive and “F” being the most) in order to detect a Glock pistol. Initially, the district had been using setting B, “but are now using C.”
In other words, the school district is using a setting on the machines that did not detect one of the most common handguns in the country. It is possible the scanners never found handguns because they acted as a deterrent or because no student brought one during this period, but it is also possible it never found any because the school district was using a setting that would not detect them. “When they have a security alert, they will have the level up to D,” the document says, although it is not explained what a security alert is or how it is triggered.
Why Spartansburg would use a lower sensitivity setting than required to detect a common handgun at all is explained in the next question, “Are you receiving annoyance alerts for items like keys and phones?” The answer, in a word, was yes.
Although phones and keys—menaces to traditional metal detectors—do not set off Evolv scanners, Spartansburg did report “3-ring binders do hit it a lot. Laptops will hit.” The school also reported that about 25 percent of students have to be searched manually using the “C” setting—the one that doesn’t detect a Glock pistol. Turning up the sensitivity setting would require even more students to be manually searched.
This information—which would seem to at the very least raise questions about Evolv’s marketing and public statements—was passed on to Mangum and Charlotte Mecklenburg Schools Chief Operations Officer Brian Schultz. It is not clear from the emails if anyone else was made aware of it, but Schultz, Mangum, and five principals from CMS high schools visited Spartanburg on February 2 to see the scanners in action, although by then the contract to buy the scanners was already being finalized.
Fambro, the CMS spokesperson, declined to discuss what sensitivity setting the district uses, but said, “we were fully informed that the systems will alert on objects that are not firearms such as water bottles, eyeglass cases, umbrellas, some 3-ring binders, and some Chromebooks and laptops” and was willing to accept this “short-term inconvenience to increase school safety.”
In fact, the school district was working on its messaging on how great the scanners are before they had even received Spartanburg’s feedback. Eleven days prior, Schultz emailed Evolv’s salesperson, Kyle Correll, and cc’ed the district's communications department about crafting a PR strategy around the rollout.
“As we get nearer to the purchase of the Evolv Express Systems,” Schultz wrote, “I want to be sure we are prepared with the potential questions from staff, community and media that we will potentially receive.” He asked for “additional information regarding other districts (Spartanburg and more) and organizations (Panthers, Disney, etc)”—referring to the Carolina Panthers, which use Evolv scanners at its stadium and, according to Correll, Chief Mangum has a “great relationship” with two of their security officials. Schultz then specifically wrote that the school was requesting the information from Evolv “so the Communications Team can begin developing our own messaging based off the implementation, use, and success of the technology/system implemented in various locations around the country.”
“Great meeting you as well,” the Evolv salesperson replied an hour and a half later. “I tagged our Director of Customer Marketing as well that’s been involved with customer success and content. We’ll get you all the necessary info to help you on your end.” In future materials presented to the school board and parents, the school used information provided by Evolv, not the feedback it received from Spartanburg itself.
Despite red flags suggesting the technology may not function as promised, the district completed the purchase of scanners for 21 schools, rolling them out in phases, beginning with seven high schools in March.
In the media, CMS said it learned from the lessons of the Phase I rollout and the system was a success. “Our team is very confident that the lessons that we learn as part of phase one implementation will help guide us with the future rollout of this initiative," said Superintendent Earnest Winston in early April.
But a month later, it was clear that hadn’t happened.
“The scanners were a cluster today,” Ardrey Kell High School principal Jamie Brooks wrote to two assistant principals and Police Chief Mangum on May 2. The rest of the email detailed how various school-related items—which Brooks calls “weapons of mass instruction”—were setting off the scanners almost constantly:
It took all 10 people to even come close to managing the chaos at the one entrance (they are supposed to eventually be at other entrances)...we do not have the manpower for this. We must have another CSA [school security] to help search these bags. Currently, the reality is that ‘weapons of mass instruction’ set off almost every child as they walk through. If you have multiple binders or spiral notebooks in your bag then it lights up and we must search. The solve I was given was literally to ask kids not to bring so many binders. Seriously? So again…weapons of mass instruction at [Ardrey Kell] are going to cause chaos everyday :) Today was probably the least safe day at AK as all hands were at the front doors instead of monitoring kids throughout the building. Brian Schultz was here today so he witnessed it first-hand.
In her response, Mangum confirmed that other schools experienced “the notebook issues” as well and they are “reviewing and will be testing other components that may enhance the scanning process with the laptops and the notebooks,” although it is unclear what this was referring to and no such process or components are mentioned again in other emails.
A week later, school board member Sean Strain emailed interim superintendent Hugh Hattabaugh asking for “an update on the rollout of the scanners,” linking to a Facebook post showing massive lines outside Providence High School.
“This is what it looked like at Providence this morning—and seemingly every morning—with the new protocols in place,” Strain wrote. “Many reports of alerting on binders, which begs the question as to whether the scanners can be adjusted to accommodate and still reliably detect larger/thicker metal objects such as guns and knives.”
This was an astute observation by Strain, as he intuited exactly the tradeoff that Spartanburg schools experienced with their own testing: Turning up the sensitivity to reliably detect guns also mistakes everyday school items as weapons, but turning the sensitivity down to provide the convenience Evolv markets as its main advantage over traditional metal detectors misses real threats. It is also a tradeoff Maye at IPVM has documented as well.
Evolv’s manual instructs customers to adjust the sensitivity setting to find the right balance between nuisance alarms and actual weapons detection, an instruction that fundamentally concedes the system cannot reliably do both detect weapons and have a tolerable level of nuisance alarms, posing much the same trade-off traditional metal detectors do.
“All visits and input from the Evolv team indicate that it takes about 2 full weeks to fully implement,” Schultz replied to Strain, a timeline that matches the general perception that customers and visitors adjust to Evolv’s limitations and not the other way around.
It was not just Evolv’s own team that was saying this, but outside experts as well. On May 26, Greg Abbott, director of security solutions at Paragon Systems, volunteered advice on how to deal with the long lines as part of a discussion with Charlotte Executive Leadership Council Security Task Force.
“There is no magic to any of these,” Abbott warned, after having talked to teams at Universal Studios and the Staples Center in Los Angeles. The recommendations: “Having teachers or staff acting in a customer service capacity” to communicate the process, “Create visuals to demonstrate the process,” “Create a separate entrance for staff” (he apparently did not know staff do not have to go through the scanners), and, the “unpopular but easiest” solution, tell students to “get there earlier.”
In her email response to the principal at Ardrey Kell, Mangum also acknowledged the need to hire more security staff to aid with the searches—the exact opposite of what Evolv promises in its sales pitch. 53 security guards were approved by the board and they planned to add more, but wanted “realistically, another CSA will probably not be available for [Ardrey Kell High School] until the new school year.” This acknowledgment that the scanners require more people to implement undermines one of Evolv’s key marketing promises.
Evolv defended its system by telling Motherboard that participating customers in the Evolv Insights program, which the company says accounts for about 30 percent of its users, have collectively detected 30,000 guns and 27,000 knives through the first six months of 2022, statistics it has also cited in press releases. Roughly speaking, this would equate to approximately 90 guns found per scanner, a startling statistic considering many of Evolv’s customers are museums, entertainment and sports venues, theme parks, and other major venues that have long conducted thorough security checks and patrons are well aware guns are not allowed inside.
When asked about the problems instituting the Evolv system, Fambro said the May 2 email was from the first day the school used the new scanners, and that afterwards “the students and staff improved the effectiveness of the process as Evolv representatives continued to work with school staff to reduce the number of searches as a result of alerts.” Motherboard then asked if part of this process was reducing the sensitivity setting and, if so, what sensitivity setting CMS changed it to. Fambro declined to comment, citing student safety concerns. But she did say the school bought an additional scanner to “improve the efficiency of ensuring our students can arrive to class on time and ready to learn.”
“What I have observed and experienced in the other schools we must adjust [sic] to every circumstance until we find the best solution,” Chief Mangum told Brooks in an email on May 2. “I know it can be frustrating, but as we saw at [another school] today (firearm discovered during the scanning), the system works.”
The case Mangum referred to was the only time an Evolv scanner in a CMS school detected a gun at any of the 21 schools to date, and has been repeatedly trotted out as evidence that, as Mangum put it, “the system works.”
This incident has been referred to repeatedly by Evolv CEO Peter George, including during an earnings call on May 11. “Our Evolv Express platform was able to identify a student who attempted to enter the District’s Berry Academy of Technology with a fully loaded handgun,” George said. “The student was stopped and arrested and thankfully a potential tragedy was avoided.” And in an August 11 press release, the company said, “In the past few months alone, two incidents have taken place where Evolv Express systems prevented a loaded gun from entering a hospital and a school where, in each case, it was believed there was intent to do harm and law enforcement was engaged.” Motherboard asked Evolv if they were referring to the incident at CMS, but Evolv did not respond to that question. Motherboard could not find any publicly reported incidents from other school districts of Evolv detectors involved in a gun being found on a school campus.
The company’s comments clearly imply the student was entering the school to commit a mass shooting and Evolv’s technology stopped it. But that’s not what the Charlotte Mecklenburg Police Department told Motherboard.
“There is no information to suggest the student planned on using the firearm or made any threats,” a spokesperson for the department told Motherboard.
George is a frequent interviewee following mass shootings, and in each interview he toes the line between clearly implying Evolv can help prevent mass shootings without ever actually saying so.
“Could our system have stopped it?” George told the Washington Post following the Buffalo supermarket shooting. “I don’t know. But I think we could democratize security so that someone planning on hurting people can’t easily go into an unsuspecting place.” And after Uvalde, George said something similar at an investor conference: “A lot of people asked me the question after what’s happened in Uvalde, would we have stopped that? The answer is when somebody goes through our system and they have a concealed weapon or an open carry weapon, we're gonna find it, period. We won't miss it.”
This, of course, is not necessarily true, as Spartanburg’s own testing showed when they were able to pass through the scanners with a Glock pistol undetected.
The emails obtained by Motherboard also discuss an incident at Julius L. Chambers High School in which a student posted a picture of himself holding a gun inside a school restroom. Publicly, the school district said “there is no evidence that any gun has made it past a weapon detector” because the date of the photo was unknown, an argument Fambro, the CMS spokesperson, says the district stands by. But, privately in emails to the communications staff, Chief Mangum stated that the incident “occurred approx. 2 weeks before the picture was posted” in May 2022—months after Evolv’s systems were installed at the school. Fambro said she did so “simply because that’s what the student in question reported.” The police report lists the date and time of the incident as between May 2 at 7:30 a.m. to May 19 at 3:45 p.m. The police did not respond to a Motherboard inquiry specifically asking if any metadata from the photo indicated when it was taken.
Maye, of IPVM, says it is also inaccurate to imply Evolv’s scanners—or any other type of gun detection system—can play a role in stopping mass shootings for the simple and obvious reason that Evolv scanners at CMS schools are for detecting concealed weapons. Anyone walking into a school with guns with the intention of committing mass murder does not care if anyone knows they have guns or not. The Uvalde shooter was detected by officials at the school before he entered the building; the shooter began firing his gun before even entering the school.
It is also a point Evolv’s competitor, PatriotOne, has stated clearly in its own marketing materials. “For example, a weapons scanner at the entryway can detect people bringing weapons into a building, but it is ineffective against an individual with malicious and specific intent – this attacker will walk right through an entry scanning system,” it wrote in a blog post specifically about school security.
After George’s many public statements, Maye asked Evolv directly if the company thinks its product can prevent mass shootings. A spokesperson replied, “We have never stated that Evolv could have made a difference in Uvalde.”
Following the Uvalde shooting, CMS received a number of emails from worried parents. A typical one, from a mother of two elementary school students, expressed a concern of “the lack of security in our elementary schools. I am aware that metal detectors are being installed in high schools, however, with the recent horrific news, it is beyond evident that these shootings are not restricted to high schools. I plead with you all to consider installing them in ALL CMS schools.”
The chair of the board of education, Elyse Dashew, responded to this note by admitting that the Evolv scanners would not have prevented the recent mass shootings. “Regarding your question about the body scanners in our high schools—these are newly installed and so far they seem to be working well,” Dashew wrote. “A body scanner would not have stopped what happened at the school in Uvalde, however (or Marjorie Stoneman Douglas HS, or Sandy Hook Elementary).”
But the school district’s communications team took a different approach. A day later, it created its own template for “the response that I am providing to citizens who email regarding school safety issues,” as Schultz wrote.
The template has a list of 13 “efforts” in place. One of them is “Body scanners at our large comprehensive high schools.” The template does not include any disclaimer—as in Dashew’s email or Evolv’s own admission—that says body scanners and metal detectors do not prevent mass shootings.
When asked about this by Motherboard, Fambro said, “While we cannot speak to those specific tragedies and what transpired in those respective schools, we can attest to CMS’ commitment to do everything in our power to protect our 141,0000 students using a variety of tools, procedures, and protocols. The Evolv system is designed to identify concealed weapons on individuals entering our schools. However, the layers of security measures we have are in place to protect staff and students from many different types of threats.”
In the weeks rounding out the 2021-2022 academic school year, there were other signs Evolv’s technology wasn’t quite as seamless as promised. On June 7, one of the tablets where the scan results are displayed was dropped and it broke. Evolv bills customers on an annual subscription rather than allowing customers to buy the scanners outright. That subscription, as CMS learned, does not include the cost of replacing tablets. It charged CMS $2,100 for a new one, double the cost of a top-of-the-line iPad Pro. The company also said it had to stop by each school individually to program a software update manually rather than via the internet like most software updates are issued.
Motherboard asked Evolv about both of these incidents, but Evolv’s response did not address them.
CMS had repeated reliability issues with the scanners not working. On May 4, an assistant principal at Rocky River High School emailed Schultz to inform him that on “two separate mornings” their Evolv scanner could not function properly due to a series of error messages. On May 17, the principal of North Mecklenburg High School also emailed Schultz with similar news that one of their scanners wasn’t working. “We are operating our single scanner at the auditorium. We are not able to accommodate moving all of our students through that single scanner entry,” the principal wrote. Evolv had to replace a part on the scanner and a technician fixed it two days later.
Despite these issues, the school district steadfastly defended the purchase in public statements and to Motherboard. In contrast to the complaints made in the emails, Schultz and members of the district’s communications team sometimes went out of their way to emphasize the scanner’s supposed effectiveness.
“They work as a preventative tool as well as a prohibitive tool; students and staff know the scanners work,” Schultz said in a prepared statement responding to a media inquiry. “The scanners have been utilized successfully but are still new to all schools.”
In putting together “some talking points” for the interim superintendent, a member of the communications staff wrote, “The last gun at Harding was found in a student book bag by an administrator—do we know if the student ditched the book bag to avoid going through the scanner? I’m trying to make the point that the scanners work and that students are aware of that, if I can truthfully make that connection.”
On June 10, Mangum answered “LAST ONE YOU ARE CORRECT, STUDENT LEFT BOOKBAG WITH GUN INSIDE AT A LOCATION NEAR THE SCANNER, WENT THRU THE SCANNER AND RETURNED TO GET THE BOOKBAG BUT ADMINISTRATOR GOT IT FIRST (AVOIDED THE SCANNER PURPOSEFULLY WITH BOOKBAG)”
But Maye says this instance has little if anything to do with Evolv or its technology. “A human detected the gun, not Evolv’s technology,” Maye told Motherboard. “In this case, if Evolv’s metal detector had been unplugged, it would have been the same outcome.”
There is no mention in the school district’s talking points for the superintendent of the times the scanners haven’t worked, of the student who got into the school bathroom with a gun despite the scanners being in place, the technical difficulties with the scanners, or the long line/entry issues.
Nevertheless, the school district not only decided to continue using the scanners, but to expand their use. This coming school year, CMS will use Evolv scanners at all middle schools, including kindergarten through 8th Grade campuses at an additional cost of $9.9 million for a four-year subscription.
“Similar to the high school rollout during the last school year,” said Chief Equity and School Performance Officer Kondra Rattley in a July 7 email, “this effort demonstrates our commitment to providing a secure school environment where students can thrive academically, socially, and emotionally.”
The documents responding to Motherboard’s public records request can be found here.