An ‘Off-the-Shelf, Skeleton Project’: Experts Analyze the App That Broke Iowa

The app used to report early results in Iowa's Democratic Presidential primary caucus was rudimentary in many ways, according to analyses by multiple Android app development experts and cybersecurity professionals who decompiled and studied the app after it was obtained by Motherboard.

The app, called IowaReporterApp and developed by a company called Shadow Inc., malfunctioned during the caucus, causing mass chaos and delaying the public reporting of results until Tuesday evening. The app was designed to rapidly report early results, not tabulate final vote counts. That means its failure will not result in the election result being altered.

But, until now, very little has been published about how the app was designed, how it was supposed to work, and what went wrong on caucus night.

Motherboard asked six cybersecurity and app development experts we trust to analyze the app. The app was built on top of React Native, an open-source app development package released by Facebook that can be used for both Android and iOS apps, according to Kasra Rahjerdi, who has been an Android developer since the original Android project was launched, and Robert Baptiste, a white-hat hacker who has exposed security flaws in many popular apps and reviewed the code. Rahjerdi said that the app contains default React Native metadata and that it comes off as a "very very off the shelf skeleton project plus add your own code kind of thing."

"Honestly, the biggest thing is—I don’t want to throw it under the bus—but the app was clearly done by someone following a tutorial. It’s similar to projects I do with my mentees who are learning how to code," Rahjerdi said. "They started with a starter package and they just added things on top of it. I get deja vu from my classes because the code looks like someone Googled things like 'how to add authentication to React Native App' and followed the instructions," Rahjerdi said.

"The mobile app looks hastily thrown together," Dan Guido, CEO of cybersecurity consulting firm Trail of Bits, told Motherboard.

IowaReporterApp had a few basic functions, according to the experts who analyzed it:

• Once a precinct chair logged in using a precinct identifier number, PIN code, and two-factor authentication, they were run through some basic information about how to run a caucus.
• Precinct chairs were asked to enter the total number of attendees at a caucus.
• Precinct chairs were then asked to enter vote totals for the first round of the caucus and the second round of the caucus. The app was then supposed to calculate how many delegates each candidate was supposed to be awarded.
• The app was supposed to then send these results to a Google Cloud Functions backend. That backend was controlled by Shadow.

In an interview with Motherboard, Shadow CEO Gerard Niemira said that the app was simple by design.

"For something like this, you don’t want to introduce complexity where there doesn’t need to be any," he said. "The point of this app was to help temporary precinct chairs do the math and get good results in the room and speed up the process, help them basically. That is a relatively simple function, it’s basically a calculator, so that’s the approach we took to it."

"The app was sound, the data that came out of the app was sound, the math that was done on the app was sound," he added. "All the the results we collected on the app were sound and have been verified as such."

One of the main reasons there was so much confusion on caucus night, Niemira said, is because of a problem that occurred when Shadow tried to move the results it collected onto a verification system controlled by the Iowa Democratic Party (IDP). During that process, a data-formatting error caused it to not be accepted.

"We have this independent verification step," Niemira said. "In the course of doing that, we had some code that would look at our results database and then move that over to the IDP’s quality control check environment. In the process of doing that, we had some faulty code that took the data and put it into a format that made it fail the checks by the IDP. That was throwing up flags, which took time to resolve."

A team of researchers at Stanford University, including former Facebook chief security officer Alex Stamos and students Jack Cable, Pierce Lowary, and Alex Zaheer, said that while analyzing the app, they found potentially concerning code within it, including hard-coded API keys. Motherboard decompiled the app and verified the presence of an API key. Stamos' concern was that a hacker could potentially change or reset data on the servers. Stamos added that he was not comfortable probing the company's backend without its permission. Shadow insisted that the app was configured correctly.

Niemira said "it would be factually inaccurate to say that someone, if given that API key and the other details in this package, could have access to any underlying data." Two other experts also leaned closer to Niemira's position, saying that the existence of the API keys in the code does not necessarily mean there could be security vulnerabilities, and is instead an expected app development practice.

"I can’t name that firm, but we did do an independent audit. That audit uncovered a few configuration items that we remediated well in advance of the caucus," Niemira said of the app's security testing.

Guido said Trail of Bits was surprised to find the app runs on Android phones as old as version 5.0, which was released in 2014.

"Any device that old is much easier to hack and harder to consider 'secure.' At minimum, we would have recommended restricting it to only phones that meet certain minimum criteria," he said.

Know anything more about the Shadow app or other apps being used in elections? We'd love to hear from you. Using a non-work phone or computer, you can contact Joseph Cox securely on Signal on +44 20 8133 5190, Wickr on josephcox, OTR chat on jfcox@jabber.ccc.de, or email joseph.cox@vice.com.

Caucus chairs on the ground experienced problems with the app beyond the reporting errors cited by Niemira. Multiple Iowa caucus chairs said they had trouble logging into the app, which Niemira believes was caused by users inputting the wrong credentials.

"A lot of volunteers were onboarded close to caucus day or even the night before, so a lot of them didn’t have the chance to properly get set up before the event started, so that’s an important thing to note," he said.

To properly login and submit results, caucus chairs had to enter a precinct ID number, a PIN code, and a two-factor identification code, each of which were six-digits long. "We saw a lot of people entering their precinct ID instead of their PIN in the PIN spot. There were some issues with not knowing where to put what credential, which is a difficult thing to design around,” Niemira said. “Having to sign in with three different six-digit numbers is confusing on the best day, but it was a call that was made in order to help keep this process as secure as possible.”

"Our message is that apps like this should be developed in the sunlight and part of an open bug bounty"

DNC chair Tom Perez said that the app would not be used again in any Democratic primaries or caucuses. The app's entire backend is down and no information can be accessed through it or from the company's servers.

"The electoral data and results have been exported from the application and are in the process of being verified through the paper record," the IDP said in a statement.

Shadow distributed the app to precinct chairs using app development testing services called TestFairy on Android and TestFlight on iOS. On Tuesday, the company seemingly deleted that particular version of the app from that service, as a link distributed to precinct chairs was no longer working. According to state records, the app was built in several months at a cost of $63,182.

"We started our engagement with the IDP in August and began requirement gatherings and beginning to develop the app at that point, so we basically had the month of August, September, October, November, and December to do it, though requirements gathering takes a long time, so we didn’t have a final production version of this until pretty close to caucus time," Niemira said.

The Department of Homeland Security said Tuesday that it offered to analyze the security of the app but the IDP declined. Bob Lord, the DNC's chief security officer, directly urged Iowa democrats to not use the Shadow app, according to the Wall Street Journal .

The app does not contain voter data, but instead is designed to send caucus counts to a Google server. The security experts we spoke to said the app is simple.

The Democratic National Committee declined to comment for this article.

Election security experts have been saying for years that we should not put election systems online, and that we shouldn't be using apps to transmit results. And, if U.S. election officials are going to use apps like this, that they should be open to scrutiny and independent security audits.

“We were really concerned about the fact there was so much opacity. I said over and over again trust is the product of transparency times communication. The DNC steadfastly refused to offer any transparency. It was hard to know what to expect except the worst,” Greg Miller, cofounder of the Open Source Election Technology Institute, which publicly warned the IDP against using the app weeks ago, told Motherboard.

Stamos echoed that sentiment. "Our message is that apps like this should be developed in the sunlight,” he said, “and part of an open bug bounty."

Samantha Cole contributed reporting.