The 2020 Iowa caucus has become a tech failure story: A flawed mobile app that was supposed to simplify reporting the caucus results ended up creating chaos and forced the Iowa Democratic Party to rely on paper ballot backups. Three days later, we still dont know who won and may never. But this wasnt the first time mobile apps were used for the caucus. In fact, both parties used mobile apps with similar functions in 2016. We just didnt hear much about them because they worked.
This was my worst nightmare when I was watching this happen, Rodney Guzman, the co-founder and co-CEO of the company that built the 2016 apps, told Recode. I was thinking to myself, oh my God, that couldve been us.
But it wasnt, because the 2016 app developers had resources that the 2020 developers sorely lacked: experience, time, and funding.
In 2016, both parties faced contested caucuses and the memory of 2012s botched Republican caucus, when Mitt Romney was incorrectly declared the winner, was fresh in their minds. A mobile app that precinct chairs could use to calculate and report results presumably faster and more accurately was seen as a way to avoid repeating previous mistakes.
So the parties partnered with Microsoft, which contracted a company called InterKnowlogy to build the app and other digital caucus infrastructure on Microsofts technology. InterKnowlogy had nearly two decades of experience in app development, worked with Microsoft before, and a track record of success with primetime apps, as co-founder and co-CEO Rodney Guzman calls them. The Iowa Caucuses 2020 app, in contrast, was built by a relatively unknown new company called Shadow, which had never done something on this scale before.
Both the 2016 app and the 2020 app had similar requirements: While their functions werent complicated, there needed to be several versions of the same app for different operating systems, devices, and, in the case of 2016, caucusing systems. Most importantly and this is unusual for app development they all had to work perfectly for their debut (and only) use. There would be no time to roll out updates and improvements. The consequences of getting it wrong, as we now know, would be very public.
But the 2020 app developers had far fewer resources than 2016s app team had. Guzman told Recode that his company had about a year of lead time, the first three months of which were spent on product design, meeting with both parties, and doing usability studies to anticipate precinct chairs needs down to the size of the digital buttons theyd have to tap.
That three months is a longer amount of time than Shadow had in total, per multiple reports that say it only got two months to build its app. (Shadow CEO Gerard Niemira seems to dispute this, telling Vice that Shadow began [its] engagement with the IDP in August and basically had the month of August, September, October, November, and December to do it, though he didnt specify when the actual coding work began.)
To me, as an engineer, that just sounds nuts, Michael Gramley, a software engineer who worked on InterKnowlogys app, told Recode. You cannot make a stable platform in two months. We needed that much time just to test everything. If they really did that, then whoever the management was in that company set their engineers up to fail.
The extra months InterKnowlogy had gave its engineers time to properly build and test the app. It also gave the product side time to properly prepare and train the precinct chairs who would be using it. Guzman and his team took into account that chairs may not own or use smartphones and had little experience with apps, let alone any familiarity with necessary security measures like two-factor authentication.
We were really concerned that a lack of training was going to be the cause of most of the issues, Guzman said. We spent a lot of time and preparation to make sure that both parties were pushing out trainingmaterial.
This proved to be a good idea. The 2020 apps main problem, according to the IDP, was a coding issue that caused partial results to be reported to the IDP headquarters. But there were widespread reports that the precinct chairs werent adequately prepared or instructed on how to use the app, and that they found it difficult to impossible to download, install, and boot on their smartphones.
One thing that shocked Guzman was that Shadow didnt put its app in Apples or Googles app stores, and instead forced users to sideload it basically, download the app through a different channel. Thats a difficult process for experienced smartphone users, let alone someone who barely uses a smartphone.
A lot of our jaws hit the floor when we saw that they didnt even use the proper mobile app stores to deploy the apps, Guzman said. It had red flags all over it.
The 2016 app, by contrast, was available in app stores for precinct chairs to download the way they would any other app by the beginning of October 2015 (you can still find both the Democrat and Republican versions in Androids app store). Gramley guesses the difference here is that Shadow didnt finish its app in time to get the app stores approval.
InterKnowlogy also built an automated phone system that precinct chairs could use to call in the results if they werent able to or didnt want to use the app. Even with all of its work training precinct chairs on the app, Guzman said a large percentage of the results (he couldnt remember the exact number) were reported through the phone. According to Gramley, there were a couple disconnections, but it was not a major issue.
This years caucus appears to have relied on human operators instead, and, the New York Times reported, there werent enough of them to handle a call load that really should have been anticipated after 2016.
Gramley joined the project about six months before the apps debut, when the prototypes were done and the work building the actual apps began. He said there were about five to seven engineers on the project working full time at that point. They had a code complete version two months before the caucus and spent the rest of the time testing it out and fixing bugs.
Even with the comparatively longer lead time, Gramley said, InterKnowlogys engineering team was working 80-hour weeks leading up to the caucus.
We pulled it off, but we had a lot of late nights leading up to it, he said.
Had InterKnowlogy been offered the same tight deadline as Shadow apparently was, Guzman said he wouldnt have accepted the project.
Theres too much at risk, he said. I just wouldnt have done it.
Time was not the only thing Shadow was short on. Federal Election Commission records show that the IDP paid Shadow just $63,000 for its work on the app. Even if you combine that amount with the $58,000 that the Nevada State Democratic Party paid Shadow for a very similar (and, now, unused) version, thats still far less than what an experienced company with the necessary talented engineers would charge for this kind of work. Gramley didnt know how much InterKnowlogy was paid, but he estimated that the engineers salaries alone would have been between $1 million and $1.5 million for six months of work.
Guzman wouldnt reveal how much InterKnowlogy was paid but said it was at least 10 times more than Shadow was.
You get what you pay for, he said.
According to the New York Times, only a quarter of the chairs in Iowa were able to use Shadows app. The rest relied on the phone system, which was overwhelmed with calls. Many chairs reporting hours-long hold times and hang-ups before they could report their results (including one while the chair was on CNN).
When all was said and done, Microsoft took a bit of a victory lap, posting about the new technology that ushered in a new era for the caucuses. PBS even cited the app and its speed as marking the beginning of the smartphone election. Ninety-five percent of the 2016 caucus results were in before 11 pm CT.
Four years later and more than 85 hours after the 2020 Iowa caucus began, the results are still not complete and the Democratic National Committee chair is calling for a recanvass. Asked for comment, Shadow pointed Recode to a statement on its website that said it sincerely regrets what happened on caucus night and will apply the lessons learned in the future.
The Iowa Democratic Party didnt respond to request for comment on why it didnt use the 2016 app for this year, and Microsoft wouldnt comment beyond a tweet from a spokesperson that said it wasnt involved in the 2020 app.
What frustrates Guzman and Gramley is that Shadows mistakes are being used to disparage the very concept of using mobile apps for elections, and InterKnowlogys success in 2016 has largely been erased. Several articles about 2020s woes didnt mention the 2016 apps existence. For instance, the Wall Street Journal said Iowas tally-by-app experiment failed.
Despite pre-caucus concerns that the app could get hacked and the results altered, in the end its undoing appears to be something decidedly less nefarious: The people behind it didnt know what they were doing. That disaster of a mobile app was the effect, not the cause. And thats a mistake humans have been making for millennia since well before apps and smartphones were even invented.
My fear is that the lesson people will take from this is that apps are bad for this, not that bad processes are bad for this, Gramley said. Its unfortunate because you can do this the right way. There is a right way to do this.
Open Sourced is made possible by Omidyar Network. All Open Sourced content is editorially independent and produced by our journalists.