sarahlee310:

It was supposed to be a “killer app,” but a system deployed to volunteers by Mitt Romney’s presidential campaign may have done more harm to Romney’s chances on Election Day—largely because of a failure to follow basic best practices for IT projects.

Called “Orca,” the effort was supposed to give the Romney campaign its own analytics on what was happening at polling places and to help the campaign direct get-out-the-vote efforts in the key battleground states of Ohio, Florida, Pennsylvania, Iowa, and Colorado.

Instead, volunteers couldn’t get the system to work from the field in many states—in some cases because they had been given the wrong login information. The system crashed repeatedly. At one point, the network connection to the Romney campaign’s headquarters went down because Internet provider Comcast reportedly thought the traffic was caused by a denial of service attack.

As one Orca user described it to Ars, the entire episode was a “huge clusterfuck.” Here’s how it happened.

[…]

To build Orca, the Romney campaign turned to Microsoft and an unnamed application consulting firm. The goal was to put a mobile application in the hands of 37,000 volunteers in swing states, who would station themselves at the polls and track the arrival of known Romney supporters. The information would be monitored by more than 800 volunteers back at Romney’s Boston Garden campaign headquarters via a Web-based management console, and it would be used to push out more calls throughout the day to pro-Romney voters who hadn’t yet shown up at the polls. A backup voice response system would allow local poll volunteers to call in information from the field if they couldn’t access the Web.

But Orca turned out to be toothless, thanks to a series of deployment blunders and network and system failures. While the system was stress-tested using automated testing tools, users received little or no advance training on the system. Crucially, there was no dry run to test how Orca would perform over the public Internet.

Part of the issue was Orca’s architecture. While 11 backend database servers had been provisioned for the system—probably running on virtual machines—the “mobile” piece of Orca was a Web application supported by a single Web server and a single application server. Rather than a set of servers in the cloud, “I believe all the servers were in Boston at the Garden or a data center nearby,” wrote Hans Dittuobo, a Romney volunteer at Boston Garden, to Ars by e-mail.

Throughout the day, the Orca Web page was repeatedly inaccessible. It remains unclear whether the issue was server load or a lack of available bandwidth, but the result was the same: Orca had not been tested under real-world conditions and repeatedly failed when it was needed the most.

more…

The rumor in some tech lists is that the Romney campaign outsourced the development of Orca to Avanade, a branch of Accenture, which is heavy in the use of developers in India, China, the Philippines, Spain, Morocco, Argentina and Slovakia.  It seems that the Romney campaign was so paranoid that the Obama campaign might learn something about their application - they did no actual training on it and didn’t turn it on until election day.

——————

The Unmitigated Disaster Known As Project ORCA

This is the story mentioned the the above article.   A Romney volunteer shares a bit about his volunteering and Election Day experience.  Lots more to learn in the comments to it - some I posted below.

On one of the last conference calls (I believe it was on Saturday night), they told us that our packets would be arriving shortly. Now, there seemed to be a fair amount of confusion about what they meant by “packet”. Some people on Twitter were wondering if that meant a packet in the mail or a pdf or what. Finally, my packet arrived at 4PM on Monday afternoon as an emailed 60 page pdf. Nothing came in the mail. Because I was out most of the day, I only got around to seeing it at around 10PM Monday night. So, I sat down and cursed as I would have to print out 60+ pages of instructions and voter rolls on my home printer. Naturally, for reasons I can’t begin to comprehend, my printer would not print in black and white with an empty magenta cartridge (No HP, I will never buy another one of your products ever again). So, at this point I became panicked. I was expected to be at the polls at 6:45AM and nothing was open. I was thankfully able to find a Kinko’s open until 11PM that was able to print it out and bind it for me, but this is not something I should have had to do. They expected 75-80 year old veteran volunteers to print out 60+ pages on their home computers? The night before election day? From what I hear, other people had similar experiences. In fact, many volunteers never received their packets at all.

At 6:30AM on Tuesday, I went to the polls. I was immediately turned away because I didn’t have my poll watcher certificate. Many, many people had this problem. The impression I got was this was taken care of because they had “registered me”. Others were as well. But apparently, I was supposed to go on my own to a Victory Center to pick it up, but that was never communicated properly. Outside of the technical problems, this was the single biggest failure of the operation. They simply didn’t inform people that this was a requirement. In fact, check out my “checklist” from my ORCA packet:

Notice anything missing? My guess is the second “Chair (if allowed)” was supposed to be “poll watcher certificate” but they put chair twice. This was an instruction packet that went out to 30,000+ people. Did no one proof-read it?

From the comments:

16 Ben and John E already know all this but I’ll share.

I had something similar happen with Lawyers for Romney. Long and short of it — they didn’t get me my precinct information until 8pm the night before. I had eight different precincts I had to spot-check and they waited 10 hours before I was supposed to be on-site. Eight. Fucking. PM. The. Night.Before.

Not only that — I was supposed to have a partner that was also going to the same precincts. They gave me his name. That’s it. No phone. No email. We had no way to contact one another.

As I told Ben and John, something weird went down. They were super organized leading up to election day. Hell, I initially got contacted back in May to help out. Something really melted down in the last 10-14 days before election day. I’m trying to determine who was in charge of that clusterfuck.

Good Christ….

32 I signed up, took the training, and tried to get into the final conference call at 8:30 pm Monday night.

Couldn’t get through. About 8:40 I got a call that auto connected me to the conference call, just in time to hear it wrap up.

I never received another email. I finally went Tuesday pm to the Romney Victory/FAIL Center to make calls the rest of the afternoon. Got to see Paul Ryan roll through if nothing else.

Just saw a post on the facebooks from my lefty uncle in The Shire crowing about spending 7 hours pounding on doors Tuesday. Low tech. Works every fucking time.

As a IT developer myself, I’d say this was a world class clusterfuck. That’s a technical term, btw.

39
Gerry, in hindsight I’m 100% sorry I suggested people sign up for ORCA. I’m sorry that I signed up for it.
But I didn’t know it at the time and I only became worried about it a week before the election. By that time it was to late. The Romney campaign told the RNC that it would take care of the poll watchers.
There really weren’t any other options unless your local county committee set up an independent strikelist/poll watcher programs. Most didn’t.
Not only did this fail epically, it prevented 30,000 of the most die hard volunteers from accomplishing anything on election day.

48 I posted this in another thread but here is the summary:

I live in WI and volunteered for ORCA in early Oct. Completed the training online. Participated in the conference calls. And never heard another word from them. I emailed every day and called several times. Nothing.

I checked on 6 precincts within 30 minutes of my house by asking friends who voted in them to look. There were zero ORCA volunteers in any of them. Zero. In Wisconsin.

68 After I completed all of the ORCA training, I took a PTO day off of work, signed on for most of the calls including the final one Monday night…

…and never heard a word from them. Nothing ever about any packet, or web app, nothing about where to go even though I asked multiple times, just nothing.

I sent them a f-u email Monday night, saying “if you had too many volunteers, don’t be dicks, just tell us.”

Now it’s obvious the problems were far more widespread than I assumed.

71 Did I mention that at 5:40am OFA had their person there, with a chair, with a huge sign to answer questions about provisional ballots. When the poll didn’t open at 6am on the dot, the OFA person was calling at 6:01am, and actually talking to a live person.

142 I often could not contribute to the Romney campaign from 3 different computers with different OSs. Their Donation shopping cart page malfunctioned due to poor design and programming, informing me in database speak that several fields were empty when they were not. I wrote them and told them to fire the webmaster as he was derelict in his or her duty. This went on for 3 months. I wonder how much money they lost as a result. One guy called me from the Romneycampaign in Boston and told me they were aware of the problem but that I could contribute by phone. Huh? Why not just get the problem fixed?

165 Here in heavily Republican Idaho, the Dems called my wife to remind her to vote. The DEMS! She’s never voted Dem in her life.

600 I worked the war room at headquarters in Boston and ORCA was an epic fail. It was epic fail from the moment we got there and that was clear. We were supposed to be there at 4:30 AM. Only one problem. The garages to the TD Garden were closed and hundreds of cars could not get in until 5 AM. 45 minute wait to drive 2 blocks. That was the first clue. 2nd clue, got in, grabbed breakfast and went to go set up computer. Could not get on internet. That happened all morning. NO connection to the net in the garden even though we were supposed to be “live” the minute we sat down. Some of the 11 swing states we were monitoring never got connected tot he interactive site. When we finally did and the problems in the field started presenting themselves we were supposed toe “solve” them, but so complex that we had no idea what to say. When people’s registration records on their devices did NOT match their paper records of voter lists from the get go, you knew that this was going to be an epic fail. Supposedly this had been tested with up to 3 million users at one time with no problem, but with 800 people in the garden trying to get on the site all at the same time, the servers at the garden thought that the system was being attacked and shut down. It was 11 AM before they figured out how to solved this problem. Talked to many of my poll watchers in Northern Wisconsin and they were ready to do ANYTHING to get Romney elected, but thought that for the most part their day was a colossal waste of time. When people tried to get into the phone system to call in their voters tabulations, they couldn’t do that either. System too busy and couldn’t absorb the loss. Many poll watches were sending in alerts about fraud and people walking in never having registered and allowed to vote on site. Who knows what happened to all those complaints. The “technical” people that were there to help the OCRA volunteers and solve problems had absolutely NO idea what they were doing and couldn’t answer a single question. Most of them were wet behind the ears know nothing college students that could not answer a single question of any kind any time. And they got irritated when it was clear that they were about a smart as the side of a barn door. A bunch of us started going “rogue” and just started telling our field contacts what to do over the phone. We knew that by 2 pm the entire system was a epic failure and that no valuable data had been collected at all and it was clear by 8 PM that Romney was going to lose big. Rich Beeson, national campaign director was pacing in the lawyer bull pen looking like he has going to toss his cookies ….. and he looked this way starting 3 hours before the polls even closed. I loved hanging out with my fellow Romneyites for almost 24 hours but this was the most colossally stupid waste of time and money in the history of presidential politics and if these “kids” that were involved in this thought they were going to work in Washington and save the country, God help us all. If Romney actually knew everything about this entire plan and had seen what we saw, EVERY SINGLE IDIOT that was involved in this project from a development and execution standpoint would have been fired !!!!! I hope all of them have to stand in the unemployment line and eat crow because not one of them deserve a job.

Long story short, cant run a campaign, cant run the country…