Washburn's World: Public Comments to the California Top to Bottom Review

Dear Madam Secretary Bowen:

I thank you for this opportunity to make a public comment on the results of the top to bottom review. My name is John Washburn. I am a resident of Germantown, Wisconsin. I have worked as a software tester and in the field of quality assurance since 1994. I currently am certified by the American Society for Quality as a CSQE; certified software quality engineer. It is a certification I have held continuously and proudly since 1998. I have read the documents [1] found on the website of the California Secretary of State and would like to submit the following comments.

I read with fascination the various attack scenarios. Many are elegant applications to voting systems of well understood attack vectors used against other computerized systems. The results are important, disturbing, and must be addressed. But, as disturbing and import as these technical findings are, I do not believe they are the most disturbing information uncovered by the top to bottom review. The most disturbing findings are:

1. The revelation that the systems are inaccessible and, in some cases, present an active obstacle to voting accessibility.

2. The revelation that vendor representations may be fraudulent.

3. The continuing evidence the NASED/ITA model for certification has failed and is not worth the paper it is written on.

4. The continuing evidence that voting systems are defect-dense.

The Systems are not Accessible

The Accessibility Review [2] by Noel Runyan and Jim Tobias is thorough, detailed, and precise in its findings. None of the three systems reviewed meets the minimum accessibility requirements of the Help America Vote Act (HAVA) or the 2005 Voluntary Voting System Guidelines [3] (2005 VVSG). Direct Recording Electronic (DRE) systems compared to precinct based optical scanning are more expensive to purchase, more expensive to test, more expensive to maintain and, by all indications, are more insecure. The justification for why American elections must endure the addition insecurity and expense of DRE systems has been that DRE systems allow disabled voters and voters in language minorities the opportunity to vote privately and independently. This accessibility review refutes this justification in exceptional detail. For the first time, someone has enumerated all of the accessibility requirements of the both HAVA and the 2005 VVSG and objectively tested for conformance. Moreover, under some conditions the DRE system is an active impediment to voting.

If the person is voting in a language other than English and which uses a non-Roman alphabet such has Chinese, the DRE screen does not render characters at all. Even if the translation were well done, it is worthless if the translated text cannot be rendered for display. This is an active impediment to voting by voters in these language minorities, an impediment they would not encounter with a paper ballot which has no trouble displaying non-Roman characters.

If the person has normal vision, normal hearing, and normal upper body strength and dexterity, but is confined to a wheel chair, the DRE system is inaccessible because the forward approach is blocked by the narrow legs of the stand, hard to reach because of the height, and subject to parallax errors. For this class of voters, the DRE either prevents voting or make voting uncomfortably arduous because of the need for a side approach. Since Wisconsin has paper ballots which are tallied by optical scanner or are hand counted, voters who are wheel chair bound can be accommodated with a clip board or a suitably low table. If the polling location has only DRE equipment though, then the DRE equipment introduces a barrier to voting which did not exist before.

Representations of the Systems May be Fraudulent

The Red Team Report for Sequoia [4] by Vigna, Kemmerer, et. al. includes several comments where the properties of the Sequoia Voting System were misrepresented to the security testing team by Sequoia. Section 4.4 and 4.8 are two such examples. Section 4.4 of the security assessment report states:

There is no way to determine which version of the firmware is running on an Edge device. The Sequoia documentation states that the firmware is stored in ROM and that checksum-based mechanisms are used to determine if the firmware has been modified maliciously. However, in reality there is no secure, hardware based mechanism to ensure that no corrupted firmware gets loaded and executed. In addition, the Edge firmware is stored on a flash memory card and can be easily overwritten. Hardware support for trusted software execution and the use of non-writable memory would protect the Edge device from a large range of attacks from both insiders and outsiders.

Section 4.8 of the security assessment reads:

In the documentation ([10], p. 3-1), it is stated that: "WinEDS currently does NOT utilize code outside of MS SQL Server and no connections or permissions are required on the server (besides SQL Client.) The lack of server access by individual users provides the application with a secure client-server environment. The election data stored on the server can only be modified by authorized users only through the application."

Unfortunately, this is not true. In fact, it is possible to connect to the database and completely compromise the MS SQL server host without using the WinEDS application. This is achieved by exploiting two security problems. First of all, the WinEDS access control procedures can be bypassed. Second, the MS SQL server delivered with the Sequoia system enables users to execute arbitrary commands.

The emphasis of the quoted sections above is mine and highlights the diplomatic language of the assessment team. The representations of section 4.8 were made by Sequoia to the Wisconsin State Elections Board during the May 16, 2007 of the Elections board. This indicates the misrepresentation by Sequoia Voting Systems on the security of WinEDS is consistent.

Another consistent misrepresentation is that the firmware of the system is in read-only memory (ROM). Instead the security team found the firmware is stored on EEPROM/Flash memory. Flash memory is the same type of memory used in a portable flash drive or an iPOD. Read-only memory is just that; read-only. Once created the contents cannot be re-written, but can only be read. While flash memory retains its contents when the power is off (non-volatile), it can be re-written (mutable). Read-only memory is both non-volatile and immutable. Flash memory is easily changed and therefore highly insecure.

Both of these representations (ROM based firmware and secured SQL architecture) are false. Since, I am not an attorney, I cannot judge whether such false representations constituted fraud. But, the misrepresentations are fundamental and hard to classify as anything other than an effort to deceive.

The NASED/ITA Testing Model has Failed

The security reports as a whole present more evidence that the NASED/ITA framework for testing and certification has been an utter failure. This is a significant problem which impacts the whole country. The NASED/ITA model was used to as the basis for the certification of EVERY voting system currently in use in the United States. With the exception of lever machines in New York, only equipment qualified by the NASED/ITA process was used in the most recent Federal election held on November 7, 2006. That this testing and certification model is ineffective and flawed is a concern for the State of California and every other state where NASED certification is a requirement to state certification

The NASED/ITA testing framework failed to find any of the findings of these three reports during repeated rounds of testing conducted over the course of several years. The results of these three reports from the Top-To-Bottom Review on the other hand were all uncovered in less than one month of examination. Each finding in the security reports is evidence of the failure of the NASED/ITA process. For illustration I will focus on only two of the findings from the Sequoia security assessment. The NASED/ITA testing and certification system failed to find:

· There is no way to determine which version of the firmware is running on an Edge device. Section 4.4 of the Sequoia Security Assessment Report.

· The Edge firmware was discovered to include a shell-like scripting language interpreter. Section 4.5 of the Sequoia Security Assessment Report.

The inability to identify the system under test is a violation of Section 8.6.d, Volume I, Section 8.7.1, Volume I, and Appendix B.3 Volume II of the 2002 VVSG.

Section 8.6.d Volume I states:

The vendor shall establish such procedures and related conventions, providing a complete description of those used to:

a. Perform a first release of the system to an ITA;

b. Perform a subsequent maintenance or upgrade release of the system, or a particular components, to an ITA;

c. Perform the initial delivery and installation of the system to a customer, including confirmation that the installed version of the system matches exactly the qualified system version; and

d. Perform a subsequent maintenance or upgrade release of the system, or a particular component, to a customer, including confirmation that the installed version of the system matches exactly the qualified system version.

Section 8.7.1 Volume I states:

Physical Configuration Audit

The PCA is conducted by the ITA to compare the voting system components submitted for qualification to the vendor’s technical documentation. For the PCA, a vendor shall provide:

Identification of all items that are to be a part of the software release;

Section B.3 Volume II (System Identification) states:

System Identification

This section gives information about the tested software and supporting hardware, including:

a. System name and major subsystems (or equivalent);

b. System Version;

c. Test Support Hardware; and

d. Specific documentation provided in the vendor's TDP used to support testing.

Since, "There is no way to determine which version of the firmware is running on an Edge device", it is not possible to meet any of these three requirements of the 2002 VVSG. How was this failure to conform missed by the vendor funded test labs during repeated rounds of testing? Paul Craft, Steven V. Freeman, and Britt Williams of the technical subcommittee of the NASED Voting Systems Board reviewed every report generated by the vendor funded ITA labs. How is it that they failed to notice that the labs were not testing for conformance to the system identification requirements? One possibility is that these three granted a waiver to Sequoia Voting Systems on the matter of conformance to standard. Such waivers to conformance are permitted by Appendix B.5 Volume II of both the 2002 and 2005 VVSG. The relevant paragraph of Appendix B.5 of the 2002 VVSG reads:

Of note, any uncorrected deficiency that does not involve the loss or corruption of voting data shall not necessarily be cause for rejection. Deficiencies of this type may include failure to fully achieve the levels of performance specified in Volume I, Sections 3 and 4 of the Standards, or failure to fully implement formal programs for qualify[sic] assurance and configuration management described in Volume I, Sections 7 and 8. The nature of the deficiency is described in detail sufficient to support the recommendation either to accept or to reject the system, and the recommendation is based on consideration of the probable effect the deficiency will have on safe and efficient system operation during all phases of election use.

As the security assessment report states, interpreters are prohibited by the 2002 VVSG. Again, how is that the vendor funded ITA labs failed to notice the presence of a prohibited interpreter during any of several rounds of testing? The problem for California on this matter is more acute. In December of 2005 it became public knowledge that the voting systems from Diebold Election Systems Inc. used prohibited interpreters and interpreted code. In response, Bruce McDannold, Interim Director of the Office of Voting System Technology Assessment, specifically asked Paul Craft and Steven V. Freeman if there were any other voting systems used in California which also had interpreters and interpreted code. In this email exchange [5], Mr. McDannold states that some think the State of California is "picking on" Diebold over the interpreted code issue. At the time Mr. Craft and Mr. Freeman stated no other voting system used in California used interpreters or interpreted code. It is ironic that the security assessment team has vindicated Diebold Election Systems. There were two voting systems in California using interpreters, but only Diebold was singled out for investigation.

Paul Craft and Steven V. Freeman are 2 of the 3 people on the technical subcommittee of the NASED Voting Systems Board. How is it they were unaware of the interpreter found in the Edge voting systems from Sequoia? Mr. Craft and Mr. Freeman were hired because of their connection with the NASED process and their expert knowledge of voting systems. The State of California specifically and directly asked both Mr. Craft and Mr. Freeman about interpreters in California Voting Systems. They stated Diebold was unique. Mr. Craft and Mr. Freeman failed the State of California when they provided this incorrect answer. One may ask what other work product from Mr. Craft and Mr. Freeman may also defective.

The Systems are Defect-Dense

Over the years, every time a vendor-independent team investigates a voting machine the team finds new, significant, and possibly election-altering defects. An incomplete list of these past studies is:

· The 2003 John Hopkins report,

· The 2003 RABA report from Maryland,

· The 2003 Compuware report from Ohio,

· The 2004 follow ups reports by Compuware to the initial 2003 Ohio report,

· The 2005 examinations by Hugh Thompson in Leon County,

· The 2005 examinations of Hari Hursti in Leon County, Florida,

· The 2006 examinations by Hari Hursti in Emery County, Utah,

· The 2006 Princeton report on the TSx,

· The 2007 report from the University of Connecticut on the AccuVote OS, and

· The 2007 report from the University of Connecticut on the AccuVote TSx.

California’s three new security assessments again find new and significant defects which are distinct from those found in prior reports. In my expert opinion this indicates that the software in these systems is defect-dense. A defect-dense system has a high number of defects per thousand lines of code. Defect-dense systems are marked by the same properties as exhibited by voting systems:

1. Different testers find different defects. In defect-sparse systems, different testers tend to find the same defects over and over. This is because there are so few defects to find that effective testing by different groups repeatedly finds the few defects present.

2. The defects found are generally severe. This is because severe defects are usually found before minor defects. Major defects are easier to detect because the behavior is manifestly incorrect and major defects tend to hide or obscure the presence of more minor defects.

Consider a line of automobiles from the fictional manufacturer Washburn Motors. What if every time a mechanic or engineer not hired by Washburn Motors examines one of my cars, they find a new, serious problem? One mechanic finds the engines stalls at 60 miles per hour. A second discovers the axles tend to break. A third notices the brakes fail intermittently in warm weather. A fourth discovers the odometer sometimes loses 18,000 miles. Would you by a car from Washburn Motors? Most would not. This is because even though they do not use the term defect-dense, most people instinctively recognize the symptoms and would avoid buying a lemon from Washburn Motors.

Voting systems currently exhibit the same behavior as the fictional cars from Washburn Motors. Every time someone not hired by the manufacturer examines the product, they find new, serious problems.

Conclusion

Secretary Bowen you face some hard choices which must be made in a short time frame. I wish I could offer more than the following suggestions.

1. Do not rely on the results of the NASED/ITA model. It has failed and the certifications issued under the program is not be worth the paper they are written on. I would urge the Secretary to consider creating a multi-state testing consortium. This idea was first presented to the state by Eric Lazarus during the Voting Testing Summit sponsored by the State of California in 2005. His paper is found here [6] is entitled: "A Vision for the Testing of Election Systems in a HAVA World". An expansion on the framework proposed by Mr. Lazarus is found here [7] and is entitled: "Testing Election Software Effectively". I have misgivings that the EAC/NIST/VSTL model currently under construction is little more than the NASED/ITA model with different acronyms.

2. To the extent possible limit the expansion of this unreliable and inaccessible voting technology. Consider technology which actually expands accessibility such as non-tallying ballot marking devices (e.g. Automark or Vote-PAD) or systems which print ballots on demand under the direction of voters. Expanding the franchise to those with disabilities or who are in a language minority is goal which resonates with the deepest aspirations of the American ideal. We should select technology which is both appropriate and effective in realizing this ideal.

3. Sequoia was asked by Bruce McDannold in December of 2005 if there were interpreters or interpreted code found on voting systems from Sequoia. What was the company response to this question? The representations made by Sequoia which have been contradicted by the security assessment team must be assessed to determine if those representations constitute fraud.

4. Determine, if possible, whether the non-conformances found by the top-to-bottom review were also found by the NASED/ITA. Testing results are under the NASED/ITA model are consider trade secrets held by the equipment manufacturer. Thus it is possible the reported non-conformances were discovered by the NASED/ITA process and granted waivers, but the disclosure of such waivers has been blocked by the assertion of trade secrets and the enforcement of non-disclosure agreements.

Even if you ultimately decide to use the currently certified systems, continue to vigorously test these systems beyond this Friday. Information acquired late is better than no information. You will need all the evidence and information possible in order to make an informed and prudent decision. I fear any decision you make on these voting systems (keep the certifications as is, decertify all, decertify some, mandate specific procedures, etc.) will deeply anger some segment of the people you have chosen to serve. On this matter I can only offer this advice: It is better to squarely face the uncomfortable truth than accept the comforting lie. Postponement should be avoided. As hard as it is in the immediate good things flow from following the truth and bad things will drown as you hide the refuge of the lie.