Voting System Standards: All Form and No Substance

I have been a long time critic of the 2002 Voting System Standards (2002 VSS) and of the 2005 Voluntary Voting System Guidelines (2005 VVSG). In fact, both sets of standards are virtually worthless. There are two reasons for this. First, the requirements enumerated in the standards are, in and of themselves, much too weak for something as vital as administering an election. Second, both sets of standards have an explicit loophole that allows almost all the requirements — weak as they are — to be ignored. This second objection was first brought to my attention two years ago by Howard Stanislevic.

We now have proof that this loophole is used by the labs in order to “pass” systems that don’t meet the standards. In the most recent certification test report submitted to the Election Assistance Commission (EAC), SysTest labs (one of the labs accredited by the EAC and the NIST) recommended certification for a voting system. SysTest recommended the system certification even though their findings showed 79 specific failures to meet the standards.

As you read this article, keep in mind that the standards actually allow 77 of these 79 failures to be ignored!

Last February, SysTest labs wrote its certification test report for a new voting system manufactured by Premier Election Solutions (formerly Diebold). The report listed the 79 problems the lab found during testing. Even so, SysTest recommended the system be certified by the EAC. Section 6 from the test report reads:

SysTest Labs has successfully completed the testing of Premier’s Assure 1.2 voting system. It has been determined that Assure 1.2 successfully met the required criteria of the Federal Election Commission Voting System Standards April 2002.

Based on testing scope and results, as detailed in this report, it is SysTest Labs’ recommendation that the EAC committee grant certification of Premier’s Assure 1.2 voting system.

I sent a letter to the EAC objecting to this recommendation for certification — not because of the 77 “acceptable” failures, but because of the two failures that require denial of certification.

The test report reads like the problem log maintained by VotersUnite! and contains the following 79 “findings”:

1. Functional, Accufeed will not feed ballots (Finding #1 page 38)
2. Doc, Batch Start card is not rejected (Finding #2 page 38)
3. Functional, Batteries are not functioning (Finding #1 on page 39)
4. Functional, BallotStation is not accepting voter access cards (Finding #5 on page 45)
5. Functional, TSX looping with use of AVPM ll (Finding #6 on page 45)
6. Doc, Housing door does not fit over the canister (Finding #7 on page 45)
7. Doc, Provisional votes flow from the BallotStations up to GEMS is not clear (first Finding #4 on page 45)
8. Doc, User's Guide does not list the limitations of the VCE (Finding #8 on page 45)
9. Doc, VCE does NOT save loaded data unless POSITION 1 is loaded (Finding #9 on page 45)
10. Doc, Statement in #1 is incorrect and may need to be removed (Finding #10 on page 45 – note, this item references the following one)
11. Doc, Section 3.1 references out of date (first Finding #1 on page 45)
12. Functional, The AVPM I and II print modules are not operating (first Finding #2 on page 45)
13. Functional, Ballot Station freezes (second Finding #1 on page 45)
14. Doc, Doc does not provide information on Recall/Retain type C (second Finding #2 on page 45)
15. Doc, Installing and Removing the OSAA (Finding #3 on page 45 – note, this finding and other below are not explained)
16. Functional, Build failed, “failed to verify file” (second Finding #4 on page 45)
17. Doc, Section 8.3: Adding and Editing an Audio file (first Finding #2 on page 52 - unexplained)
18. Doc, Section 14.4: Exporting Required Translations (first Finding #3 on page 52 - unexplained)
19. Doc, Section 5.3: Importing Election Data into AIMS (Finding #4 on page 52 - unexplained)
20. Doc, Section 15.1: Importing Text Translations (Finding #5 on page 52 - unexplained)
21. Doc, Section 4.1: Getting Started (Finding #6 on page 52 - unexplained)
22. Doc, Section 4.2: Logging On (Finding #7 on page 52 - unexplained)
23. Info, Section 5.1: Setting Up a New Election in AIMS (Finding #8 on page 52 - unexplained)
24. Info, Section 5.1: Setting Up a New Election in AIMS (Finding #9 on page 52 - unexplained)
25. Doc, Section 5.1: Setting Up a New Election in AIMS (Finding #10 on page 52 - unexplained)
26. Doc, Section 5.3.1: Importing Diebold GEMS Data (Finding #11 on page 52 - unexplained)
27. Doc, Section 12.8: Languages (Finding #12 on page 52 - unexplained)
28. Doc, Section 12.8: Languages (Finding #13 on page 52 - unexplained)
29. Doc, Section 6.1: Removing the Ink Cartridge (Finding #1 on page 52 - unexplained)
30. Doc, Section 6.5.3: Setting the Admin Password (second Finding #2 on page 52 - unexplained)
31. Doc, Section 6.5.1: Printing the Operation Log (second Finding #3 on page 52 - unexplained)
32. Doc, Section 1.5: Assisting the Voter who uses an AT Device (first Finding #4 on page 53)
33. Functional, Audio script does not match text script (Finding #5 on page 53)
34. Functional, Ballot title not voiced on 2nd race screen (Finding #6 on page 53)
35. Doc, Section 1.3: Using the AutoMARK VAT with an AT Device (Finding #7 on page 53 - unexplained)
36. Doc, instructions/procedures are not provided (first Finding #8 on page 53)
37. Doc, Flash Memory Card the Pin # is not provided in the doc (first Finding #9 on page 53)
38. Doc, Japanese text for 'Write-in' different than what was input (first Finding #10 on page 53)
39. Functional, Both Chinese languages have the same text on select button (first Finding #11 on page 53)
40. Functional, Voting Instructions are not translated (Finding #3 on page 53)
41. Functional, Audio is not provided when voting with a UAID device (Not Supported) (second Finding #4 on page 53)
42. Doc, Issues with foreign fonts (second Finding #8 on page 53)
43. Doc, Section 4 - The listed fonts do not cover all Vietnamese characters (second Finding #9 on page 53)
44. Functional, Cannot select the Contrast or Large Text buttons with the Keypad on the R6 (second Finding #10 on page 53)
45. Functional, No visual cue for every error-type audio cue (second Finding #11 on page 53)
46. Doc, Section 12.1.3 of Appendix D is incorrect (Finding #12 on page 53)
47. Functional, 6 and the 4 keys on the keypad do not function as described (Finding #13 on page 53)
48. Informational, dB levels of 113-115 were consistently measured. (Finding #14 on page 53)
49. Functional, A tenth language is not selectable via the keypad device (Finding #15 on page 53)
50. Functional, Report was not found to exist (Finding #2 on page 53)
51. Functional, Error message when attempting to connect to Assure Security Service Host (first Finding #1 on page 59)
52. Doc, Missing information in the documentation on how to create the MDB file (second Finding #1 on page 59)
53. Functional, OSX unit allowed a blank ballot w/no message (third Finding #1 on page 59)
54. Doc, Document does not contain enough info to resolve the errors (first Finding #2 on page 59)
55. Functional, OSX unit failed when attempting to download via modem (first Finding #3 on page 59)
56. Functional, Central Scan application failed to open the system audit log (fourth Finding #1 on page 59)
57. Functional, Error can't connect to the security service host (second Finding #2 on page 59)
58. Doc, (discrepancy is not valid) (second Finding #3 on page 59)
59. Functional, users and roles that were assigned in the Access Control window are lost (first Finding #4 on page 59)
60. Functional, documentation does not specify how the user is notified when the Vote Center is finished loading (Finding #5 on page 59)
61. Functional, Users and Rolls lost that were assigned in the Access Control window (Finding #6 on page 59)
62. Functional, Messages block user from being able to run PCS (Finding #7 on page 59)
63. Doc, Unable to use “Next card” and “Previous card” buttons (Finding #8 on page 59)
64. Functional, Message – “workspace could not be opened” (Finding #9 on page 59)
65. Doc, VCE requires a commercial off the shelf (COTS), (fifth Finding #1 on page 59)
66. Doc, Voter Card Encoder does not have an ENTER button (third Finding #2 on page 59)
67. Doc, Acceptance Test do not match the steps for encoding a voter access card (third Finding #3 on page 59)
68. Doc, Prompts stated in the user’s guide match do not match VCE (third Finding #4 on page 59)
69. Doc, User’s guide does not explain what a master voter access card is (Finding #5 on page 60)
70. Doc, User’s guide does not provide clear and detailed steps on how to overwrite a ballot key (Finding #6 on page 60)
71. Doc, No explanation of how to create a master voter access card (Finding #7 on page 60)
72. Functional, GEMSVoterCardData.txt file disappears (Finding #1 on page 64)
73. Doc, GEMS documentation does not specify how to setup a Blanket Open Primary (Finding #3 on page 69)
74. Functional, Exception error was encountered (Finding #4 on page 69)
75. Functional, 2.0.1 users guide is shown. The installed software is the 2.0.4 (Finding #6 on page 74)
76. Doc, Automark System Installation and Maintenance Guide procedures not adequate (Finding #1 on page 82)
77. Doc, AutoMARK 3010 Poll Worker's Guide, no indication for message "Alert! (Finding #3 on page 82)
78. HW, Unit inoperable; caused loss of data from the operations log (Finding #2 on page 82)
79. Info, Changes in the total printed count in the operations log on the ATS VAT can be caused by abnormal shutdown (Finding #4 on page 82)

Except for the last two findings on this list, there is nothing that would prevent ANY NIST-accredited test lab from recommending this system for certification by the EAC. None of the 2002 VSS requirements violated by the first 77 findings are enforceable requirements!

Some definitions are in order.

• A requirement is a declarative statement of the form "The system shall..." or a declarative statement of the form "The system shall not..." . Two example requirements from the 2002 VSS are:
  • From section 2.2.2.1a of Volume I of the 2002 VSS: To ensure vote accuracy, all systems shall record the election contests, candidates, and issues exactly as defined by election officials;.
  • From paragraph 5.7.2d of Volume I of the 2002 VSS: the telecommunications components of a voting system shall notify the user of the successful or unsuccessful completion of the data transmission.
• An enforceable requiremen t is a requirement such that if the requirement is not met, then the system cannot be certified.

There are almost no enforceable requirements in either the 2002 VSS or the 2005 VVSG. This is because both standards have a conformance exception clause (loophole) which covers nearly every requirement listed in either standard. The exemption paragraph is found in section B.5 of Appendix B of Volume II of the 2002 VSS and the 2005 VVSG . From the 2002 VSS, paragraph B.5 reads [emphasis mine]:

Of note, any uncorrected deficiency that does not involve the loss or corruption of voting data shall not necessarily be cause for rejection. Deficiencies of this type may include failure to fully achieve the levels of performance specified in Volume I, Sections 3 and 4 of the Standards, or failure to fully implement formal programs for qualify [sic] assurance and configuration management described in Volume I, Sections 7 and 8. The nature of the deficiency is described in detail sufficient to support the recommendation either to accept or to reject the system, and the recommendation is based on consideration of the probable effect the deficiency will have on safe and efficient system operation during all phases of election use.

The first 77 of the 79 uncorrected deficiencies listed above fit within the conformance exemption (loophole) defined in both sets of standards. Some of these 77 violations are also violations of federal law, and yet they are allowed by the standards. For example:

• Findings 33-49 (in my list above) demonstrate that the system under test fails to conform to the accessibility requirements for blind voters and the accessibility requirement for some foreign language voters. This means the system violates Section 301(a)(3) of the Help America Vote Act of 2002 (HAVA). See page 3 of the EAC advisory on compliance with this section of HAVA.

  
  

• Non-conformance 72 involves the destruction of an election record. The destruction of an election record used in a federal election prior to 22 months after that election is a felony under Title 42, Chapter 20, Subchapter II, § 1974.

But, since none of these specific 18 non-conformances nor 59 of the remaining 61 non-conformances involve vote loss or corruption of voting data, none of them prevents the lab from recommending certification.

The last two finding involve the corruption or loss of voting data. So, even with the loophole of paragraph B.5, the system cannot be recommended for certification. SysTest erred when it recommended the system for certification, but only because the voting system was so broken that even the B.5 loophole would not allow the system to be recommended for certification.

SysTest’s report demonstrates that with the broad nature of the loophole found in paragraph B.5, there is almost no failure that would prevent a system from being certified by the EAC, or that would have prevented qualification under the previous NASED/ITA program . So, it should come as no surprise to anyone that we read about failure after failure, in election after election, by voting systems “certified” by the NASED Voting Systems Board .

The 2002 VSS and 2005 VVSG are made toothless because of the conformance exemption found in paragraph B.5. But, as broad and eviscerating as the loophole is, it is not a blanket exemption.

The report I examined was written by SysTest about a Premier/Diebold system. But, the problems with the 2002 VSS and 2005 VVSG apply to systems from all vendors, not just Premier/Diebold, and they apply to all of the NIST-accredited test labs, not just SysTest. SysTest found 79 non-conformances to the 2002 VSS, 18 of which also violate federal law. But, because of the loophole paragraph, the system as tested, but containing only the first 77 of the 79 non-conformances, is certifiable.

The problem with the whole national certification program of the EAC is that the program is built on standards that are standards in form only, without substance. Nearly every voting system currently in the field and used in elections since November 2006 was "certified" to the 2002 VSS under the previous NASED/ITA system. Nearly every voting system to be sold to counties and municipalities until 2012 (and perhaps beyond) will be "certified" to the 2005 VVSG.

The labs are quick to state that they do not certify equipment. The EAC is quick to point out that the testing is done by the labs and not by the EAC. For years the voting equipment vendors have claimed that the 2002 VSS and 2005 VVSG are "comprehensive and rigorous". This real world application of the 2002 VSS to an actual voting system by a NIST-accredited laboratory should demonstrate how toothless and ineffective the 2002 VSS and 2005 VVSG are.

As hard as this is to believe, the EAC certification program is a substantial improvement over the qualification program administered by NASED. With EAC certification, it is possible to find out the known deficiencies that are present in the system used to administer your election. How many known deficiencies are lurking within the equipment certified by NASED and currently used in an election near you? You are not allowed to know! Neither are your local election officials.

Certification test reports, listing the deficiencies of the systems certified by NASED, will never be published. This is because the reports are considered trade secrets by the vendors. The reports are also considered trade secrets by the ITA testing labs that tested the systems. Most importantly, the reports are also considered trade secrets by the chairman and every member of the NASED Voting Systems Board.

This means that systems with known deficiencies certified by NASED were sold to unsuspecting counties and municipalities. With the old NASED/ITA program, these local election officials were left in the dark and never had the opportunity to learn of known deficiencies before they used the voting systems in elections. Instead, they get to learn of the deficiencies as the systems fail during the administration of real elections in the real world.

At least, under the EAC certification system, local election officials have a chance to learn how their voting systems will fail before they use the voting system to administer a real election. This is how the certification process has “improved” under the EAC.

Is this sufficient improvement? Certainly not. Listing the known deficiencies in obscure, technical reports on a government website has some small benefit. Better still, don’t certify systems that violate the standards. Under the standards though, the NIST-accredited labs can recommend certification and the EAC can certify systems which violate most of the requirements of the standards. In my opinion the EAC should not certify such systems nor should the NIST-accredited labs make such a recommendation. But, the standards as written and enforced almost never prohibit certification of a voting system.

The central problem is that both standards (the 2002 VSS and the 2005 VVSG) allow lax and probably uneven enforcement of nearly all of the system requirements listed in the standards. This is because the loophole created by paragraph B.5 is nearly without limitation, moreover, paragraph B.5 provides no meaningful guidelines on when to enforce and when not to enforce a system requirement found in the standard. It is possible the same problem may be found in two different systems and one system is certified and the other is not.

Until this core loophole in paragraph B.5 is closed:

1. The standards do not prohibit the NIST-accredited labs from recommending the certification of a system that violates virtually any combination of requirements listed in the standards.

   
   

2. The standards do not prohibit the EAC from certifying a system that violates virtually any combination of requirements listed in the standards.

As long as a voting system does not lose or corrupt vote totals, the voting system can violate any or all other requirements listed in the standards and still be certifiable.

This is why the “standards” (2002 VSS and 2005 VVSG) used to test and certify voting systems are all form and no substance and will be without substance for the next several years.