FROM:

Teresa Hommel

Chair, Task Force on Election Integrity, Community Church of New York

TO:

New York State Board of Elections

40 Steuben Street

Albany, NY 12207-2108

Comment on Draft Voting Systems Standards

Section 6209.6 Examination Criteria

--------------------------------------------------------------------------------------------------------

HOMMEL SUGGESTION - THREE ADDITIONAL SUBSECTIONS

D. USABILITY TEST MOCK ELECTION

1. After a voting system passes all tests described above in this section, the State Board shall conduct a Usability Test Mock Election on a minimum of four voting systems of the type submitted for certification, which shall consist of the Escrow System and a minimum of three additional systems of the same type.

2. The vendor shall supply, in addition to the Escrow System, three or more additional voting systems without fees to the State Board for the purposes of the Usability Test Mock Election, Professional Hacking Test and Public Hacking Test, and which shall be returned to the vendor after the three tests are completed..

3. A minimum of ten business days prior to the Usability Test Mock Election for any voting system, and five days prior to the selection of participants, the State Board shall post notification of such selection and test on their web site, and shall send notification by mail and email to each county, each political party recognized in the state of New York, and each member of the public who has previously requested in writing to receive such notification.

a. Each political party recognized in the state of New York shall be requested to provide two legal observers and two technical observers for such test, and such observers may participate in any part of such test and may attend any training provided by vendors as part of such test.

b. Each County Board shall be requested to submit the name of their county for the random selection of one county to participate in the Usability Test Mock Election. If no County Board submits the name of their county, the State Board shall use the names of all counties for the random selection procedure provided that no county shall be selected more than once for such purpose.

c. Members of the public shall be requested to submit their names for selection as Test Election Inspectors, Test Voters, and Test Observers. If an insufficient number of members of the public submit their names, Election Inspectors from several counties shall be invited to work as Test Election Inspectors, Test Voters, and Test Observers. Such several counties shall include the county where the test is to be held and all adjacent counties.

d. Notification of the Usability Test Mock Election shall include the following:

1). The day and time for random selection of participants.

2) The method by which members of the public may submit their names for selection as participants.

e. The State Board shall perform a random selection procedure conducted at the offices of the State Board at the date and time announced to select participants in the Usability Test Mock Election including the county to provide Test Staff, and persons to serve as Test Election Inspectors, Test Voters and Test Observers.

f. Any member of the public may attend and observe such random selection procedure.

g. Members of the public who are not selected to work as Test Election Inspectors, Test Voters or Test Observers may attend and observe the Usability Test Mock Election, and one member of the public per voting system may use a recording device such as a Cam Corder to record any and all activities at such test.

4. Test Staff

a. The staff of the County Board of the county selected to participate in the Usability Test Mock Election shall work as the Test Staff for such test.

b. The vendor shall provide the State Board with all training materials and documentation intended for use by County Board staff in counties that will use the vendor’s voting system. The adequacy of vendor training, training materials, and documentation shall be tested to determine whether they are sufficient to enable county staff to achieve independent competence in the conduct of elections, and such test shall consist of evaluation of the competence of the Test Staff in conducting the Usability Test Mock Election.

1) The vendor shall provide the Test Staff with all training in system handling, maintenance, trouble-shooting, service, repair; ballot programming, election day preparation and operation, training of Election Inspectors and Voters, transmittal of precinct tallies to the central tabulating location, post-election tabulation, and all other tasks required for the independent conduct of elections with the voting system.

2) After certification of the voting system, such training materials and documentation used by the vendor in training the Test Staff shall be retained by the State Board as part of the Escrow System and shall be listed in the Escrow Component List.

5. Test Election Inspectors

a. A sufficient number of persons shall be selected by the State Board to work as Test Election Inspectors so that each specific voting system in the test has two Test Inspectors assigned to it.

b. The State Board shall pay Test Election Inspectors the same fee for their training and work as is paid to Election Inspectors in the county where the Usability Test Mock Election is held.

c. The Test Staff shall provide the Test Election Inspectors with all training, training materials, and voting system documentation necessary to perform all tasks required to act as Test Election Inspectors at the Usability Test Mock Election.

6. Test Voters and Test Observers

a. A sufficient number of persons shall be selected by the State Board to work as Test Voters and Test Observers so that each specific voting system in the test has two Test Voters and three Test Observers assigned to it.

b. The Test Staff shall provide the Test Voters and Test Observers with all appropriate training.

7. Procedure for conducting the Usability Test Mock Election

a. The ballot from the most recent federal election held in the Test Staff's county shall be used for the Usability Test Mock Election.

b. The Test Staff shall prepare the voting systems for the Usability Test Mock Election without assistance from any vendor staff or technicians, but may use the training materials and documentation they received from the vendor in the course of their training.

c. The State Board shall create a Deck of Test Ballots with a minimum of 200 ballots with votes to be entered by Test Voters.

1. The Deck of Test Ballots shall comply with the standards specified in “Guidelines for Creating a Deck of Test Ballots” by John Washburn.

2. More than 200 ballots shall be created if needed in order to provide one ballot with each possible combination of votes and to comply with the “Guidelines for Creating a Deck of Test Ballots.”

3. Such ballots shall be printed on paper with one ballot per page and sequentially numbered in large print visible on the face of each ballot. One copy of the Deck of Test Ballots shall be made for each specific voting system to be tested, and the ballots in each copy shall be randomly sorted into different sequence, so that the ballots shall be entered in a different sequence on each specific voting system in the Usability Test Mock Election.

4. Each copy of the Deck of Test Ballots shall be bound into a binder that can lie flat on a table when open. The cover of each such binder shall identify the serial number of the specific voting system on which such ballots shall be entered, and each such binder containing a copy of the Deck of Test Ballots shall be called the Test Ballot Book.

5. The State Board shall provide each pair of Test Voters with one Test Ballot Book with the ballots to be entered on the specific voting system that such pair of Test Voters shall work with.

6. The State Board shall make one copy of each Test Ballot Book, to be called the Observer Test Ballot Log, for use by Test Observers who shall observe the entry of such ballots. On each page an area shall be designated for an observer to manually write the time when the ballot was cast, any malfunction or failure that occurred during the entry of the ballot, the method used to remedy the malfunction or failure, and whether or not the ballot was able to be entered and cast

d. Voting procedure.

1) For Usability Test Mock Elections with Optical Scan voting systems, each Test Election Inspector shall provide each Test Voter with one paper ballot at a time in a manner similar to a real election.

2) For Usability Test Mock Elections with DRE voting systems, each Test Election Inspector shall provide each Test Voter with one smart card (or access code or other item as needed by the DRE voting system) at a time, to enable the voter to enter and cast one ballot in a manner similar to a real election.

3) The two Test Voters assigned to each specific voting system shall take turns entering one ballot at a time.

e. If an Optical Scan voting system is being tested, a minimum of 20 ballots shall be created using each accessibility attachment, and a minimum of 20 ballots shall be created using each minority language required to be used in New York state in any county.

f. If a DRE voting system is being tested, a minimum of 20 ballots shall be entered using each accessibility attachment, and a minimum of 20 ballots shall be entered using the language display for each minority language required to be used in New York State in any county.

g. The actions of each Test Voter shall be witnessed by the three Test Observers assigned to the same specific voting system.

1. The first Test Observer shall visually witness the entry of votes, printout and examination of the voter verified permanent paper record, and the casting of each ballot.

2. The second Test Observer shall use a recording device such as a Cam Corder to make a permanent film with the visual and auditory image of the Test Voter's work.

3. The third Test Observer shall make a handwritten note on each page of the Observer Test Ballot Log to confirm the correct entry of that ballot or to describe how such ballot was entered if entered differently than prepared, and to make notes describing any irregularities or machine malfunction or failure encountered in entering each vote, the method used to remedy the malfunction or failure, and whether or not the ballot was able to be entered and cast

h. The State Board shall provide light-weight and simple-to-use recording devices for use by Test Observers, and shall retain the recordings so made as long as the voting system is certified for sale and use in the State of New York. Within one week upon receiving a request for a copy of such recording from any member of the public, the State Board shall makes a copy of such recording for such member of the public for a fee not to exceed the actual cost of making such copy.

8. End-Of-Day Procedure

a. An area in the same room where Test Voting is to take place shall be designated as the Central Reporting Location for the Usability Test Mock Election, and shall be supplied by the vendor with the appropriate equipment.

b. After all votes have been entered on each specific voting systems, the State Board shall declare the end of voting.

c. After the end of voting, Test Election Inspectors shall use the procedures they learned from the Test Staff to extract end-of-election tallies and other information from the voting systems, and transmit them to the Central Reporting Location in the same manner to the extent possible as after a real election.

d. The Test Election Inspectors for each machine shall manually count the votes cast on the specific voting system to which they were assigned, and compare such tallies to the tallies produced by the voting system.

1. During such counting procedure, the Test Voters and Test Observers assigned to the same voting system shall work as observers, and one of these persons shall use a recording device such as a Cam Corder to make a permanent film with the visual and auditory image of the counting procedure.

2. Within one week upon receiving a request for a copy of such recording from any member of the public, the State Board shall makes a copy of such recording for such member of the public for a fee not to exceed the actual cost of making such copy.

e. The Test Staff shall use a projector and screen to display the logs extracted from each voting system and shall compare the contents of each such log to the records provided by the Observer Test Ballot Log.

9. Criteria for passing the Usability Test

a. The voter-verified paper record for each ballot and the tallies for each voting system must be 100% accurate.

b. The voting system's election day activity log must be 100% accurate.

c. Every machine in the test must work with no malfunctions or failures.

d. The Test Staff must be able to perform all activities independently in order to conduct the election, including pre- and post-election activities.

e. The Test Election Inspectors must be able to perform all activities independently to facilitate voting.

f. The Test Voters must be able to vote without undue difficulty using every method provided by the voting system including all accessibility attachments and minority language interfaces.

E. PROFESSIONAL HACKING TEST

1. Prior to certification of any voting system, the State Board shall commission a professional hacking test to examine and attempt to compromise the Escrow System.

2. Criteria for passing the test shall be that the Escrow System withstands all attempts to compromise it.

F. PUBLIC HACKING TEST

1. Prior to certification of any voting system and the Usability Test Mock Election for such system, the State Board shall invite any member of the public to participate in the Public Hacking Test as a Test Hacker. All persons who wish to participate as Test Hackers shall observe the training conducted by the vendor for the Test Staff for the Usability Test Mock Election, provided that prior to such observation such members of the public are not an employee of any vendor and sign a non-disclosure agreement.

2. On the day following the Usability Test Mock Election, all voting systems used in that test shall be available for members of the public to act as Test Hackers and test the security of the voting system by attempting to compromise it.

a. County Board staff who worked as Test Staff at the Usability Test Mock Election may serve as Test Staff for this test. If an insufficient number of such Test Staff are available, members of the public who observed the training of Test Staff may act as Test Staff.

b. Members of the public who did not observe the training of Test Staff may act as Test Election Inspectors, Test Voters, and Test Observers as appropriate if needed by the member of the public attempting to hack each specific system.

c. Two Test Observers shall use recording devices such as Cam Corders to make permanent films with the visual and auditory image of all activities performed with each specific voting system at this test.

3. Criteria for passing the test shall be that the voting system withstands all attempts to compromise it.

--------------------------------------------------------------------------------------------------------

REASON FOR NEW SUBSECTIONS

SUBSECTION D. USABILITY TEST MOCK ELECTION

If systems are going to fail, or are unusable by the kind of people who will be county board of elections staff, election inspectors, or voters, it is better to discover this before certification. This is when vendors are most highly motivated to fix the systems or simplify their use. It is best to reveal and correct weaknesses before purchase and rollout of the equipment in an election, which would cause chaos and lawsuits.

Many members of the public distrust electronic voting systems. If the systems work, the public should see this before cynicism about the equipment lowers voter turnout.

To maintain our bipartisan administration of elections, vendor technicians must serve in an advisory capacity only. Vendor technicians must not perform tasks on behalf of bipartisan elections staff. Therefore vendors must fully train sufficient staff appropriate to fully manage the number of systems acquired. The Usability Test Mock Election enables the State Board to determine that the vendor is capable of providing such training.

E. PROFESSIONAL HACKING TEST and F. PUBLIC HACKING TEST

If a voting system is easily tampered, it is better to discover this before certification. This is when the vendor will be mostly highly motivated to fix it.