Voting System Qualification Test Report Dominion Voting Systems, Inc.
Contents
1. Voting System Qualification Test Report Dominion Voting Systems Inc Democracy Suite Release 4 14 17 Version 3 May 2015 FLORIDA DIVISION OF i ELECTIONS g at 1 Florida Department of State Division of Elections R A Gray Building Room 316 500 S Bronough Street Tallahassee FL 32399 0250 Contents Executive SUIMIMALY ss sciccsssccscesensedes sencdeceeaciscies vance erase eoero E aE aiii 3 Introduction System OVervieW ccssccecssccesscecsseeecssececssecesseecssseecsaececseecessecesaeeseaaeceeseeceseeeeaeeseaaeceeseeens 4 BCO ii A A A a de 4 System KONE ET EEE KE Ne 4 Components under Review rrrnnnrnrsrnnrnsnnnnnnrnvvrnssnsnnnnnnrnnsrnssnsnannnnrnnnrnssrs nar nn nnnrnssnnnnnnnnnnnenssnannnnnnnnnnssee 6 Conduct of Tests Findings cccccccessceesseceescecesseecssececsseceseecssecesaeceessecesseecsaeeeeaaececseecsseseesuseeeaaecesees 6 Physical AUdituursa gadd AA od 6 Functional System Audit PPPPPOONON O DU 0 PC O AaS TENEAN AASE AAAA 7 Voting Equipment Menus Administrative and Diagnostic Reports cccononccooncnnnnnconononanonancnnnns 7 Mock Electo 7 Mass Ballot Count ICE Precinct Tabulator ccccccccccecsssessssececececsssesssaeseeceseessesesaeeeeeeseessesnsaees 8 Accessibility Sound Pressure Level ccccesssssccecececessesnsecececeseesesaeaeeeeeceseeaaaaeeeeeceseessaaeaeeeeeens 9 Contest RECOUNTS s sies dacec cee canci n raida de dan2. 4 The vendor s TDP specifies the following COTS headphones Sony MDR G45LP headphones Democracy Suite Florida Overview Manual or Cyber Acoustics ACM 70 or equivalent Component Version List The vendor supplied AVID FV 060 headphones for testing which are equivalent to the COTS headphones specified Florida Department of State Voting System Qualification Test Report Division of Elections DVS Democracy Suite Release 4 14 17 Version 3 Findings BVSC found that the ICE complies with the applicable statute The results of the sound pressure level tests for the ICE are in the table below Table 3 Sound pressure level test results ICE Sound Pressure Level Test Results ICE Average Maximum Average Minimum Intermediate Level Volume Volume dBA dBA dBA Right Headphone Left Headphone Contest Recounts BVSC conducted a recount to verify compliance with section 102 141 7 F S and the FVSS BVSC selected one countywide race and one district wide race in the general election The recount was conducted using an ICE dual monitor Per the TDP instructions BVSC suppressed non recount races in the EMS results software Findings BVSC found that the voting system complied with applicable statutes and standards Democracy Suite allows the user to report results from only the affected races Furthermore the system permits a recount on more than one race at a time as demonstrated by processing both the countywide race and d
3. 529 610 Florida Department of State Voting System Qualification Test Report Division of Elections DVS Democracy Suite Release 4 14 17 Version 3 Findings The ICE met the acceptance criteria for the precinct scanner mass ballot count as shown in table below Table 2 Acceptance criteria for ICE ICE Mass Ballot Count Acceptance Criteria Expected Accepted Did the memory registers overflow No v Did the public counters increment appropriately Yes Y Did the tabulated results agree with predetermined vote totals Yes Y Number of errors must not exceed 1 in 1 000 000 vote targets An error lt 1 1M vote Y is defined as a target scan that produces a result other than the expected targets result Number of multiple feeds must not exceed 1 in 5 000 ballots A multiple lt 1 5K ballots v feed occurs when the machine pulls multiple ballots and does not catch the error Number of incorrect rejections of ballots must not exceed 3 lt 3 total ballots Y Accessibility Sound Pressure Level The sound pressure level test is conducted to verify conformance to section 101 56062 1 g i F S which describes the sound pressure level standards for a voting system s audio voting features BVSC conducted a sound pressure level test on the ICE because the vendor upgraded the firmware since the previously certified version BVSC used an ITU T P 50 test signal which it incorp
4. documents the recommended marking devices in its user manual and other documentation and the use of appropriate marking devices will positively influence the scanner s ability to read vote targets properly marked vote targets will also contribute toward minimizing ballot misreads and ambiguous mark messages Battery Life BVSC verified that the ICE dual monitor setup when disconnected from the electrical outlet functioned for the time period stated in the TDP documentation The documentation states that the battery pack is capable of powering the system for a period of two hours under normal operating conditions 100 voters per hour and two ADA voters per hour 2 BVSC powered up the ICE using electrical power with the dual monitor attached but no accessible voting session initiated The ICE was set to Standard voting mode with the battery fully charged Once the ICE was ina ready to cast ballots state BVSC disconnected the ICE from electrical power and left the unit running on battery power until the unit drained the battery and it performed an automatic controlled shutdown operation BVSC reconnected the AC adapter and reviewed the audit logs to determine the length of time the machine remained in a usable state before complete drainage of the battery power occurred No votes were cast during the battery interval The table below reflects the outcome of this test Table 4 Battery life test results Battery Li
5. per ballot length BVSC cast ballots into the ICE and compared the results Findings The ICE accepted all ballots except a 17 inch ballot and two 19 inch ballots with folds through the write in vote targets The ICE misread the 17 inch ballot and the two 19 inch ballots were interpreted as ambiguous marks The ICE s behavior in these test cases is considered an expected result since folds through a write in are outside of the vendor s ballot printing specifications In other words ballots with such folds would be considered not meeting the vendor s requirements for proper ballot preparation Users of the ICE are cautioned to implement quality controls to ensure ballot preparations comply with the vendor s printing specifications Scanner Sensitivity Although no requirements exist for this test under Florida law and FVSS BVSC subjected the ICE to scanner sensitivity testing BVSC obtains results solely for information purposes The purpose of the scanner sensitivity test is to observe the scanner s ability to read marks made by various types of marking instruments pens pencils highlighters etc The sensitivity test also demonstrates the scanner s ability to detect a marking when the vote target is not fully or properly marked such as Y O etc BVSC created the test decks by marking the first position on blank ballots with various marking instruments The test deck included two ballots for each marking instr
6. HAVA provisions for accessibility voting The voting system consists of an election management system EMS two types of optical scan precinct count tabulators the ImageCast Precinct ICP and the ICE and a digital scan central count tabulator the ImageCast Central ICC The certification test activity limited its scope to testing of the ICE precinct scanner as no other voting system components were modified from the previously certified version Version 2 The Bureau of Voting Systems Certification BVSC conducted the certification qualification testing in two phases Phase consisted of a physical audit to verify the setup configuration of the EMS the restoration of elections and a functional audit Phase Il comprised the conduct of mock elections and all election cycle events BVSC conducted specific tests during this phase to verify compliance with standards for sound pressure levels and to observe ballot sensitivity BVSC also conducted other tests to verify compliance with Florida Voting System Standards Florida Statutes and Rules and HAVA as well as general observations of the voting system BVSC also conducted a mass ballot count on the ICE precinct tabulator Phase Il was a noticed event open to the public Qualification test results affirm that the Democracy Suite Release 4 14 17 Version 3 met applicable requirements of the Florida Voting Systems Standards Florida Statutes and Rules and the Help America Vote Act HAVA for usab
7. and Administrative Rules Mock Elections BVSC conducted mock elections incorporating multiple card ballots of varying ballot lengths 11 inch to 19 inch BVSC used two election types primary and general The tests included both hand marked and machine marked ballots as well as those cast via the accessible voting session on the ICE and used single card and multi card elections BVSC simulated an election using the ICE from initial preparations and Logic amp Accuracy pre election activities through voting election activities election night and precinct level reporting post election and reporting activities BVSC used ballot test decks and pre determined results to compare to actual results Pre Election Activities Pre election activities included coding or verifying the coding of the election database for the two elections primary and general preparing the election media preparing the ballot test decks preparing and validating the expected results and preparing the voting equipment The primary election definition included a universal primary contest UPC Election Activities Election activities included opening polls casting ballots using test decks including hand marked ballots and ballots marked using the ICE dual monitor setup and closing polls Post Election and Reporting Activities Testing included uploading and verifying election results in the EMS by election group absentee early voting Election Day an
8. coeds vaacecde rae E aE E Aa E Aao E PASE E E SEAE 10 Modems rerne eor A A e eE a e a O Eei Eain 10 Folded Bt 11 Scanner SENSITIVI ci A et 11 Battery bear 12 ICE Dual Monitor aaret 12 IR Security OSO tettet 13 Source Code REM Wise ads 13 Continuous Improvement RECOMMENAATIONS cccsscceescecssececssecesscecesseecsaeeecsaeceeseecsstecesaeeeeaeeenees 13 CONC Mii A ee 13 PAD DEN GICES aa estate 14 Vel ao aa coco A A OEP A E 14 Figures Index aaret Tree EA 15 TADIES INdSX sunket eek 15 Florida Department of State Voting System Qualification Test Report Division of Elections DVS Democracy Suite Release 4 14 17 Version 3 Executive Summary Dominion Voting Systems Inc DVS submitted an application requesting Florida voting system certification of Democracy Suite Release 4 14 17 Version 3 which is a modification to a previously certified voting system Democracy Suite Release 4 14 17 Version 2 approved on April 9 2014 DVS request included two modifications to the previously certified system a the firmware of the ImageCast Evolution ICE precinct tabulator added a Dual Display feature that allows for concurrent voting sessions for Americans with Disabilities ADA and non ADA voters and b the list of compatible modems added a new cellular modem for use with the ICE s results reporting The Democracy Suite voting system is a paper based voting system with an element for compliance with Help America Vote Act
9. d Standards Conduct of Tests Findings The test objective included the examination of the ICE firmware which added a Dual Display feature that allows for concurrent voting sessions for Americans with Disabilities ADA and non ADA voters The test objective also included the examination of a new modem the Verizon UML295 cellular modem to be added to the list of modems currently certified for use in the ICE tabulator Physical Audit BVSC conducted a physical audit to verify that the voting system under test matched the specifications described in the application and the Technical Data Package TDP documentation Findings BVSC found no discrepancies with the setup of the Democracy Suite voting system configurations or with the firmware update on the ICE precinct tabulator Florida Department of State Voting System Qualification Test Report Division of Elections DVS Democracy Suite Release 4 14 17 Version 3 Functional System Audit BVSC conducted a functional system audit to verify that all components of the voting system operate as described in the TDP Voting Equipment Menus Administrative and Diagnostic Reports BVSC performed a functional audit by testing all available menu options and administrative reports as well as systems functions in the course of testing Findings The system performed as indicated in the vendor s TDP and in accordance with the Florida Voting Systems Standards FVSS Florida Statutes
10. d provisional voting BVSC uploaded results directly for the primary election and used modem communications for the general election Results were compared against expected results Post election activities included generating reports Florida Department of State Voting System Qualification Test Report Division of Elections DVS Democracy Suite Release 4 14 17 Version 3 Findings The voting system performed as indicated in the vendor s TDP and in accordance with FVSS Florida Statutes and Administrative Rules Mass Ballot Count ICE Precinct Tabulator Because the vendor updated the firmware of the ICE since the previously certified version BVSC had to conduct a mass ballot count of the ICE precinct tabulator BVSC performed the mass ballot count on the ICE using a 2014 primary election definition The minimum requirement was a ballot count of 9 900 ballots on a single scanner The test deck contained 1 006 one card ballots Staff ran the test deck through the ICE 10 times for a total of 10 060 ballots Specific details follow Table 1 Mass Ballot Count details for ICE Election definition used Miami Dade County 2014 Primary Election Ballot length 17 inches Number of scanner units used 1 Number of test decks 1 Number of runs per test deck 10 Number of ballots per deck 1006 Number of cards per ballot 1 Total number of ballots cast 10 060 Total number of vote targets
11. ed on a ballot box Figure 1 ICP only Figure 1 ImageCast Precinct tabulator ICP ImageCast Evolution ICE is a precinct level optical scanner with ballot marking capability audio voting using the Audio Tactile Interface ATI and a tabulator connected to a ballot box Audio accessible voting may be accomplished on the ICE via the main monitor as shown in Figure 2 or through an external or dual monitor attached to the main unit Figure 3 The dual monitor setup allows an accessible voting session to occur while the unit is in standard operating mode and scanning paper ballots Figure 2 ImageCast Evolution ICE with ballot box Figure 3 ICE with secondary dual monitor for accessible voting Florida Department of State Voting System Qualification Test Report Division of Elections DVS Democracy Suite Release 4 14 17 Version 3 e The central count scanners are the ImageCast Central ICC tabulators Figure 4 These systems use commercial off the shelf COTS hardware with Dominion software Figure 3 ImageCast Central tabulator ICC model Canon DR X10C shown Components under Review This review included only test activities to verify that the modifications to the ICE firmware functioned as indicated by the vendor as well as regression testing to ensure that the change did not affect the voting system s previously verified compliance with applicable Florida Statutes Administrative Rules an
12. fe Test Results Battery Life per TDP Actual Battery Life ICE dual monitor setup 2 hours 5 hours 15 minutes Findings The battery packs powered the ICE dual monitor setup for longer than the stated minimum battery life It is expected that in a real world scenario in which the unit would be utilized during the battery interval battery power would drain quicker relative to the number of ballots processed ICE Dual Monitor BVSC examined the ICE with the dual monitor functionality BVSC verified that the ICE dual monitor setup functioned as expected Functions observed included serial and simultaneous voting ballot casting using main and secondary monitors and external monitor failure as well as other tests 12 2 08 ImageCast Evolution System Operating Procedures Rev 1 1 0 163 February 11 2015 pg 15 12 Florida Department of State Voting System Qualification Test Report Division of Elections DVS Democracy Suite Release 4 14 17 Version 3 Findings BVSC found that the ICE dual monitor setup functioned as expected IR Security Sensor The infrared IR paper sensor is a security feature which when enabled for an election allows only ballots printed on special IR reactive ballot stock to be cast The infrared IR paper sensor contained in the ICE and ICP ICP A detects paper infused with IR reactive elements when paper without the IR reactive elements is inserted as in the case of a fraudulent ballot
13. g Systems Certification DVS Dominion Voting Systems Inc EAC U S Elections Assistance Commission EED Election Event Designer EMS Election Management System FVSS Florida Voting Systems Standards HAVA Help America Vote Act ICC ImageCast Central Count Tabulator ICE ImageCast Evolution Precinct Count ADA Tabulator ICP ImageCast Precinct Count Tabulator LAN Local Area Network RTR Results Tally and Reporting 14 Florida Department of State Voting System Qualification Test Report Division of Elections DVS Democracy Suite Release 4 14 17 Version 3 Figures Index Figure 1 ImageCast Precinct tabulator ICP csccssssecssscecsnscesseeeseseecseeeecsaesesauecseseesseeeessaeessaeeseneeees 5 Figure 2 ImageCast Evolution ICE with ballot box ccccecssscesssecssseeceeeeecseessaeeseseecseeeessaeessaeeseneeees 5 Figure 3 ICE with secondary dual monitor for accessible voting ccsccsssseceeseeceseeesseeessseeseneeess 5 Figure 4 ImageCast Central tabulator ICC model Canon DR X10C SHOWN c ccceecsesssseesseesseeeees 6 Tables Index Table 1 Mass Ballot Count details for ICE oooonocccononocccananoncnonononnnonononononononn corno nn cocoa nn cn rra nn corran nnnans 8 Table 2 Acceptance criteria tor ICE carac iii 9 Table 3 Sound pressure level test results ICE ececssceessneceessaeceeseaaeeesseaaeeeeseaaeeeeseaaeeeeseaaeeeeeeeaes 10 Table 4 Battery life testresultS ainia iniano rior caesa
14. ility and accessibility BVSC therefore recommends certification of the referenced voting system Florida Department of State Voting System Qualification Test Report Division of Elections DVS Democracy Suite Release 4 14 17 Version 3 Introduction System Overview Dominion Voting Systems Inc DVS submitted an application requesting Florida voting system certification of Democracy Suite Release 4 14 17 Version 3 DVS sought certification of a modification to a previously certified voting system Democracy Suite Release 4 14 17 Version 2 Version 2 Certificate 140409 approved on April 9 2014 DVS request included two modifications to the previously certified Version 2 system a the firmware of the ImageCast Evolution ICE precinct tabulator added a Dual Display feature that allows for concurrent voting sessions for Americans with Disabilities ADA and non ADA voters and b the list of compatible modems added a new cellular modem for use with the ICE s results reporting The Democracy Suite voting system is a paper based voting system with an element for compliance with HAVA provisions for accessibility voting The voting system consists of an election management system EMS two types of optical scan precinct count tabulators the ImageCast Precinct ICP and the ImageCast Evolution ICE and a digital scan central count tabulator the ImageCast Central ICC The certification test activity limited its scope to testing
15. istrict wide race in one recount Modem DVS has added a new cellular modem to the list of modems that can be used with the ICE precinct tabulator BVSC examined the voting system s ability to report and accumulate results from the ICE using the installed Verizon UML295 cellular modem BVSC uploaded the results from the mock general election via a dial up analog modem Zoom 3095 V 92 external modem as well as the wireless modem Verizon UML295 cellular modem Findings The system functioned as expected 5 Must be greater than 97 dBA decibels weighted Maximum volume minus minimum volume Must be greater than 20 dB 7 Must be between Minimum volume 12 dB and 97 dB 10 Florida Department of State Voting System Qualification Test Report Division of Elections DVS Democracy Suite Release 4 14 17 Version 3 Folded Ballot Although Florida law and FVSS do not require this test BVSC conducted a folded ballot test to simulate absentee ballot processing The objective was to observe the behavior of the ICE precinct count tabulator when it scans folded ballots BVSC used a test deck comprised of elections with four ballot lengths 11 inch presidential preference primary PPP election 14 inch municipal election 17 inch primary election and 19 inch general election Each test deck included different fold types Z fold C fold a fold through a vote target a fold through a write in and the maximum number of folds allowed
16. of the ICE precinct scanner as no other voting system components were modified from the previously certified version Version 2 BVSC conducted the certification qualification testing in two phases Phase consisted of a physical audit to verify the setup configuration of the EMS the restoration of general and primary elections with their required media and a functional audit Phase Il comprised the conduct of mock elections and all election cycle events including loading the tabulators with the requisite media opening polls activities and reports feeding ballots closing polls activities and reports and election night and post election reporting BVSC conducted specific tests during this phase to verify compliance with standards for sound pressure levels and to observe ballot sensitivity as well as other tests to verify compliance of the voting system under test BVSC also conducted a mass ballot count a required test for tabulators that undergo any configuration and or parts modification Phase II was a noticed event open to the public Background The Florida Division of Elections certified Dominion Voting System s Democracy Suite Release 4 14 17 Version 1 voting system on August 14 2013 Democracy Suite Release 4 14 17 Version 1 Revision 1 on January 15 2014 and Democracy Suite Release 4 14 17 Version 2 on April 9 2014 Testing for this iteration of the system Democracy Suite Release 4 14 17 Version 3 occurred in the BVSC te
17. orated into an election definition The test signal replaced the initial sound file normally heard by a voter at the beginning of an accessible voting session The election definition played the test signal as a loop BVSC took all sound pressure level measurements after the elapse of a complete loop to capture instrument readings across the entire loop The test equipment included a Type I IEC 318 Br el amp Kjaer Artificial Ear a Br el amp Kjaer Free field inch microphone Type 4189 and Br el amp Kjaer model 2250 A analyzer BVSC conducted the test using the commercial off the shelf COTS headphones provided by the vendor t ITU T P 50 ITU T is the telecommunication standardization sector of the ITU which is the International Telecommunication Union ITU is a United Nations specialized agency for information and communication technologies The P 50 represents one of their P Series objective transmission standards measures used for testing the transmission quality of artificial voices 2 IEC International Electrotechnical Commission IEC 318 is a measure used for ear simulators as defined in ITU T P Series standards 3 Briel amp Kjaer 2250 Analyzer A hand held analyzer and sound level meter that performs high precision measurement tasks in environmental occupational and industrial application areas The manufacturer calibrated all equipment used in this test within the recommended calibration timeframes
18. rean lecastatcensaviadeaeeaudaceecausdede 12 15 Florida Department of State Voting System Qualification Test Report Division of Elections DVS Democracy Suite Release 4 14 17 Version 3 Florida Department of State KEN DETZNER Secretary of State 16
19. st laboratory in Tallahassee Florida in April and May 2015 System Overview The DVS Democracy Suite voting system election management system consists of the following software applications e Election Event Designer EED a client application that integrates the jurisdiction districts contests and candidate database as the main pre voting phase e Results Tally and Reporting RTR a client application used for integrating election results acquisition validation and reporting e Audio Studio AS a client application that facilitates production of audio files Florida Department of State Voting System Qualification Test Report Division of Elections DVS Democracy Suite Release 4 14 17 Version 3 Application Server APPS a server application for executing processes such as rendering ballots generating audio files and election files etc Data Center Manager DCM a server application used in the back end data center configuration File System Service FSS a Windows service application that helps read and write files on memory cards Election Data Translator EDT an end user application used to export election data from an election project and import election data into the election project The following scanning and tabulating devices and ADA voting device are used with Democracy Suite ImageCast Precinct ICP tabulator is an optical scanner accessible voting device with ballot review The ICP is attach
20. the tabulator rejects it Counties have the option to enable this feature and print their ballots on the IR reactive paper IR security ballot stock or use conventional ballot stock with the feature disabled BVSC regression tested the IR Security Sensor feature on the ICE to ensure that it functioned properly Findings BVSC found that the IR Security Sensor on the ICE operated as expected Source Code Review BVSC examined the source code using the Klocwork source code analysis tool Findings BVSC determined that the source code posed no significant safety security or operational risks Continuous Improvement Recommendations During testing staff encountered no issues that prevent certification of the Democracy Suite Release 4 14 17 Version 3 voting system Conclusion Qualification test results affirm that Democracy Suite Release 4 14 17 Version 3 met applicable requirements of the Florida Voting Systems Standards Florida Statutes and Rules and the Help America Vote Act HAVA for usability and accessibility The Florida Division of Elections Bureau of Voting Systems Certification therefore recommends certification of the referenced voting system 13 Florida Department of State Voting System Qualification Test Report Division of Elections DVS Democracy Suite Release 4 14 17 Version 3 Appendices Acronyms ADA Americans with Disabilities AS Audio Studio BVSC Bureau of Votin
21. ument a baseline ballot with selected ovals fully marked and a test ballot marked with a horizontal line 1 millimeter thick through the center of the target For testing the scanners ability to detect a variety of improperly marked targets BVSC marked the target using one of the vendor s two approved pens Sharpie Fine Point Permanent Marker black BVSC compared these results to a baseline of the same targets fully marked 9 using the same pen 8 Folds through a vote target are outside the ballot printing specification as outlined in the vendor s documentation See Dominion Voting Systems ImageCast Printing Specification Version 1 0 0 23 November 8 2013 p 19 Dominion Voting Systems ImageCast Printing Specification Version 1 0 0 23 November 8 2013 p 19 10 Ibid 11 2 09 ImageCast Evolution System Maintenance Manual Version 1 1 0 128 February 12 2015 p 15 11 Florida Department of State Voting System Qualification Test Report Division of Elections DVS Democracy Suite Release 4 14 17 Version 3 Findings BVSC observed that the precinct and central count scanners are able to detect a wide variety of marks made by several different marking instruments The ICE consistently accepted fully marked ovals but gave ambiguous mark messages for targets with 1 millimeter line markings regardless of the marking instrument used including the vendor s recommended pen Although the vendor clearly
Download Pdf Manuals
Related Search