Home

Software Qualification Test Plan

1. Polling Place o Load the database for this election o Generate VID cards for this election on this dedicated laptop as may be done in an actual election Cards are generated on demand as voting occurs At FFVT o Load election database and set VID Laptop location to match the FFVT location o Vote ballots including Chinese Spanish provisional regular ADA unit qualified write in verify incorrect voter actions including time out response for fleeing voter o Close poll and export tally o Change FFVT name to simulation a different polling place and repeat 3 above steps ooo O 0O O Or O In the Tally application o Import FFVT tabulations o Resolve provisional ballots o Import the OVT tabulation O Tally and print reports and verify accuracy of vote totals CIBER Version 1 2 April 10 2006 Proprietary Summary of Report and Addendums The first two entries in this summary identify the components that were certified with a predecessor system that contained EMS and OVT components that were modified and included in the VOTE TRAKKER 1 2 0 Voting System The VOTE TRAKKER DRE hardware in 1 2 0 is completely new The configurations of the predecessor system that had been certified versions 4 7 5 and 4 7 6 provide the history of the EMS and OVT components of VOTE TRAKKER 1 2 0 Original Report Created 02 23 04 Software Hardware Source Code Functional Test VOTE Optical Vote Trakker 1 5 0 COTS Canon Manage Ballot Date R
2. the text needs to be clarified Users should not be shown options that they are not supposed to use Vol Il Sec System Operations Procedures R2 0 Maintenance and Repair Manual 2 8 6 a Operations Support Version 1 05 has been received R1 0 1 Maintenance and repair The vendor shall provide manual is referenced but not yet documentation of system submitted operating procedures that meets the following requirements a Defines the procedures required to support system acquisition installation and readiness testing these procedures may be provided by reference if they are contained either in the system hardware specifications or in other vendor documentation provided to the ITA and to system users and CIBER Version 1 2 April 10 2006 Proprietary 21 2002 Reqt Description Ciber Comments Avante Comments Vol Il Sec 2 10 1 b b System operations for voting system functions performed at the polling place R2 0 1 Figures are now labeled correctly in Ballot Loading and Pre Election Manual Ver 5 2 6 2 Ballot Loader and Pre Election Manual has been revised to state 3 VTs 3 Described in Polling Official s Manual Version 1 05 Section 3 page 6 R1 0 1 Figures 4E and 4F on page 14 of the Ballot Loading and pre election manual are mislabeled 2 The ballot Loading and pre election manual says that no more that 4 VTs can be plugged into a single circuit The Polling Officials Manual says no more
3. OO OOO O e Optical VOTE TRAKKER OVT version 1 7 0 o Optical VOTE TRAKKER 1 7 0 o Count Ballots 2 00 4 Hardware Ciber utilized the following hardware in the system test configuration e AI Technology VOTE TRAKKER Model EVC308 FF Voting Machine Firmware version 1 2 0 Serial Number V TF0300108 e Canon Optical Scanner Model DR 5020 Serial Number BR309526 CIBER Version 1 2 April 10 2006 Proprietary 4 e AI Technology Magnetic Card Reader Writer Model ACR8000RF The Vote Trakker unit was located at the Wyle facility and was an actual unit that had completed the Wyle hardware and functional tests The Optical Scanner used was located at the CIBER facility and is the unit that was certified by CIBER in the previous submittal by this vendor That unit has been continuously located at the CIBER facility since the completion of those tests All firmware source code was reviewed by the CIBER code reviewer COTS Software e MS Windows 2000 operating system e Microsoft SQL Server Professional Series Version 7 e MS Word 2000 e Roxio Easy CD Creator 5 e Object WControl v3 1 ActiveX Control JFL Peripheral Solutions Inc File name Obj TWAIN ocx e Barcode deciphering Softek Barcode Version 3 03 SoftekBarcode Control ActiveX Control Softek Software Inc e Language synthesizer engines o IBM Via Voice TTS Runtime v6 610 Chinese www wizzardsoftware com o U S English Lernout amp Hauspie TruVoice engine 2 0 licensed unde
4. 09 09 09 09 09 02 07 07 07 07 07 0 9 09 37 09 43 09 09 09 253 09 09 14 09 09 09 58 09 09 09 09 0951 09 1 0931 0931 09 31 09 31 09 1 09 1 09 1 09 1 09 1 09 1 09 1 16 1 09 1 09 1 091 09 1 09 1 09 51 09 1 09 1 09 1 09 09 09 00 13 10 18 18 1 5 OOO OU O COO OOGO HO O GOO U oO oO O O O O O O O O O CO OS O A BN ON DOO 0 00 O Be Be CON NNINhNOO OO OO JO ONNN O O O OO 46 50 10 54 54 A o o y o G G o A A A A lt L A a a g o g lt a a S AM AM AM AM AM HJA A A PAON N N N D Re Ae CR PEE REEE Re eR Ry Ce ee CR GR UR ORE RRR RR RRR RE eR KR KKK 36 06 jpg 1 KB 6 18 2003 07 07 5pg 1 KB 6 18 2003 07 08 5pg 1 KB 6 18 2003 07 frmPartySelection frm 42 KB 1 24 2006 09 frmPartySelection frx 1 KB 6 8 2005 01 frmVote frm 582 KB 1 24 2006 09 InsertCard avi 29118 KB 3 16 2004 12 LanguageTranslationl028 mdb 1068 KB 7 1 2005 12 LanguageTranslation3082 mdb 1068 KB 7 1 2005 12 mdlDifferentControl bas 3 KB 1 24 2006 09 mdlFontControl bas 47 KB 1 24 2006 09 mdlFullFaceVote bas 10 KB 1 24 2006 09 mdlGenerateBallot bas 40 KB 1 24 2006 09 mdlLongTest bas 46 KB 1 24 2006 09 mdlScheme bas 6 KB 1 24 2006 09 mscal oca 29 KB 8 28 2002 12 MSCAL OCX 87 KB 7 10 1997 11 PaperRecord doc 19 KB 8 15 2005 12 Thumbs db 15 KB 3 23 2005 01 TouchScreenDriver txt 1 KB 9 7 2005 01 Voting org 3 KB 8 9 2001 12
5. KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB 2 17 2 2 17 2 2 17 2 2 17 2 2 17 2 2 17 2 2 17 2 2 14 2 2 14 2 2 17 2 2 17 2 2 17 2 1 24 2 2 17 2 2 17 2 2 17 2 2 17 2 2 17 2 2 17 2 2 17 2 2 17 2 2 17 2 6 9 2 2 17 2 11 10 2 2 17 2 2 17 2 2 17 2 2 17 2 2 17 2 2 17 2 2 17 2 2 17 2 9 29 2 7 28 2 1 14 2 1 14 2 9 29 2 10 3 2 9 14 2 1 21 2 1 21 2 9 14 2 12 21 2 12 21 2 12 21 2 9 29 2 9 29 2 9 14 2 9 22 2 8 1 2 8 1 2 7 29 2 O OF 00 70 0 0 0 OO 0 0 0 gt 0 0 0 OC O O D 0 0 0 0 O 0 00 O O O O O 0 ee 0 0 gt 0 0 O O O O E 0 0 0 0 0 O O T 06 11 06 11 06 11 06 11 06 11 06 11 0 6 T1 04 04 06 11 06 11 0 6 Tass 07 06 11 06 11 06 11 06 11 06 11 06 1 06 11 06 11 0 6 115 05 11 06 11 04 06 11 06 11 UG ls 0 6 ts UG LI 06 11 06 11 06 11 06 06 06 05 05 05 03 03 05 05 05 03 03 05 05 05 05 05 05 05 05 05 05 05 10 TI 06 06 10 05 01 02 02 01 10 10 10 10 10 02 01 07 07 125 NONNNNN UY HOEFENEFENNNNNNNNNONNN BWBNNNN DN DN LDN Ln K D S Ln Ln Ln Ln K A WWW JOR o Ss al NS PwWwwWoBRWEHE Oo ds us to oO H ds ds WwW WO 1 40 1 40 1 40 1 40 1 40 1 40 40 04 06 1 40 1 40 40 28 1 40 1 40 1 42 1 42 42 48 1 42 1 28 1 30 1
6. KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB 12 15 2 12 15 2 12 15 2 12 15 2 12 14 2 12 15 2 12 14 2 12 15 2 9 6 2 12 14 2 12 15 2 12 15 2 12 15 2 12 15 2 12 15 2 12 14 2 12 15 2 12 15 2 12 15 2 12 15 2 12 14 2 12 14 2 6 9 2 12 14 2 11 10 2 12 14 2 12 7 2 12 14 2 12 14 2 12 14 2 12 14 2 12 21 2 12 14 2 3 23 2 12 15 2 12 15 2 12 15 2 7 26 2 12 15 2 12 15 2 12 15 2 12 15 2 12 15 2 12 15 2 12 5 2 12 15 2 12 15 2 12 15 2 12 15 2 12 15 2 12 15 2 7 13 2 12 15 2 11 30 2 12 15 2 12 15 2 O O O O O O OD Oy O O 0 0 10 0 00 00 0 00 0 0 O CO 00 OO OO O 00 0 O OO O O O 0 0 0 0 0 0 R O O O 0 0 0 Oo O 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 09 09 09 09 07 09 07 09 Lis 07 09 09 09 09 09 OPS 09 09 0 9 09 07 07 01 07 06 Os 0 34 07 07 07 07 qd 07 03 09 09 09 08 09 09 09 09 09 09 05 09 09 08 09 09 09 10 09 06 09 09 05 05 05 05 48 Oss 22 05 09 223 05 05 05 05 05 50 05 09 05 058 D2 222 Ass 223 Tis PAR 43 40 40 40 223 30 223 04 05 05 05 50 05 05 05 05 05 05 23
7. U A OA O O O LO OOO O 0 O O 00 o Go US AS Go NS o SS aS US SS Sci is SES 204 249 04 204 204 04 5097 04 04 04 2 08 04 30 04 nodos 04 04 04 04 05 13 10 13 18 18 19 16 20 20 06 45 sie 543 48 08 28 28 28 28 28 28 530 30 00 30 44 54 42 58 12 54 40 56 56 36 56 06 58 58 58 38 58 24 58 08 58 58 58 58 00 40 50 46 10 54 54 58 58 10 14 12 56 08 PU Q 8 0 8 Q Q Q 9 9 UY Uo g O 0 0 0 09 9 9 9 9 9 9 9 9 9 9 Q 9 o 9 9 9 U U WQ Q R R E R R R 32 createdb sql Data mdb drop_proc sql drop_tables sql frmAbout frm frmAbout frx frmKeyIn frm frmKeyIn frx frmTransferBallot frm Insert_Init_Data sql Language Instruction doc LanguageTranslation1028 mdb mdlDatabaseSetting bas mdlErrInfo bas mdlFormControl bas mdlLoadBallot bas mdlMain bas Signature Jpg Tabulation mdb TouchScreenDriver txt Transfer_tables sql TransferTabulation txt VoteTrakker Load Ballot DataPas Manage Ballot Data AUTORUN INF Configure txt CR_MANAGE_BALLOT_DATA MDB create_proc sql Create_tables sql CREATEDB SQL Data mdb drop_proc sql drop_tables sql frmAbout frm frmAbout frx frmKeyIn frm frmKeyIn frx frmManagement frm Insert_Init_Data sql Language Instruction doc LanguageTranslation mdb LanguageTranslation3082 mdb Load_Ballot_Data exe mdlCDR bas mdl
8. and R1 0 Maintenance and repair manual is referenced but not yet submitted Vol ll Sec System Test and Verification Test specifications have not been 2 7 1 Specification Development Test provided ITA shall design and execute Specifications test cases for full Functional Testing In the event that test data is not available the ITA shall design test cases and procedures equivalent to those ordinarily used during product verification Vol ll Sec System Test and Verification Test specifications have not been 2 7 2 Specification Qualification Test provided ITA shall design and execute Specifications test cases for full Functional Testing The vendor shall provide specifications for verification and validation of overall software CIBER Version 1 2 April 10 2006 Proprietary 17 2002 Description Ciber Comments Avante Comments Reqt performance Vol ll Sec System Test and Verification Test specifications have not been 2 7 2a Specification Qualification Test provided ITA shall design and execute Specifications test cases for full Functional Testing These specifications shall cover a Control and data input output Vol Il Sec b Acceptance criteria Some provided ITA will expand and 2 7 2b add Vol Il Sec c Processing accuracy Test specifications have not been 2 7 2C provided ITA shall design and execute test cases for full Functional Testing Vol Il Sec d Data quality as
9. ballots Provisional Ballots After Hour Provisional Ballots e English and Spanish text and voice recording e Voice recording of ADA text phrases e Multi line write in Voting Devices Utilized OVT FFVT Procedures e In Manage Ballot Data application create a database for this election e Generate a Ballot using Generate Ballot Data as follows o Import precincts districts political party configuration by importing the template created in the fnGE01_050930 test above Define levels as State County City Precinct and Ward Use MS Paint to Create Icons for each party and store in party_icon folder Enter contests for each party and some non partisan and setup contest groups Setup cross party voting rules Add candidates to contests and order candidates Enter bond issue with custom text Set to rotate all contests by precinct Set system settings for voting machines voting strategy voter paper receipt format allow voting of any precinct at any machine Add Spanish language as available language Export text phrases to translate to Spanish translate them using CIBER laptop then import translation back o Record Spanish and English text phrases using voice recording in System Settings o Use MS Paint to create signature file for paper ballots and store in Paper_Ballot_Folder 0000000 oo e Preview ballots using the voting test application and modify ballots and system settings as necess
10. 01 03 05 25 05 05 08 05 05 05 583 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 OS 05 On 05 05 05 05 05 05 05 05 00 14 06 31 54 49 53 3 6 02 34 246 56 42 SO 9 47 535 24 36 18 36 38 38 38 38 38 38 40 40 40 40 40 40 40 40 40 40 40 42 42 42 42 42 44 44 44 44 44 46 46 46 48 48 48 48 00 14 56 08 38 26 50 04 54 40 44 26 16 02 HS 9 9 9 9 g g g 9 9 a a g g o o 9 9 9 9 g g 9 g 9 g g g g g 9 g g g g g g domo 29 Tabulation mdb 248 Thumbs db 26 TouchScreenDriver txt 1 Transfer_tables sql 2 TransferTabulation txt 1 Event Log Event LogReport doc 19 frmAbout frm 4 frmAbout frx 1 frmDBConnection frm 8 frmDBLog frm 9 frmDBLog frx 1 frmDocument frm 2 frmDocument frx I frmLogin frm 3 frmMain frm 20 frmMain frx 3 mdlErrInfo bas 7 mdlMain bas 4 mdlPrinter bas 18 VoteTrakker Event LogPassword mdb 76 Generate Ballot Data AllCandidate doc 19 AllContestPosition doc 19 AllContestPositionByDistrict doc 19 AllDistrictPrecinct doc 19 AllPrecinctDistrict doc 19 Default wav 1 ElectionDistrict mdb 120 frmAbout frm 4 frmAbout frx 7 frmAutoSetCandidateOrder frm 41 frmAutoSetCandidateOrder frx ih frmBallotType frm 65 frmBallotType frx 1 frmCandidate frm 88 frmCandidate frx 1 frmCandidateCharacterOrder frm 16 frmCandidateCharacterO
11. 03 04 04 04 04 04 01 04 01 08 04 04 O33 05r 07 05 03 02 HhHotHOgo RR od oO E 04 09 06 10 04 11 LES dois Ass 01 08 04 10 05 04 04 49 24 E 02 246 58 246 58 246 224 5 63 3 20 20 20 222 222 212 242 00 34 SS Los BFP WB DB Ba O OKRA HDD OD 06 258 31 Les 54 49 10 30 202 098 SOT 09 49 DS 24 56 21 01 45 09 09 09 09 49 34 13 48 54 B 01 44 22 38 58 50 04 18 20 20 30 20 46 40 44 56 10 24 14 16 56 08 38 22 50 04 40 26 42 36 42 46 40 28 38 06 42 42 44 44 00 56 10 52 24 26 HS 8 9 9 o g g o a 9 g g g g g 9 g g g g g U Pp Q RQ R G R WQ G Q WQ WQ E R E R R Q Q R Q WQ Q Q E Q 33 DTSReport doc 19 KB 4 15 2005 11 frmAbout frm 5 KB 12 7 2005 11 frmAbout frx 3 KB 10 14 2004 01 frmDBLog frm 41 KB 12 7 2005 11 frmDBLog frx 1 KB 4 6 2005 02 frmDocument frx 1 KB 9 27 2001 03 FrmEnter frm 9 KB 12 7 2005 11 frmLogin frm 6 KB 12 7 2005 11 frmMain frm 12 KB 12 7 2005 11 frmMain frx 3 KB 3 21 2005 12 frmMassCD frm 20 KB 12 7 2005 11 frmMassCD frx 1 KB 7 29 2005 04 frmNoFinishLocation frx 1 KB 2 26 2002 01 frmNoFinishTypeln frx 1 KB 12 11 2001 04 frmNumber frm 26 KB 12 7 2005 11 frmNumber frx 1 KB 4 6 2005 02 frmProgress frm 2 KB 12
12. 04 54 46 26 16 02 42 14 56 52 54 40 56 12 08 10 10 46 48 48 00 12 d O O A O A O AA AAA A R ANA A AN AN Q Q DOY 0 0 0 0 0 0 Q 9 09 Q 0 ppp DOY Y OO Q pp Q 9 0 Q 0 0 W OO DOY 0 0 09 0 0 Q 0 Q Q 0 pp 0 ow d 39 frmDuplicateVID frm frmDuplicateVID frx FrmEnter frm frmLogin frm frmMain frm frmOption frm frmOption frx frmOverVote frm frmOverVote frx frmPaperBallot frx frmPrintReport frm frmPrintReport frx frmProgress frm frmTicket frm frmTicket frx frmUnderVote frm frmUnderVote frx frmVIDValidation frm frmVIDValidation frx frmWriteIn frm frmWriteln frx frmWriteInSummary frm mdlErrinfo bas mdlExportTabulation bas mdlGenerateTicket bas mdlLogIn bas mdlLongTest bas mdlMain bas mdlSetting bas mdlShell bas mdlVote bas mssccprj scc Optical Vote Trakker exe OpticalBallot Image mdb OpticalVoteTrakker PDM Scanner VotingPassword mdb Scanner RES CIBER Version 1 2 April 10 2006 19 10 10 53 2524 116 76 32 Proprietary KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB 2 1 5 1 2 1 2 1 2 1 2 1 7 2 1 2 7 2 7 2 7 2 7 2 9 7 2 2 17 2 7 7 2 6 6 2 2 17 2 7 7 2 27 2 17 2 14 2 12 7 2 2 17 2 7 7 2 2 17 2 12 20 2 2 17 2 7 7 2 2 1 2 1 2 1 2 1 2
13. 05 05 41 05 05 05 373 03 43 05 OS 22 22 22 22 22 54 54 54 54 54 48 54 38 48 54 54 54 54 56 56 56 56 56 36 36 10 38 12 40 24 40 50 42 38 24 26 26 24 26 26 26 28 28 30 10 30 32 22 32 32 32 56 34 08 34 34 A A A m 9 9 99 28 frmSampleBallot frm frmSampleBallot frx frmSetting frm frmTally frm frmTally frx frmTestPrinter frm frmTransferTally frm frmVerifyBarcode frm frmVerifyBarcode frx mdlCDR bas mdlCheckMouseMove bas mdlCheckRawDataFile bas mdlCheckTime bas mdlCheckVID bas mdlColor bas mdlDevice bas mdlDisableLowLevelKeys bas mdlErrInfo bas mdlExportTabulation bas mdlFormMaganemnt bas mdl ImportTabulation bas mdlMachineInformation bas mdlMain bas mdlMsg bas mdlNetworkSetting bas mdlPrinter bas mdlReadDBSetting bas mdlReceiptNumber bas mdlRecord bas mdlShell bas mdlSimpleBallotStyle bas mdlSpeak bas mdlUserManagement bas mdlVerifyProvisional bas mdlVidCard bas mdlVote bas mdlVoteNumber bas mdl WriteDB bas PROPS ICO Database Script Files AUTOR UN INF create_proc sql Create_tables sql CREAT EDB SQL drop_ Event Inser Data mdb drop_proc sql tables sql Log mdb t_Init_Data sql Language Instruction doc Readme txt SMJBD SMJBD SMJCO P32 DLL P32 txt MON INI CIBER Version 1 2 April 10 2006 188 12 46 45 34 15 24 657 7
14. 09 sO 48 09 09 09 09 09 48 09 09 09 09 04 08 08 08 50 08 08 08 08 08 08 23 08 0 9 41 09 24 54 24 28 24 24 50 24 26 50 24 24 24 24 24 24 26 26 26 26 26 26 26 26 26 26 26 26 52 26 26 26 28 28 28 28 28 28 28 30 38 52 52 52 24 54 54 54 54 54 58 10 58 00 22 00 O RR 0 o 0 o RR RR DR Ro OW o o 9 9 o Ro DR o Ro Ro 9 9 9 0 0 9 6 A g g g a g o RA 0 9 9 S 0 Re Ry he CB Rs CRE IRS IRS CR BRE ORR AON N N Ry D CR 1 D OR Bs RS UR M PR Re ER ys tie o tie CR GR RE GB US RR RR RRR RY Be o RS IBS 35 frmMsg frm frmMultiLanguage frm frmPaperRecord frm frmPrintResult frm frmPrintResult frx frmProgress frm frmReadReceipt frm frmSampleBallot frm frmSampleBallot frx frmSetting frm frmTally frm frmTally frx frmTestPrinter frm frmTransferTally frm frmVerifyBarcode frm frmVerifyBarcode frx mdlCDR bas mdlCheckMouseMove bas mdlCheckRawDataFile bas mdlCheckTime bas mdlCheckVID bas mdlColor bas mdlDevice bas mdlDisableLowLevelKeys bas mdlErrInfo bas mdlExportTabulation bas mdlFormMaganemnt bas mdlImportTabulation bas mdlMachineInformation bas mdlMain bas mdlMsg bas mdlNetworkSetting bas mdlPrinter bas mdlPrinter OBJ mdlReadDBSetting bas mdlReceiptNumber bas mdlRecord bas mdlShell bas mdlSimpleBallotStyle bas mdlSpeak bas mdlUserManagement bas mdlVerifyProvisional bas mdlVidCar
15. 1 2 1 2 1 2 1 2 1 2 1 7 2 7 2 7 2 4 2 7 2 7 2 7 2 7 2 7 2 7 2 1 9 2 2 17 2 12 21 2 9 12 2 8 3 2 9 7 2 O SOS O O O O 0 0 0 OS OO O OO OO O OD O O 0 0 0 0 0 0 O O CO O OO OO O 06 11 01 06 11 06 11 06 11 06 11 09 0 67 Ad 0511 05 06 11 05 11 0 65 Tl 02 01 06 11 01 06 11 01 0 6 Lis 01 0 6 Tis 06 11 06 11 02 06 11 0 6 Tas UG Ts 0 6 Aas UG Avis 05 05 02 06 05 05 05 05 06 06 03 06 04 05 03 05 02 06 09 N O Bo H o H Or q Ol NNNNNONNN UN ON BN BON WBN WWN UN MM DNDN W DN at Tet ket Tet ek ION K k JO bp pppy OrRFNFRN W H two An O W t 46 40 1 46 1 46 1 46 1 46 B 248 23 6 522 48 36 1 48 236 22 48 24 50 31 6 790 238 1 50 L352 Soe 40 8 92 1 52 1 52 1 52 1 52 L252 40 22 04 44 18 38 DP 9 Q Q R Up R G E W E R E Q R Q R Q WQ Q R E E R R E R R Q D DOY 0 0 09 Q 09 Q 0 0 Q 0 0 W 0 W Q 0 Q 9 0 0 0 py W Q pp Q 9 0 9 0 0 W ow d 40 APPENDIX C FUNCTIONAL TEST REQUIREMENTS FUNCTIONAL TEST APPROACH AND FINDINGS The test approach was to functionally test the changed or added EMS and OVT functions with test cases that exercised those functions and also provided an overall regression test of those components After Wyle completed its qualification testing of the FFVT and had indicated that they
16. LH A H A MM M E MLA OM LA LA PEN H WQ R R R WQ E R WQ E E WQ WQ R R Q Q R Q R R R Q E Q E D oo 0 0 0 09 Q 0 Q 0 Y Y Y 0 Y Y Y Q Y PP Q 0 0 9 0 W Q 0 OO mdlGetWriteIn bas mdlKeyStroke bas mdlLanguage bas mdlLevelDifinition bas mdlLimitKey bas mdlLog bas mdlManageDatabaseInfo bas mdlMapDrive bas mdlNormalMsg bas mdlPaperBallot Image bas mdlPassword bas mdlShowLanguage bas mdlShutDown bas mdlSQLDataType bas mdlStrFunctions bas mdlSupervisor bas mdlSystemInfo bas mdlTallySub bas mdlUserRight bas mdlWord bas frmBallotType frm frmBarcode frm frmBarcode frx frmNoVID frm frmNoVID frx frmTally frm frmTally frx mdlBarcode39 bas mdlBase36 bas mdlData bas mdlPaperBallotVarConst bas mdlTally bas mdlVIDValidation bas CountBallotsSourceCode BallotData cpp BallotData h barcode cpp barcode h CountBallots aps CountBallots clw CountBallots cpp CountBallots dsp CountBallots dsw CountBallots h CountBallots ncb CountBallots opt CountBallots plg CountBallots rc CountBallotsDlg cpp CountBallotsDlg h DataCheckDlg cpp DataCheckDlg h DBDlg cpp DBD1g h CIBER Version 1 2 April 10 2006 bh bh HA Ww LA H w LA J BI MA GO KA Ln KA KA On GO S M LU W HO O GO KA LU d WQ GO d KA KM dy Hs a K GR O WRF O UU Y 1273 w 78 NUNN Proprietary KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB
17. Recall and modified during the ballot preparation Judicial portion of this test Voting Devices Utilized FFVT and OVT Procedures e Load the LA database provided by Avante e In Generate Ballot data O O O O Removed contests from database to get a total of 30 Add 5 amendments Set Ballot and FFVT Database settings View ballots and adjust settings to fit the ballot on screen e On the OVT software running on the Ballot Preparation computer generate the paper ballot e Inthe Generate VID application generate 200 VIDS e Inthe OVT O O Or O OO Import the election database Print ballots Mark and scan ballots mark one to verify scanner sensitivity Adjust sensitivity of scanner and recount votes verify sensitivity changed Resolve write ins overvotes undetvotes Export the tabulation to FM e Use Manage Ballot Data to export the election database with the paper ballot definition e At the FFVT Qualified by Wyle O import the ballot vote 5 ballots export results to CD and close poll e In Tally application O O O O Set tally level to state in order to tabulate multiple counties Import FFVT tabulations CDs using the mass import option Import OVT totals Print all tally reports and verify accuracy CIBER Version 1 2 April 10 2006 Proprietary 46 Test Case syPE01 Configuration Primary Election no Cross party voting Definition
18. and verified independently shall be uniquely an mnemonically named shall be small enough to be easily followed and understood shall have a single entry point and process flow within the module shall be restricted e Control constructs Software must adhere to acceptable constructs e Naming conventions Names shall be chosen to enhance readability and intelligibility of the program shall be consistent and shall be unique within an application e Coding conventions Software shall adhere to basic coding conventions e Comment Conventions All modules must contain headers and provide header comments describing information contained in the header Descriptive comments shall be provided to identify objects and data types Evaluation The AI Technology VOTE TRAKKER voting system developers use Visual Basic 6 0 and C languages to construct a system of applications for election data processing The implementation of these languages utilize object oriented design and programming methodology to enhance understandable and maintainable of the code Findings The following is the list of components that were reviewed as a part of this version release All source code was reviewed to the 2002 standards Also listed are the types of anomalies found in each component All anomalies were corrected All source code meets the standards of the 2002 VSS CIBER Version 1 2 April 10 2006 Proprietary 25 Component EMS 5 2 9 Language Visual Ba
19. during system tests which required changes to the source code In each case CIBER reviewed the source code change witnessed the build of the installation files using that source code and conducted additional tests to validate the correction A total of 82 exceptions were recorded during the functional and system testing of this system which were resolved as follows e 37 were exceptions that required change to the code e 20 were exceptions that required changes corrections to the TDP System test conducted by CIBER used the FFVT that WYLE had successfully tested and was recommending for certification by the FEC All testing that used the FFVT was performed at the Wyle facility and coordinated with Wyle staff CIBER Version 1 2 April 10 2006 Proprietary 11 e 25 were test procedure mistakes incorrect operating procedures or were withdrawn after it was concluded the system was working correctly CIBER verified all resolutions by test or in the case of TDP changes by inspection of the documentation Appendix C provides a summary of each functional and system test case that was executed After completion of final system testing CIBER concludes that AI Technology VOTE TRAKKER Voting System meets the functional requirements provided by the FEC as well as the additional requirements stated or derived from the TDP 5 2 VOTING SYSTEM HARDWARE Wyle Laboratories performed functional environmental and hardware testing of the FFVT a
20. generated for the fnMaxCandidates test Generate a Ballot using Generate Ballot Data as follows o Rename district level for city to polling place and remove a polling place o Modify rotation groups to rotate one contest by polling place and the remaining by precinct o Set system settings for voting machines voting strategy voter paper receipt format Preview ballots using the voting test application and modify ballots and system settings as necessary Use Generate VID application to generate 5000 Voter Identification numbers In Generate Ballot Data o Set system settings to allow multi language set election name Add qualified write ins In MS Paint create party ICONs Add translations for Chinese and Spanish text export to separate computer translate and import translation using Generate Ballot Data Export Election Database To CD using the Manage Ballot Data Application In the Optical Vote Trakker OVT Computer import this election database using Load Ballot Data o Set paper ballot formatting options and generate full information ballots for one language Generate one ballot for Chinese language and one for Spanish language Generate and print on full information ballot using the full face layout Mark ballots include write in overvote and undervote contests Scan ballots Resolve write ins undervotes and overvotes o Export OVT tabulated results to Flash Memory a On the VID laptop at the
21. inserted access to meet the specific requirements of Volume I Section 6 2 2 Access Control Measures Vol Il Sec System Security Specification R2 0 System Security Specification 2 6 2 Access Control Measures 5 2 5 Rev A deleted reference to broadcasting module The vendor shall also define and R1 0 The last sentence of section 6 2 provide a detailed description of mentions a broadcasting module is the methods used to preclude this related to LAN or wireless unauthorized access to the access broadcasting control capabilities of the system itself Voll Sec Security for Transmission of Official Data R3 0 attachment has been removed Removed the attachment This change is 6 6 1 a Over Public Communications Networks No telecommunications or networking of included in Rev B of Section 6 TDP General Security Requirements for systems feature is removed in this Version 5 2 5 Systems Transmitting Data Over P ublic release However the attachment to Networks Section 6 of the TDP discusses VPN and data connections etc This should All systems that transmit data over public not apply since the system no longer telecommunications networks shall uses telecommunication and should be removed from the TDP Section a Preserve the secrecy of a voter s ballot choices and prevent anyone from violating ballot privacy Note a c also in functional checklist Vol ll Sec d Physical facilities and R2 0 manual received 2 6 6 d arrangements
22. numbers 800 000 e Export Election Database To CD using the Manage Ballot Data Application e In the Optical Vote Trakker OVT Computer import this election database using Load Ballot Data and then o Set paper ballot formatting options and optional text and generate full information ballots o Mark ballots fill some with 25 mark and 50 mark include write in overvote and undervote contests o Scan ballots verify sensitivity to partially marked choices invalid ballot detection o Resolve write ins undervotes and overvotes o Export OVT tabulated results to Flash Memory and save for use in functional test fnTallyGen e Verify election loads on FFVT and cast votes for each precinct and save FFVT Tabulation file for imput to TallyGen functional test NOTE the Election database CD and OVT FM exported from this test are used in the fnTallyGen Test CIBER Version 1 2 April 10 2006 Proprietary 42 Test Case fnPEO1 051020 Configuration Primary Election with Cross party voting e 4 Precincts 1 split Definition 1 State 2 counties 3 e 5 political parties municipalities 2 polling places absentee e 1 statewide contest 3 assembly district contests 2 senate ballots district contests 3 county contests 5 municipal contests 3 supervisory district contests one bond issue e Contest types N of M Straight Party Ranked Recall Judicial e Candidate rotation by District e Regular ballots Absentee paper
23. steps for recovering 5 4 RECOMMENDATION FOR QUALIFICATION It has been demonstrated through the TDP review source code review and functional testing that the AI Technology VOTE TRAKKER 1 2 0 voting system successfully meets the required acceptance criteria of the 2002 FEC Voting System Standards CIBER Version 1 2 April 10 2006 Proprietary 12 It is upon completion of this testing that CIBER recommends to the NASED committee that Avante International Technologies Inc VOTE TRAKKER 1 2 0 voting system be certified CIBER Version 1 2 April 10 2006 Proprietary 13 APPENDIX A TECHNICAL DATA PACKAGE TDP WORKSHEET TDP REVIEW APPROACH AND FINDINGS The TDP review is an evaluation for compliance with FEC guidelines for TDP completeness and quality The FEC standards state that vendor documentation relating to voting system hardware shall be submitted with the system as a precondition of qualification testing These are the items necessary to define the product and its method of operation to provide vendor technical and test data supporting the vendor s claims of the system s functional capabilities and performance levels and to document instructions and procedures governing system operation and field maintenance CIBER reviewed the changes in all documents and provided comments to AI Technology identifying the issues and the associated voting standard AI Technology resolved all issues in the final submittal of the associated d
24. than 3 3 The Polling Officials Manual does not mention a flash drive but that is in the configuration according to the Ballot Loading and pre election manual One of these documents should explain how the flash memory is used Vol Il Sec 2 10 1 C c System operations for voting system functions performed at the central count facility R3 0 1 Verified information has been Corrected page 9 This change is included revised or included as needed in Rev A for Tally and Reporting Manual R2 0 1 Tally and Reporting Manual 5 2 6 Version 5 2 6 is revised to delete electronic transmission on page 4 but not on page 9 2 Tally Manual page 26 and 32 revised This item is complete R1 0 1 Tally manual states on pages 4 and 9 that voting results can be transported from the polling place by electronic transmission That is inconsistent with statements that there is no use of telecommunication 2 On page 25 of the Tally manual it states that the VT exports 2 copies of ballot image data but then mentions only the VT hard drive and the flashcard The flash card is not mentioned in the Polling officials manual The export media needs to be specified and made consistent with the other manuals Vol Il Sec 2 10 1 Personnel Deployment and Training Requirements Personnel A description shall be presented of which functions may be carried out by user personnel and those that must be performed b
25. 0 L330 312 250 50 132 E 32 1332 Le32 1 34 1 34 46 02 28 28 28 16 34 40 40 24 12 312 10 28 00 04 50 42 302 258 Z R E R R R Q E R AR E R E E R R R Q R WQ WQ E E E R pP Z U GQ Q R E R Q WQ WQ WQ E E R pP P Y D 0 0 0 QW Q 09 Q 0 0 Q 0 0 W 0 W Q 0 Q Q 0 0 0 0 9 9 W Q 0 OD DOY O O 0 0 Q 9 9 Q pp 9 0 0 9 DO 38 IdentifyImage cpp IdentifyImage h ImageProcess cpp ImageProcess h ModulesSetup exe msadol5 tlh msado15 tli objtwain cpp objtwain h resource h StdAfx cpp StdAfx h TestResultDlg cpp TestResultDlg h twain h twcontrol cpp twcontrol h Database Script Files create_proc sql Create_tables sql CREATEDB SQL drop proc sql drop tables sql Event Log mdb Insert Init Data sql SMJBDP32 DLL SMJBDP32 txt SMJCOMMON INI Tabulation mdb Thumbs db Transfer_tables sql VoteTkVocF acs VoteTkVocM acs Voting RES Optical Vote Trakker 0001 111 bmp 0002E001 bmp 0004E001 bmp 0004E002 bmp 0004E003 bmp 0006E001 bmp 0006E002 bmp 0006E003 bmp 0007E001 bmp Cardl jp clsPaperBallot cls DTSReport doc frmAbout frm frmAbout frx frmBarcode frx frmCreateOption frm frmDBLog frm CIBER Version 1 2 April 10 2006 u LA A a KA 85 19 657 19 24 268 31 31 248 26 47 Proprietary KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB
26. 06 12 03 06 01 06 05 06 01 06 12s 06 125 02 06 05 06 06 08 06 04 06 08 06 03 06 04 06 12 06 05 10 08 Has OE ow BP BP OE HAS NASA OE NESSES OK OK OB Bw ds DN BBE BY BP OH 004 PB BRP WW DB BOHR OW O A O N O U O DO IO 0 O O UW O O N OF O O 00 0 00 0 UW 0 O O O KO KO W IO O DOO KO LU 0 O lO N O O KA LO O O O WTO D O a Re H N gt UW H oO O H ooo Has er t NONN A d m WW ds O R 0 t Eset Ho 0000 O 0100 Or FPF FP BBN ys 00 NO G MM M M Gm KM UQ KM L ha Ln ha LAMM KM MM M O K MM C9 KM d bM MM hM RNU w N o AN 0000000010 0100 d gt sa APA BP 0 K O Gn KA GO K KA KA AHN KFOOHKD GO 31 LanguageTranslation3082 mdb MachineData mdb mdlErrInfo bas mdlFontControl bas mdlGenerateCode bas mdlMain bas mdlSetCandidateOrder bas mdlTemplate bas mdlVoteSetting bas mfrmMain frm MSCAL OCX OneContestCandidate doc PollWorkerData mdb PrecinctSubprecinct mdb SystemSetting mdb Template mdb Template_Transfer_tables sql Generate VID frmAbout frm frmAbout frx FrmEnter frm frmIssuePollWorkerCard frm frmIssuePollWorkerCard frx frmKeyIn frm frmKeyIn frx frmLogin frm frmProgress frm frmSearchPrecinct frm frmSearchPrecinct frx frmVoters frm frmVoters frx frmWrite frm GenerateVID mdb mdlErrinfo bas mdlExport bas mdlMain bas mdlSetting bas mdlShow bas VoteTrakker VID GeneratorPasswo Load Ba
27. 2 15 2005 09 05 54 P mdlGetAllNetworkMachine bas 8 KB 12 15 2005 09 05 54 P CIBER Version 1 2 April 10 2006 Proprietary 27 mdlGetWriteIn bas mdlKeyStroke bas mdlLanguage bas mdlLevelDifinition bas mdlLimitKey bas mdlLog bas mdlManageDatabaseInfo bas mdlMapDrive bas mdlNormalMsg bas mdlPaperBallot Image bas mdlPassword bas mdlShowLanguage bas mdlShutDown bas mdlSQLDataType bas mdlStrFunctions bas mdlSupervisor bas mdlSystemInfo bas mdlTallySub bas mdlUserRight bas mdlWord bas frmBallotType frm frmBarcode frm frmBarcode frx frmNoVID frm frmNoVID frx frmTally frm frmTally frx mdlBarcode39 bas mdlBase36 bas mdlData bas mdlPaperBallotVarConst bas mdlTally bas mdlVIDValidation bas Card gif clsPaperBallot cls clsVoiceTxt cls frmAbout frm frmAbout frx frmADA frm frmBattery frm frmDateTime frm frmHandlePoll frm frmInsertCard frm frmKeyIn frm frmKeyIn frx frmLoadBallot frm frmMachine frm frmMachine frx frmMain frm frmMsg frm frmMultiLanguage frm frmPaperRecord frm frmPrintResult frm frmPrintResult frx frmProgress frm frmReadReceipt frm CIBER Version 1 2 April 10 2006 m bh HA Ww H LA w H md Was dA WO KA LH KA KA On GO S Now KA On GO GO KA LU d WQ GO d KA KM B La LJ LA A A LA Das KM SH No A LJ 9 A 0 0 LU O o oo 123 Proprietary KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB
28. 6 06 08 06 IZ 06 08 06 06 12 06 18 05 49 34 153 T2 BS 04 162 2 18 16 187 46 Les 182 47 AR SIE SS Ls OP w HO UMa wo 0 0 KA NON KOs Bow BABA NASAIS OE NH R RAK BROW Ol das ON OOOO O lO AO WOW 0 Y WOW WON 00 FF 00 fF 42 14 00 56 10 40 42 48 42 42 10 42 08 42 42 20 42 42 44 24 208 2 56 52 28 22 0 752 54 2 56 322 258 206 58 28 00 336 00 236 02 26 02 02 36 04 16 06 10 06 06 28 06 PM PM PM AM PM 0 9 9 9 9 g 9 9 9 9 g g 9 g A A R R M m 09 rh 30 frmInit frm frmInit frx frmLevel frm frmLevel frx frmLogin frm frmMachine frm frmMachine frx frmMultiInstruction frm frmMultiInstruction frx frmMultiLanguage frm frmMuniciplity frm frmMuniciplity frx frmOrderCandidate frm frmOrderCandidate frx frmParty frm frmParty frx frmPrecinct frm frmPrecinct frx frmPrecinctDistrict frm frmPrecinctDistrict frx frmPrintOption frm frmPubQuestion frm frmPubQuestion frx frmRecordingVoice frm frmRecordingVoice frx frmRestrict frx frmRestrictParty frm frmRestrictParty frx frmRotateDistrict frm frmRotateDistrict frx frmRotateGroup frm frmRotateGroup frx frmSelectGroup frm frmSelectGroup frx frmSelectUser frm frmSelectUser frx frmSetBallot frx frmSetBallotType frm frmSetBallotType frx frmSetVoteType frm frmSlateCandidate frm frmSlateCandidate frx frmSlateContest frm frmSlate
29. 7 2005 11 frmRawData frm 48 KB 12 7 2005 11 frmRawData frx 1 KB 4 6 2005 02 frmSelectAddress frm 23 KB 12 7 2005 11 frmSource frx 1 KB 10 1 2001 09 frmSourceInformation frm 46 KB 12 7 2005 11 frmSourceInformation frx 1 KB 4 6 2005 03 frmSourceTabulation frm 70 KB 12 7 2005 11 frmSourceTabulation frx 1 KB 4 6 2005 03 frmTally frm 190 KB 12 7 2005 11 frmTally frx 1 KB 12 7 2005 03 frmTally log 1 KB 4 14 2003 06 frmTypeInTabulation frx 1 KB 4 13 2003 10 frmVerifyProvisional frm 30 KB 12 7 2005 11 frmVerifyProvisional frx 1 KB 7 29 2005 04 mdlDriver bas 16 KB 12 7 2005 11 mdlErrInfo bas 6 KB 12 7 2005 11 mdlExportTabulation bas 12 KB 12 7 2005 11 mdlImportTabulation bas 50 KB 12 7 2005 11 mdlMain bas 3 KB 12 7 2005 11 mdlMassCD bas 35 KB 12 7 2005 11 mdlVerifyProvisional bas 2 KB 12 7 2005 11 VoteTrakker TallyPassword mdb 76 KB 1 11 2002 01 Full Face Vote Trakker 1 2 Common Modules clsAppLog cls 28 KB 1 24 2006 09 clsCRC cls 17 KB 1 24 2006 09 clsDBWrapper cls 8 KB 1 24 2006 09 clsRecorder cls 17 KB 1 24 2006 09 cReadTextTabulation cls 5 KB 1 24 2006 09 FrmChangeSupervisor frm 19 KB 12 27 2005 01 frmMsg frm 9 KB 12 21 2005 12 frmOneSource frm 53 KB 1 24 2006 09 frmOneSource frx 1 KB 4 4 2005 12 frmProgress frm 3 KB 12 21 2005 12 CIBER Version 1 2 April 10 2006 Proprietary A 0 09 a o M M R R R RM MM M o a o LO mm m O 0 0 9 0 0 0 0 9 9 Re URS GBC URS GRR RE GR BR oy
30. 9 1124 24 268 31 484 ST Proprietary KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB 12 1 15 2 7 29 2 12 1 12 1 15 2 15 2 9 2 2 12 1 12 1 12 1 15 2 15 2 15 2 11 30 2 12 1 12 1 1271 127 1 12 1 12 1 12 1 12 1 12 1 12 1 12 1 12 1 12 1 12 1 12 1 12 1 12 1 12 1 12 1 12 1 12 1 12 1 12 1 12 1 12 1 12 1 12 1 12 1 12 1 15 2 15 2 15 2 15 2 15 2 15 2 15 2 15 2 15 2 15 2 15 2 15 2 15 2 15 2 15 2 15 2 15 2 15 2 1542 15 2 15 2 15 2 15 2 15 2 15 2 15 2 15 2 15 2 15 2 OO Or O OO 00 O O L OO O 0000 00 O O 0 0 0 0 0 0 O O O O 0 060 0 0 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 7 26 1999 8 9 2 7 28 2 7 1 2 2 18 2 7 1 2 7 28 2 2 1 2 8 28 2 10 31 2 12 7 2 1 5 2 6 20 2 6 20 2 12 18 2 01 05 05 03 05 05 05 02 05 05 05 02 02 02 O O 0 0 gt 0 0 O O O Oi SO SO O 09 03 09 09 01 09 09 09 05 09 09 09 09 09 09 09 09 09 0 9 3 0 93 0 9 0 9 09 09 09 09 09 09 3 09 09 09 09 09 09 09 09 09 09 02 03 Oo 03 05 03 04 03 03 04 09 12 12
31. AVANTE INTERNATIONAL TECHNOLOGY Qualification Test Report Original Report for VOTE TRAKKER Version 1 2 0 created 02 27 2006 Ciber Revision 1 1 created 04 05 2006 Ciber Revision 1 2 created 04 10 2006 04 10 06 NASED Number N 1 12 22 22 003 2002 Prepared For The National Association of State Election Directors Sie Saat CIBER Inc Independent Test Authority Prepared By ciber CIBER Inc 7501 South Memorial Parkway Suite 107 Huntsville AL 35802 Document Revision 1 0 Original Report 1 1 2006 04 05 Added clarification in Section 3 under evaluation Added all components of system as tested in the Summary of Reports and Addendums Section 1 2 2006 04 10 Added text to the Hardware Subsection of Section 3 confirming that the units identified in that sub section had been certified Added text to Section 5 3 clarifying that a restart of the scanning not system reboot was valid recovery action and indicating actual test recovery performed CIBER Version 1 2 April 10 2006 Proprietary 1 1 INTRODUCTION CIBER is pleased to submit this report summarizing the qualification testing of the Avante International Technologies Inc AI Technology VOTE TRAKKER 1 2 0 voting system The AI Technology VOTE TRAKKER is an integrated voting system that includes three major components VOTE TRAKKER EVC308 FF DRE Voting Machine FFVT OPTICAL VOTE TRAKKER optical mark sense voting system OV
32. Contest frx frmState frm frmState frx frmTransferLanguage frm frmTransferLanguage frx FrmUser frm FrmUser frx frmVoterVID frm frmVoterVID frx frmWard frm frmWard frx LanguageTranslation mdb LanguageTranslation1028 mdb CIBER Version 1 2 April 10 2006 21 5 32 1068 1068 Proprietary KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB 12 2 2 12 2 2 12 2 2 7 26 2 12 2 2 12 2 2 7 26 2 12 2 2 3 17 2 12 2 2 12 2 2 7 26 2 12 2 2 3 15 2 12 2 2 10 17 2 12 2 2 10 31 2 12 2 2 7 26 2 12 2 2 12 2 2 7 26 2 12 2 2 11 11 2 8 22 2 12 2 2 3 15 2 12 2 2 7 29 2 12 2 2 3 15 2 12 2 2 3 17 2 12 2 2 3 17 2 2 18 2 12 2 2 7 29 2 12 2 2 12 2 2 7 26 2 12 2 2 3 17 2 12 2 2 7 26 2 12 2 2 5 25 2 12 2 2 5 19 2 12 2 2 10 17 2 12 2 2 8 1 2 11 11 2 10 26 2 OO OO OO O OO D 0 0 0 0 0 0O 0 0 T gt gt OO OO Or O OVO SO OD O OO OO O O O G E L 0 C CO O O O O O O 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 03 05 05 05 05 05 05 05 05 05 05 03 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 06 06 06 08 06 06 08 06 12 06 06 08 06 01 06 LZ 06 06 06 08 06 06 08
33. DeployKeyInForm bas mdlErrInfo bas mdlMain bas TouchScreenDriver txt Transfer tables sqgl TransferTabulation txt VoteTkVocF acs VoteTkVocM acs VoteTrakker Load Ballot DataPas VoteTrakker Manage Ballot DataP CIBER Version 1 2 April 10 2006 3572 24 ot 24 31 484 1068 70 17 248 76 280 657 79 3572 24 53 29 31 484 174 1068 840 N BH SOS HH RN AMO Y Y Proprietary KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB 2 18 2 12 7 2 7 28 2 2 1 2 8 5 2 3 15 2 8 5 2 7 26 2 8 5 2 10 31 2 12 7 2 5 25 2 8 5 2 8 5 2 8 572 87572 8 5 2 3 12 2 12 8 2 9 7 2 11 16 2 12 8 2 1 11 2 8 9 2 9 12 2 8 3 2 7 28 2 7 1 2 2 18 2 12 14 2 7 28 2 2 1 2 9 6 2 9 28 2 9 6 2 8 20 2 9 6 2 10 31 2 12 7 2 7 3 2 10 18 2 8 5 2 9 6 2 9 6 2 9 6 2 9 6 2 9 7 2 11 16 2 12 8 2 7 18 2 7 26 2 1 11 2 3 10 2 OOO OO 0 O O 0 0 0 0 0 0 00 0 O O O O O O Oo O O O O 0 0 0 00 0 10 0 O 00 00 0 0 0 0 0 O O O OOOPS 03 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 04 04 05 04 04 02 01 02 03 05 05 03 05 05 05 05 04 05 01 05 05 05 02 04 05 05 05 05 05 05 04 04 01 01 02 05 02 01 04 03 04 01 04 07 04 04 09
34. Environment 2 8 3 System Installation and Test Specification 2 8 4 Operational Features 2 8 5 Operating Procedures 2 8 6 Operations Support 2 8 7 Appendices TDP Section 8 is organized with the required sections Vol ll Sec System Operations Procedures 2 8 2 Operational Environment The vendor shall describe the system environment and the interface between the user or operator and the system R2 0 1 Ballot Preparation Manual Version 5 2 6 has been revised states the choice is grayed out permanently 2 0 References in System Operations Procedures 5 2 5 Ver A have been revised and verified R1 0 1 8 1 1 Section 8 1 1 of the Ballot Preparation manual refers to wireless network setting options but wireless is not included in this release 2 The references in Section 8 2 appear to not point to the correct sections in the referenced manuals Vol ll Sec System Operations Procedures 2 8 2a Operational Environment The vendor shall identify all facilities furnishings fixtures and utilities that will be required for equipment operations including equipment that operates a Polling place R2 0 Reference has been revised to state Polling Offical s Manual and references are correct R1 0 Reference to Poll Worker manual is incorrect CIBER Version 1 2 April 10 2006 Proprietary 20 2002 Description Ciber Comments Avante Comments Reqt Vol I
35. KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB 9 23 2 9 23 2 9 14 2 8 1 2 1 21 2 7 13 2 7 13 2 11 4 2 11 4 2 6 2 2 11 1 2 1 2 2 9 23 2 8 1 2 10 24 2 1 21 2 1 21 2 7 28 2 7 1 2 2 18 2 7 28 2 2 1 2 8 28 2 10 31 2 6 20 2 6 20 2 12 18 2 12 8 2 3 23 2 11 16 2 7 18 2 7 26 2 10 25 2 2 1 2 1 2 1 2 1 2 1 2f 2 1 2 1 2 1 3 1 2 1 2 1 2 1 5 1 2 1 2 1 2 1 4 2 4 2 4 2 4 2 4 2 7 2 7 2 7 2 T2 2 2 7 2 4 2 7 2 1 2 2 2 7 2 7 2 O O O 0 0 0 6 0O 0O OC CO 6 O Ore O OC 0 0 0 0 OC 00 0 O O0 0 O O O O O G OO 0 00 O Ov Oy DO 0 0 0 0 0 O 05 05 05 05 03 01 01 02 02 05 02 03 05 05 02 03 03 05 05 03 05 05 02 05 02 02 02 04 05 04 01 01 05 06 06 06 06 06 06 06 06 04 06 06 06 11 05 11 03 06 11 06 11 07 06 03 06 02 06 06 12 12 08 11 123 06 O7 O7 01 01 01 01 09 09 09 09 09 09 06 11 Ka K O N D d h 02 53 29 52 2 20 30 30 36 36 Sos 44 04 28 52 222 152 252 Bus OW WM W Wu JJ o os 54 49 23075 202 5 234 2 24 75 93 247 7538 2 18 305 234 248 22 Soo 42 378 22 3 22 22 04 04 04 077 Digig 1 44 L322 244 244 18 1 44 1 44 16 38 50 14 26 46 46 42 42 00 56 08 38 50
36. KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB 12 27 2 5 16 2 1 24 1 24 1 24 2 5 11 2 2 2 12 21 2 1 24 2 12 21 2 12 21 2 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 2 2 2 2 2 2 2 2 2 2 2 2 2 2 12 27 2 1 24 2 12 27 2 1 24 2 V2F 21 2 T 1 24 1 24 1 24 1 24 1 24 12 dz dz 2 2 12 27 2 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 2 2 2 2 3 23 2 2 2 2 7 26 2 2 2 2 2 2 2 12 5 2 2 2 12 15 2 1 24 2 O O OGG O Ov O O 0 0 0 0 0 0 0 0 000 0 O O OvyO O O O O OD O O OO O O O O O 0 0 0 0 OD O O O OO O O 05 05 06 05 06 06 05 06 05 05 06 06 06 06 06 06 06 06 06 06 06 06 06 06 05 06 05 06 05 05 06 06 06 06 06 05 06 06 06 06 05 06 06 06 05 06 06 06 06 06 06 05 06 06 05 06 01 03 09 03 09 09 12 09 12 12 09 09 09 09 09 09 09 09 0 9 US 0 9 3 09 09 09 Ody 09 01 09 12 01 09 09 09 09 09 01 09 09 09 09 03 09 09 09 08 09 09 09 09 09 0 9 05 09 09 08 09 48 534 09 52 2 09 09 26 09 33 26 09 09 09 09 09 09 09 09 09 09 09 09 09 09 48 09 48
37. Optical Vote Trakker Common Modules clsAppLog cls 28 KB 2 17 2006 11 clsCRC cls 17 KB 2 17 2006 10 clsDBWrapper cls 8 KB 2 17 2006 11 clsRecorder cls 17 KB 1 24 2006 07 cReadTextTabulation cls 5 KB 2 17 2006 11 FrmChangeSupervisor frm 19 KB 2 17 2006 11 frmMsg frm 9 KB 2 14 2006 04 frmOneSource frm 53 KB 2 17 2006 11 frmOneSource frx 1 KB 4 4 2005 10 frmProgress frm 3 KB 2 17 2006 10 frmQualifyWriteIn frm 28 KB 2 17 2006 11 frmQualifyWriteIn frx 1 KB 5 16 2005 01 frmTransferRichText frm 1 KB 2 17 2006 11 frmTransferRichText frx 1 KB 5 11 2005 01 frmTransferRichText log 1 KB 1 25 2003 12 mdlBasicDatabaseOperation bas 15 KB 2 17 2006 11 mdlBringWindowToTop bas 9 KB 2 17 2006 11 mdlBuildDB bas 24 KB 2 14 2006 04 mdlCheckContestNumber bas 4 KB 1 24 2006 07 mdlCheckLogin bas 2 KB 2 17 2006 10 mdlCheckVoteSetting bas 13 KB 2 17 2006 10 mdlComPort bas 4 KB 1 24 2006 07 md1lCRC bas 8 KB 2 17 2006 10 mdlDBConnection bas 9 KB 2 17 2006 11 mdlDriverInfo bas 16 KB 2 14 2006 04 mdlEncrypt bas 6 KB 2 17 2006 11 mdlEventLog bas 13 KB 2 17 2006 11 mdlFileControl bas 5 KB 2 17 2006 10 mdlFileFolder bas 4 KB 2 17 2006 11 mdlFileManagement bas 3 KB 2 17 2006 11 mdlGetAllNetworkMachine bas 8 KB 2 14 2006 04 CIBER Version 1 2 April 10 2006 Proprietary RBRBRBRRRBRBRRBRRRRRRRR Do E oO go a g A o A y o E A E K to oO H A H to N UW H N LJ
38. Other items relevant to the system the previous certification review and evaluation shall be submitted those copies are automatically along with this documentation accepted Only the documents that such as disks tapes source have changes should be resubmitted code object code and sample A number of documents in the list have output report formats dates and revision levels different than last certified copies and are expected to be delivered prior to functional testing I can provide a list of those documents if requested Vol Il Technical Data Package Scope R4 0 Received Appendix E with Same as above Sec 2 1 procurement information Both formal documentation and R3 0 Did not find procurement notes of the vendors system information for all third party software development process shall be listed in General System and Design submitted for qualification tests Spec 5 2 5 Rev B R1 0 More documents remain to be delivered Vol II Sec d Software design and R3 0 verified footer is updated Updated the footer in Rev B of Section 5 2 1 1 1d specifications R2 0 update received Part of footer TDP Version 5 2 5 Item c Hardware specification states Technical Data Package 5 2 5 and part states Technical Data Package 5 2 6 Scope List of Documentation states it should be 5 2 5 Rev A R1 0 Update is expected Vol II Sec e System and test verification Update is expected Section 12 remains the same one 2 1 1 1e specificatio
39. State 1 county 2 municipalities 1 polling place absentee ballots e 4Precincts 1 split e 5 political parties e 1 statewide contest 3 assembly district contests 2 senate district contests 3 county contests 5 municipal contests 3 supervisory district contests one bond issue e Contest types N of M Recall Judicial Group e Candidate rotation by District e Regular ballots Absentee paper ballots Provisional Ballots After Hour Provisional Ballots e English only e Multi line write in Voting Devices Utilized OVT FFVT Procedures e In Manage Ballot Data application create a database for this election e Generate a Ballot using Generate Ballot Data as follows o Create districts precincts sub precincts Setup judicial groups Oo O OO OO Enter contests for each party and some non partisan Setup rotation groups to demonstrate district rotation Add candidates to contests and order candidates Set system settings for voting machines voting strategy voter paper receipt format Preview ballots using the voting test application and modify ballots and system settings e Use Generate VID application to generate 2000 Identification Numbers e Inthe OVT application on the Ballot Generation computer generate paper ballot formats e Export Election Database To CD using the Manage Ballot Data Application e In the Optical Vote Trakker OVT Computer import this election d
40. T and the VOTE TRAKKER Election Management System EMS software 1 1 TEST AGENCY TEST AGENCY HISTORY AND CAPABILITY CIBER Inc has been providing IT consulting services for over 20 years Although the ITA division name has changed due to an aggressive acquisition and merger market the ITA division of the company has had the same leadership in place since inception Founded in 1974 the company s consultants now serve client businesses from 35 CIBER 10 DigiTerra 4 Solution Partners and 4 Enspherics offices in the U S Canada and Europe With offices in six countries CIBER s 4 500 IT specialists continuously build test and upgrade our client s systems to competitive advantage status CIBER provides a single source for IT solutions including e Full solution ASP services e Applications maintenance and support e Testing and IQA e Web and database hosting e Enterprise solutions including SAP Oracle and Peoplesoft e Application outsourcing e eBusiness from architecture through execution e Knowledge management and training The company has been involved in numerous QA and IQA testing projects for commercial state and federal government customers CIBER has an interim accreditation as an Independent Test Authority ITA through the National Association of State Election Directors NASED 1 2 TEST APPROACH The submitted system is the initial qualification test for the FFVT Hardware and its embedded firmware That hardwar
41. and was allocated diagrams that represent a A description of the functional components or subsystems as defined by the vendor e g environment election management and control vote recording vote conversion reporting and their interconnection Vol Il Sec g Benchmark directory listings for updated database and software design Section 5 and Appendix J were updated 2 219 all software including firmware document is to be submitted and included in the CD that Tom has elements and associated documentation included in the vendor s release as they would normally be installed upon setup and installation Vol Il Sec b Quality attributes such as R1 0 accept mainly a hardware issue Reference to the Maintenance Manual 2 2 2 b reliability maintainability usability R1 0 no quantitative analysis no about the MTBF was added in Rev A of availability and portability supporting analysis or description of Section 2 any testing design features to ensure reliability level Note text in Section 3 indicates MTBF to be provided in maintenance manual Vol ll Sec Software Design and Specification R4 0 Received Appendix E with No change from 4 7 5 2 5 3 Software Overview procurement information R3 0 Did not find procurement The vendor shall also include a information for all third party software certification that procured software listed in General System and Design items were obtained directly from Spec 5 2 5 Rev B the m
42. anufacturer or a licensed Certification Licenses not found dealer or distributor Vol ll Sec e Software testing and verification R1 0 Section 12 referenced new No change from 4 7 5 2 5 4e standards including internal version not provided to date Hold vendor procedures that can assist review until vendor confirms version to in determining the program s be used correctness and ACCEPT REJECT criteria Vol II Sec f Quality assurance standards or R2 0 ISO certification information has 2 5 4 f other documents that can be used been received and verified as valid thru by the ITA to examine and test the 8 10 08 software These documents Reference on page 4 47 updated not include standards for program flow received and control charts program R1 0 Page 5 46 refers to ISO documentation test planning and certification is that still valid The for test data acquisition and reference on page 5 47 is to a version reporting of the quality manual used in the previous certification and does not refer to the QA manual delivered in this TDP otherwise there is a good discussion of QA procedures that were implemented Vol ll Sec e Data input output device R3 0 verified touch codes have been Removed the touch codes This change 2 5 5 1 e protocols and removed is included in Section 5 for Version 5 2 5 CIBER Version 1 2 April 10 2006 page 5 51 indicates that a sequence of touches is used by voting official from o
43. ary e Use Generate VID application to generate 1000 Identification Numbers e Use the OVT Application on this same ballot preparation computer to o Set ballot format options create and print reference ballots and associated sample ballots e Export Election Database To CD using the Manage Ballot Data Application e In the Optical Vote Trakker OVT Computer import this election database using Load Ballot Data and then o Mark ballots include write in overvote and undervote contests o Scan ballots verify invalid ballot detection o Resolve write ins undervotes and overvotes o Export OVT tabulated results to Flash Memory and save for use in functional test fnTally e Save the exported CD to validate that the system settings cause the expected result at the FFVT NOTE the Election database CD and OVT FM exported from this test are used in the fnTally Test CIBER Version 1 2 April 10 2006 Proprietary 43 Test Case fnTally Configuration Primary Election with Cross party voting e 4 Precincts 1 split Definition 1 State 2 counties 3 e 5 political parties municipalities 2 polling places absentee e 1 statewide contest 3 assembly district contests 2 senate ballots district contests 3 county contests 5 municipal contests 3 Database is same as used for supervisory district contests one bond issue fnPEO1_051020 OVT tabulation file were e Contest types N of M Straight Party Ranked Recall generate
44. ase Manual Panasonic Image Enhancement Technology NA On File Previous Release Manual Canon Scanner User Manual NA On File Previous Release Manual Lexmark T522 Printer User Manual NA On File Previous Release Manual MS User Interface Windows Design Guidelines NA On File Previous Release CIBER Version 1 2 April 10 2006 Proprietary 6 Manual Configuration Management Plan for Vote Trakker 1 1 0 On File Previous Release J Database and Software Design Specifications amp Coding 1 1 0 06 02 05 Conventions Manual Sample of Warehouse Facilities Plan NA On File Previous Release L California Procedures NA On File Previous Release Manual Connecticut Moderator Handbook Rev 1 0 On File Previous Release Manual Quality Manual ISO9001 2000 Rev 2 On File Previous Release Manual Quality Operation Procedures and Forms Rev A On File Previous Release Manual Quality Work Instructions and Forms Rev A On File Previous Release 4 SYSTEM OVERVIEW The VOTE TRAKKER voting system includes the FFVT voting machine the OVT paper ballot component and the EMS software for ballot preparation and vote tabulation The FFVT provides touch screen voting using a large display that allows all contests to appear on one page Refer to Wyle Report 52667 01 for a description of this device It accepts ballots prepared using the EMS software and the votes it records are trans
45. atabase using Load Ballot Data o Print ballots o Mark ballots include write in overvote and undervote contests o Scan ballots verify invalid ballot detection o Resolve write ins undervotes and overvotes o Export OVT tabulated results to Flash Memory e In Tally import the OVT results and verify they match actual ballot votes e In FFVT o Import the election cast votes including write ins and provisional votes o Verify provisional votes at the FFVT o Close poll and export tally e In Tally o Import FFVT tabulation and event log o Tally and verify tabulation results o Print logs and reports CIBER Version 1 2 April 10 2006 Proprietary 47 Test Case syGE01 Configuration General Election e 4 Precincts 1 split Definition 1 State 2 counties 3 e 5 political parties municipalities 2 polling places absentee e 1 statewide contest 3 assembly district contests 2 senate ballots district contests 3 county contests 5 municipal contests 3 supervisory district contests one bond issue e Contest types N of M Straight Party Ranked Recall e Candidate rotation by District and Precinct e Regular ballots Absentee paper ballots Provisional Ballots After Hour Provisional Ballots e English Spanish Chinese languages e Multi line write in Voting Devices Utilized OVT FFVT Laptop at Polling Place dedicated to VID Card printing Procedures Restore the election database that was
46. d Repair Dave Alampi Manual has been received This item is complete 2 Section 11 7 still contains reference to Vote Trakker Configuration Management Plan Does a separate CM plan exist 1 Section 11 7 c refers to the maintenance and repair manual which is not yet provided This reference and the reference to the assembly manual need to be enhanced to show what sections paragraphs in those manuals are being referenced 2 The same section 11 7 c refers to the Vote Trakker Configuration Management Plan Which is what this section of the TDP is named could not locate any copy of a separate CM Plan If that document exists please provide Otherwise please correct the references Vol ll Sec Quality Assurance Program R2 0 Change notes have been 2 13 System Change Notes received Vendors submitting a system for testing that has been tested previously by the test authority and issued a qualification number shall submit system change notes These will be used by the test authority to assist in developing and executing the test plan for the modified system Vol ll Sec Quality Assurance Program R2 0 Change notes have been 2 13 a System Change Notes received The system change notes shall include the following information a Summary description of the nature and scope of the changes and reasons for each change Vol ll Sec b A listing of the specific changes R2 0 Change notes have been 2 13 b made citing th
47. d bas mdlVote bas mdlVoteNumber bas mdlWriteDB bas PROPS ICO Voting Full Face CIBER Version 1 2 April 10 2006 188 12 46 45 34 15 24 HAHAHA Proprietary KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB HA HA HA 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 1 24 2 2 7 13 2 2 1 30 2 2 2 2 7 29 2 2 2 9 2 2 2 2 2 1 30 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 12 kz 72 1 12 2 2 2 2 2 2 2 2 2 2 2 2 2 0 1090 O O eD E e AE O O On OG OG O 0 00 70 O 0 OO O OO OO O O O 0 0 0 0 0 0 O O E O Or ora O O 06 06 05 06 05 06 06 06 05 06 06 05 06 06 06 05 06 06 06 06 06 06 06 06 06 06 06 06 06 06 06 06 06 03 06 06 06 06 06 06 06 06 06 06 06 06 7 26 1999 6 18 2003 6 18 2003 6 18 2003 6 18 2003 6 18 2003 09 09 10 09 06 0 9 09 09 03 09 09 01 09 09 09 05 09 09 0 9 US 0 9 3 09 09 09 09 09 09 09 09 09 09 09 09 08 09 09 09 09 09 09 09
48. d in that test is input to this test Judicial Voting Devices Utilized FFVT not certified used only to generate test input data Procedures e Use Load Ballot Data Application to load the fnPE01_051020 database e In Tally application o Set tally level to state in order to tabulate multiple counties Import OVT totals using specified path option Create qualified write in candidate Validate OVT imported correctly Use an available FFVT to enter and tabulate votes for election database fnPE01_051020 Enter ADA and regular votes Export the tabulation file Import FFVT Tabulation file Validate provisional votes o Print all tally reports and verify accuracy 0000 oo Test Case fnTallyGen Configuration General Election e 4 Precincts 1 split Definition 1 State 2 counties e 5 political parties 3 municipalities 2 polling places absentee e 1 statewide contest 3 assembly district contests 2 senate ballots district contests 3 county contests 5 municipal contests 3 Database is same as used for supervisory district contests one bond issue fnGE01_050930 OVT tabulation file e Contest types N of M Straight Party Ranked Recall generated in that test is input to this test Judicial Voting Devices Utilized Bar code scanner for recount FF VT used to generate input data for test Procedures e Load the previously exported fnGE01_050930 database e In Tally applicati
49. ds 5 1 3 Functional Test Description The main goal of functional testing is to verify that the changes to the previously certified VOTE TRAKKER EMS and OVT conformed to FEC Standards and to provide a regression test of the entire system to verify it continued to operate as documented See Wyle report 52667 01 for a description of functional testing of the FFVT The functional testing performed by CIBER included creating ballots for voting tallying both DRE and paper ballots and reporting results for primary and general elections The testing validated overall system performance functional operation security and audit requirements as specified in the FEC Voting System Standards CIBER recorded all exceptions and tracked their resolution The vendor resolved the exceptions by updating documentation to clarify functions and operations or by modifying the software Some exceptions were found to be due to factors that did not require any changes by the vendor such as incorrect test procedure test conductor error or misinterpretation of the function After successful completion of the functional tests CIBER witnessed the software compile build and creation of the installation package This installation package was then installed at the CIBER facility and end to end system tests performed to verify the integrated operation of the EMS software Full Face Vote Trakker and Optical VOTE TRAKKER Some additional exceptions were identified
50. e is integrated with modified versions of the EMS software previous version 4 7 6 and the OVT previous version 1 5 0 that were CIBER Version 1 2 April 10 2006 Proprietary 2 previously certified The test approach considered this release as a new system but recognized that the EMS and OVT had been previously tested and certified The qualification testing of those components of the system did not repeat the reviews and inspections for documents and source code that had not been altered from the previously certified release System testing included all components as an initial release Wyle Laboratories performed hardware and functional testing of the FFVT They also reviewed the Technical Data Package associated with this hardware CIBER performed the source code review of all firmware that resides in this hardware The OVT includes a Optical Scanner and COTS desktop computer with custom software The optical scanner hardware is a COTS scanner that was certified in testing by CIBER as part of the previously certified OVT version 1 5 0 The same scanner was submitted with this version of the system and had not been altered from the previous certification so that it was not necessary to repeat the hardware testing required for the scanner Significant changes were made to the OVT custom software which required CIBER to conducted functional testing of the OVT CIBER reviewed the Technical Data Package associated with the OVT and inspected all sou
51. e specific system received configuration items changed and providing detailed references to the sections of the documentation Proprietary 23 2002 Description Ciber Comments Avante Comments Reqt changed Vol ll Sec c The specific sections of the R2 0 Change notes have been 2 13 C documentation that are changed received or complete revised documents if more suitable to address a large number of changes Vol II Sec d Documentation of the test plan R2 0 Change notes have been 22 13 and procedures executed by the received vendor for testing the individual changes and the system as a whole and records of the test results CIBER Version 1 2 April 10 2006 Proprietary 24 APPENDIX B SOURCE CODE REVIEW SOURCE CODE REVIEW APPROACH AND FINDINGS The source code review is an evaluation for compliance with FEC guidelines and AI Technology standards for software quality This report details the results of the AI Technology VOTE TRAKKER voting system software evaluation The evaluation is an assessment of the source code considering the following characteristics e Selection of programming languages High level programming language shall be used e Software integrity Self modifying dynamically loaded or interpreted code is prohibited e Software modularity and programming Software shall be designed in a modular fashion Modules shall have a specific function which can be tested
52. eaning Below is a list of all source code files that were delivered to CIBER for review EMS 5 2 9 Common Modules clsAppLog cls 28 KB 12 15 2005 09 05 50 PM clsCRC cls 17 KB 12 15 2005 09 05 50 P clsDBWrapper cls 8 KB 12 15 2005 09 05 50 P clsRecorder cls 17 KB 12 15 2005 09 05 50 P cReadTextTabulation cls 5 KB 12 15 2005 09 05 50 P FrmChangeSupervisor frm 19 KB 12 14 2005 07 22 44 P frmMsg frm 9 KB 9 6 2005 11 09 36 A frmOneSource frm 53 KB 12 15 2005 09 05 52 P frmOneSource frx 1 KB 4 4 2005 12 02 02 P frmProgress frm 3 KB 12 2 2005 06 48 52 P frmQualifyWriteln frm 28 KB 12 14 2005 07 22 46 P frmQualifyWriteIn frx 1 KB 5 16 2005 03 53 54 P frmTransferRichText frm 1 KB 12 15 2005 09 05 52 P frmTransferRichText frx 1 KB 5 11 2005 03 52 28 P mdlBasicDatabaseOperation bas 15 KB 12 15 2005 09 05 52 P mdlBringWindowToTop bas 9 KB 12 15 2005 09 05 52 P mdlBuildDB bas 24 KB 9 6 2005 11 09 36 A mdlCheckContestNumber bas 4 KB 12 15 2005 09 05 52 P mdlCheckLogin bas 2 KB 12 7 2005 11 08 22 P mdlCheckVoteSetting bas 13 KB 12 2 2005 06 48 54 P mdlComPort bas 4 KB 12 15 2005 09 05 52 P md1lCRC bas 8 KB 12 15 2005 09 05 52 P mdlDBConnection bas 9 KB 12 15 2005 09 05 52 P mdlDriverInfo bas 16 KB 12 15 2005 09 05 52 P mdlEncrypt bas 6 KB 12 15 2005 09 05 52 P mdlEventLog bas 13 KB 12 15 2005 09 05 54 P mdlFileControl bas 5 KB 12 15 2005 09 05 54 P mdlFileFolder bas 4 KB 12 15 2005 09 05 54 P mdlFileManagement bas 3 KB 1
53. ev 3 8 1 TM RD e P DEM Generate VID Rev 4 0 5 Vote Trakker Hardware Version 1 4 0 Software Version 4 7 5 Generate Ballot Data Rev 5 1 1 Load Ballot Data Rev 3 5 1 Event Log 4 0 0 Tally 4 0 2 Vote Trakker Hardware Version 1 3 0 Software Version 4 7 5 VT Printer COTS Seiko M N KPU S347 L Optical Vote Trakker Rev 1 5 0 Count Ballots Rev 1 00 4 Original report consisted of full TDP and source code review to the 2002 standards Full functional system integration test to 2002 standards Addendum 1 Created 06 11 04 Software Hardware Source Code Functional Test VOTE Optical Vote Trakker 1 5 0 COTS Canon Manage Ballot Date Rev 3 8 1 TM o ee ARO PRO Ca Generate VID Rev 4 0 5 Vote Trakker Hardware Version 1 4 0 Software Version 4 7 6 Generate Ballot Data Rev 5 1 1 Load Ballot Data Rev 3 5 1 Event Log 4 0 0 Tally 4 0 2 Vote Trakker Hardware Version 1 3 0 Software Version 4 7 6 VT Printer COTS Seiko M N KPU S347 S Optical Vote Trakker Rev 1 5 0 Count Ballots Rev 1 00 4 Addendum 1 consisted of changes to code of the hardware only The software ITA verified with the hardware ITA that the functionality of the software was not affected by the hardware change Original Report Created 2 27 06 Software Functional Test Hardware Source Code CIBER Version 1 2 April 10 2006 Proprietary VOTE TRAKKER EVC308 FF 1 2 0 COTS Scan
54. evision history 4 2 7b Variables objects and data types need comments upon declaration 4 2 7c Code needs in line comments 4 2 7e Commenting is not uniform 5 4 2h Indenting is inconsistant 5 4 2k Lines is more than 80 columns in width 5 4 20 No message on exit 5 4 2q Over 5 levels of indirection 5 4 2r Over 5 levels of indenting 5 4 2t Implied comparisons not allowed 5 4 2u Constants should be defined enumerated or have comments explaining its meaning CIBER Version 1 2 April 10 2006 Proprietary 26 Component Optical Vote Trakker 1 6 9 Language Visual Basic and C Noted amp Corrected 4 2 3b Not legal name Anomalies 4 2 3e Module does not have a single entry and exit point 4 2 3f and 5 4 1 Module contains a nonpermissible construct 4 2 Tal Module header needs a purpose 4 2 7a2 Module header needs comments for units called 4 2 7a3 Module header needs comments for input and outputs 4 2 7a4 Module header needs comments for file referenced 4 2 7a5 Module header needs comments for global variables 4 2 7a6 Module header needs revision history 4 2 7b Variables objects and data types need comments upon declaration 4 2 7c Code needs in line comments 5 4 2s Variables must be initialized where declared where permitted 5 4 2t Implied comparisons not allowed 5 4 2u Constants should be defined enumerated or have comments explaining its m
55. f 60 candidates in that contest o Set Ballot and FFVT Database settings And set default polling place to Birmingham precinct Use Manage Ballot Data to export the election database At the FFVT o import the ballot vote regular provisional and after hour provisional ballots Close poll and export tabulation o Modify FFVT address to simulate different physical precinct and repeat voting as in previous step o Modify FFVT address to simulate different physical precinct and repeat voting as in previous step In the OVT o Import the election database Create paper ballots and print them Mark and scan ballots to simulate incorrect ballots as might occur in real election Resolve write ins overvotes undervotes invalid ballots Print all OVT tabulation reports o Export the tabulation to FM In Tally application o Set tally level to state in order to tabulate multiple counties Import FFVT tabulations CDs using the mass import option Import OVT totals Print all tally reports and verify accuracy Print reports and logs O O O O 0000 CIBER Version 1 2 April 10 2006 Proprietary 45 Test Case syAvanteLA Configuration General Election e 1 precinct 1 ballot style Definition One ballot style to handle large e 9 political parties number of contests and candidates e 30 contests 270 candidates 5 amendments Election database was provided by Avante e Contest types N of M Straight Party Ranked
56. ferred to the EMS for central tabulation The OVT consists of custom software that resides on a COTS PC and controls a commercial scanner Ballot definitions prepared with the EMS are loaded to the OVT The OVT is used to print the paper ballots scan and tally the voted paper ballots and print reports of the results The tabulated results are transferred to the EMS central tally for consolidation with the FFVT votes The EMS provides an election administrator with all the tools required to prepare count and report an election in accordance with local laws Each election is kept in its own database The EMS provides the tools to manage and administrate these databases The following overview is a summary of the capabilities provided by this system for a full statement of the systems capabilities refer to the Vendors Functional Requirements document and operating manuals listed in Section 3 The basic functionality of VOTE TRAKKER voting system is grouped as follows CIBER Version 1 2 April 10 2006 Proprietary 7 e Security The EMS OVT and FFVT are initialized into a locked environment that allows the election official to access the database only through the authorized application Functions are secured by user ID and password Information is exchanged with the OVT and FFVT using a flash memory or Compact Disk and protected against viewing or alteration with unauthorized software No networking or public transmission is used e Sys
57. frmQualifyWriteln frm frmQualifyWriteln frx frmTransferRichText frm frmTransferRichText frx mdlBasicDatabaseOperation bas mdlBringWindowToTop bas mdlBuildDB bas mdlCheckContestNumber bas mdlCheckLogin bas mdlCheckVoteSetting bas mdlComPort bas mdlCRC bas mdlDBConnection bas mdlDriverInfo bas mdlEncrypt bas mdlEventLog bas mdlFileControl bas mdlFileFolder bas mdlFileManagement bas mdlGetAllNetworkMachine bas mdlGetWriteIn bas mdlKeyStroke bas mdlLanguage bas mdlLevelDifinition bas mdlLimitKey bas mdlLog bas mdlManageDatabaseInfo bas mdlMapDrive bas mdlNormalMsg bas mdlPaperBallot Image bas mdlPassword bas mdlShowLanguage bas mdlShutDown bas mdlSQLDataType bas mdlStrFunctions bas mdlSupervisor bas mdlSystemInfo bas mdlTallySub bas mdlUserRight bas mdlWord bas Card gif clsPaperBallot cls clsVoiceTxt cls frmAbout frm frmAbout frx frmADA frm frmBattery frm frmDateTime frm frmHandlePoll frm frmInsertCard frm frmKeyIn frm frmKeyIn frx frmLoadBallot frm frmMachine frm frmMachine frx frmMain frm CIBER Version 1 2 April 10 2006 N oo HA N LA N GU to Ho 0 QO KA Ln 4 OON KA KM A q 4 0 LU OWA O O 0 Ss YY DN ds ys o OFF LA bh hb HA HA HA Ww HAHAHA Ds NH w K 0 a LJ 9 A 00 O 00600 E 123 20 59 58 Proprietary KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB
58. g is to demonstrate compliance with levels of design performance and quality claimed for them by manufacturers The tests are also intended to demonstrate that the system meets or exceeds the requirements of the Federal Election Commission FEC Voting System Standards The scope and detail of the requirements for qualification have been tailored to the design and complexity of the software submitted by AI Technology for testing The qualification test procedure is intended to discover defects in software design and system operation which should they occur in actual election use could result in failure to complete election operations in a satisfactory manner The tests have been designed to evaluate system compliance with the requirements of the FEC Voting System Standards The examination includes selective in depth examination of software the inspection and evaluation of system documentation and optional tests verifying system performance and function under normal and abnormal conditions 3 SYSTEM IDENTIFICATION The Full Face VOTE TRAKKER 1 2 0 voting system submitted by AI Technology for qualification testing consisted of the following software hardware and documentation Software See Wyle Report 52667 01 for firmware included in the FFVT e Election Management Software EMS Version 5 2 9 o Event Log 4 0 4 Generate Ballot Data 5 2 9 Generate VID 4 2 0 Load Ballot Data 3 5 7 Tally 4 1 6 Manage Ballot Data 3 8 6 Test Voting 1 2
59. he day the voting totals the images of each cast ballot and the audit logs are transferred to CD for transporting to the central tally location Options allow for voting and verifying of provisional ballots and after hour provisional ballots o The OVT accepts the same election database that is loaded into the FFVT The OVT is used to print the paper ballots and to scan the ballots after voting It also accumulates totals provides for resolution of overvotes undervotes and invalid ballots The results are tallied and then transferred to flash memory or CD for transporting to the Central Tally facility e Post Election o The EMS Tally program provides the capability to receive the information from each FFVT and OVT and to consolidate tally and report the election results The polling official can view an image of each ballot cast can resolve provisional votes and can print reports Multiple tally sites can be defined and their totals can be transported to a central site for consolidation e Tools o The Test Voting application provides a preview capability for the election official to see the ballot on the EMS host computer monitor switch back to the Ballot Generator application to edit the definition and then return to the Test Voting application to view the modified ballot CIBER Version 1 2 April 10 2006 Proprietary 9 o The language translation of ballot text requires MicroSoft Access to view the phrases to be
60. ision Document Section Date 1 Scope 5 2 5 RevF 02 24 06 2 System Overview 5 2 5 Rev 10 10 05 3 System Functionality Description 5 2 5 Rev 09 12 05 4 System Hardware Specification 4 7 5 RevB 10 11 05 5 System Design and Specification 5 2 5 Rev E 02 24 06 6 System Security Specification 5 2 5 Rev D 01 05 06 7 System Test and Verification Specification 5 2 5 07 18 05 8 System Operation Procedures 5 2 5 Rev F 02 24 06 9 System Maintenance Procedures 5 2 5 Rev E 02 24 06 10 Personnel Deployment and Training Requirements 5 2 5 Rev 10 10 05 11 Configuration Management Plan 5 2 5Rev A 10 10 05 12 Quality Assurance Program 5 2 5Rev A 02 04 04 A State Certification Letters 4 6 7 On File Previous Release B Sacramento County Report of Early Voting 4 6 7 On File Previous Release C United States Patent Applications 4 6 7 On File Previous Release E Procured Software Licenses N A 11 15 05 Manual System Administration and Security Management 5 2 9 12 02 05 Manual Manual Ballot Preparation Manual Vote Trakker EMS 5 2 9RevB 01 25 06 Manual Ballot Loading and Pre Election Management Manual 5 2 9 12 02 05 Manual Poll Worker s Manual EE308 FF 1 2 0 1 25 06 Manual Tally and Reporting Manual 5 2 9RevA 01 05 06 Manual Optical Vote Trakker Operating Manual 1 7 0 12 28 05 Manual Maintenance and Repair Manual EVE308 FF 1 2 0 1 25 06 Manual Panasonic scanner Operating Instructions NA On File Previous Rele
61. l Sec b Central count facility and R3 0 verified corrected references Added reference to Secton 8 page 5 This 2 8 2 b R1 0 Reference to Optical VT Manual change is included in Rev B for Section 8 appears to point to wrong section of TDP 5 2 5 We think that the reference is correct now If not please let us know what need to be referenced Vol ll Sec c Other locations R2 0 Section 7 5 of Polling Official s 2 8 2 C Manual Version 1 05 deleted this R1 0 The referenced section in the poll worker s manual discusses sending the voting results by electronic transmission to the tally site This appears inconsistent with other statements that no networking or wireless is used Vol Il Sec System Operations Procedures R2 0 Ballot Loading and Pre election 2 8 4a Operational Features Manual Ver 5 2 6 Section 3 clarifies to refer user to Maintenance and Repair The vendor shall provide Manual for calibration instructions documentation of system Verified Maintenance and Repair operating features that meets the Manual Ver 1 05 Section 4 includes following requirements instructions R1 0 Provided in ballot preparation a Provides a detailed description manual of all input output control and in Ballot Loading and pre election display features accessible to the manual page 5 provides the option to operator or voter run the calibration test but tells the user not to do it then assumes it was executed in the following paragraph
62. l operation of the system These discrepancies were reported to AI Technology and tracked as test exceptions until CIBER verified the applicable documents had been corrected Upon final review of the aforementioned documents and receipt of the Wyle report of their review Wyle report 52667 01 CIBER concludes that the TDP submitted by AI Technology meets the requirements under the FEC standards of 2002 Appendix A TDP Review provides more detailed information about the reviewed documents 5 1 2 Source code review summary The code was reviewed in order to evaluate its compliance with the FEC standards for source code These standards are intended to ensure that the overall objectives for logical correctness system integrity reliability and accuracy are being met It was also reviewed for its adherence to any AI Technology coding standards CIBER Version 1 2 April 10 2006 Proprietary 10 CIBER reviewed the source code and provided the vendor with a list of deficiencies The vendor corrected the deficiencies and re submitted the source code This review correct process continued until all deficiencies were successfully resolved Appendix B identifies the types of deficiencies that CIBER found in each component Appendix B also provides a list of all files submitted by the Vendor The Vendor corrected all deficiencies in the final delivery CIBER determined that the EMS OVT and FFVT source code conforms to the FEC 2002 voting system standar
63. llot Data 01 3pg 02 3pg 03 3pg 04 4pg 05 5pg 06 3pg 07 3jpg 08 5pg AUTORUN inf Configure txt create proc sql Create tables sql CIBER Version 1 2 April 10 2006 116 3152 92 340 657 19 Proprietary KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB HA 1 11 2 10 26 2 12 2 2 12 2 2 12 2 2 12 2 2 12 2 2 12 2 2 12 2 2 12 2 2 7 10 1 9 23 2 10 21 2 11 1 2 2 25 2 5 27 2 11 16 2 12 7 2 5 26 2 12 7 2 12 7 2 4 5 2 12 7 2 12 7 2 12 7 2 12 7 2 12 7 2 3 18 2 12 7 2 1 30 2 12 7 2 5 31 2 12 7 2 12 7 2 12 7 2 12 7 2 12 7 2 2 22 2 HA 6 18 2 6 18 2 6 18 2 6 18 2 6 18 2 6 18 2 6 18 2 6 18 2 8 9 2 9 6 2 7 28 2 7 1 2 O O O OL O O O O O O LO SO O OO GO 0 00 0 0 0 0 0 0 0 000 O 0 0 O O 0 00 O O O OO O O OD DUO O 05 05 05 05 05 05 05 05 05 05 97 05 04 04 05 05 04 05 05 OS 1d 05 05 05 11 05 1 05 11 05 ti 95 li 05 1 05 05 05 05 05 Tl 05 11 05 11 05 LI 05 11 07 05 03 03 03 03 03 03 03 03 01 02 05 05 10 07 06 06 06 06 06 06 06 06 TAs 11 02 LAs 05 05 08 07 07 07 07 07 07 07 07 03 08 05 03
64. llot formats and g Ballot preparation edit listings ITA Note also included in Functional Checklist R3 0 verified information has been added The System Administrator Manual contains the information about the Event Log module Vol Il Sec 2 8 Technical Data Package System Operations Procedures This documentation shall provide all information necessary for system use by all personnel who support pre election and election preparation polling place activities and central counting activities as applicable with regard to all system functions and operations identified in Section 2 3 System Functionality Description above The nature of the instructions for operating personnel will depend on the overall system design and required skill level of system CIBER Version 1 2 April 10 2006 R2 0 Polling Official s Manual Version 1 05 has been received R1 0 Poll workers Manual is referenced and not submitted yet This may be the Polling Officals Manual Proprietary 19 2002 Description Reqt Ciber Comments Avante Comments operations support personnel Vol Il Sec Technical Data Package System 2 8 Operations Procedures The system operations procedures shall contain all information that is required for the preparation of detailed system operating procedures and for operator training including the sections listed below 2 81 Introduction 2 8 2 Operational
65. n the computer monitor as it would appear on the FFVT The election official can view and interact with the ballot as the poll workers and voters will on election day This preview provides a close approximation of the ballot appearance on the standard computer monitor but a final review on the actual FFVT is necessary for complete verification of the ballot o The election officials use the Load Ballot Data program to transfer the election database into the FFVT and OVT from the CD created with the Manage Ballot Data program The Load Ballot Data program is loaded in response to prompts from the FFVT during initialization and will auto load when inserted into the OVT The user must enter the database name and correct login This program also is used to restore a database to the EMS host computer when the Election CD is used as a backup media e Election Day o The election day operations are controlled by the options selected during election setup as desired for that jurisdiction In general the polling place contains a VID computer that is used to generate a VID card for each voter The voter inserts the card into the FFVT which selects the correct ballot for that voter s precinct location and if appropriate political party The unit contains an audio unit for assisting handicapped voters The voter selects the desired language from a list of languages provided by that jurisdiction and then votes At the end of t
66. ner Canon DR 5020 VOTE TRAKKER EVC308 FF DRE Voting Machine Version 1 2 0 Election Management Software EMS Version 5 2 9 Event Log 4 0 4 Generate Ballot Data 5 2 9 Generate VID 4 2 0 Load Ballot Data 3 5 7 Tally 4 1 6 Manage Ballot Data 3 8 6 Test Voting 1 2 Optical Vote Trakker Rev 1 5 0 Count Ballots Rev 2 00 4 FFVT firmware version 1 2 0 CIBER Version 1 2 April 10 2006 Proprietary 50
67. ng The database becomes the active database for all other EMS applications This application is also used to transfer the completed election to Compact Disk CD for loading of the FFVT s and the OVT s During that transfer it does a final validity check of the election database and terminates the transfer with warning messages if it detects an error or omission in the election definition o The Generate Ballot Data application provides access to the selected database for defining election parameters These include the jurisdictional organization of districts and precincts and the contests and candidates that are to be included in the election The election official also defines parameters controlling the appearance of the ballot and operational characteristics of the FFVT and the assignment of FFVTs to polling places When the user exits this application it performs a validity test of the users selections and prints warning messages for conditions that it determines are potential problems o The Generate VID program is used to generate VID numbers that are issued to the voters on election day The numbers are assigned randomly and are appended with information at each polling place so that the resulting number is always unique across the entire election CIBER Version 1 2 April 10 2006 Proprietary 8 o A Test Voting program duplicates the logic and display characteristics of the FFVT in the EMS host environment The screen appears o
68. ns approved for Version 4 7 5 Note development test specs and qualification test specs per Vol II Sec 2 7 Vol ll Sec g User system operations R3 0 verified All user manuals have been submitted in 211 19 procedures Update is expected the CD Vol ll Sec i Personnel deployment and R3 0 verified footer has been updated Updated the footer This change is 2 1 1 1 1 training requirements R2 0 update received Part of footer included in Rev A of Section 10 TDP states Technical Data Package 5 2 5 5 2 5 and part states Technical Data Package 5 2 5 Rev A Vol II Sec j Configuration management R3 0 verified Section 11 and Appendix were included 2 1 1 1 j plan Update is expected in the CD that Tom has Vol ll Sec I System change notes R3 0 verified receipt Documentation list has been updated to 2 1 1 1 1 R1 0 The documentation list refers to identify the change notes for 5 2 5 version 4 7 12 change notes Change notes for 5 2 5 should be identified CIBER Version 1 2 April 10 2006 Proprietary 15 2002 Description Ciber Comments Avante Comments Reqt Vol Il Sec System Overview System R3 0 Verified the pictures have been Pictures are provided This change is 221a Description included in Rev A included in Rev A of Section 2 TDP 5 2 5 Some pictures were not provided where The system description shall they were referenced and blank space include paragraphs drawings
69. ocuments The following table shows all issues that were identified by CIBER and the responses by AI Technology Each cycle of comments responses was assigned a revision number R1 R4 The Ciber Comments column identifies the revision on which the associated comment was reported to AI Technology In this table the term Avante refers to AI Technology Inc Note During the functional testing of the system additional discrepancies were identified and reported to the Vendor These items were entered into the functional test exception reporting log as TDP Issues and then tracked until CIBER verified the corrected document CIBER Version 1 2 April 10 2006 Proprietary 14 2002 Description Ciber Comments Avante Comments Reqt VolllSec Technical Data Package Scope R4 0 closed based on Avante We think that all necessary documents 21 explanation that date is completed date were included in the CD disc which was This section contains a description by Avante given to Tom by Dave Alampi Please of vendor documentation relating R3 0 Please explain the dates in your provide us the list of documents that are to the voting system that shall be List of Documentation Are these the still missing submitted with the system as a electronic file date precondition of qualification R1 0 A list is provided Not all testing documents in the list were submitted Some of these were not changed from
70. on o Set tally level to state in order to tabulate multiple counties o Import OVT totals o Validate OVT imported correctly Use an available FFVT to enter and tabulate votes for election database fnGE01_050930 Enter ADA and regular votes Export the tabulation file Create qualified write in candidate Use an available FFVT to enter votes on the fnGE01_050930 Include ADA votes provisional and regular votes Import FFVT Tabulation file from FM Validate provisional votes Print all tally reports and verify accuracy Clear all tally results and conduct recount by scanning barcodes on voting receipts Consolidate voting machine logs Print reports and logs oo O O 00 00 CIBER Version 1 2 April 10 2006 Proprietary 44 Test Case fnMaxCandidates Configuration General Election e 4Precincts 1 split Definition 1 State 2 counties e 5 political parties 60 candidates in one contest 3 municipalities 3 polling places absentee e 1 statewide contest 3 assembly district contests 2 senate ballots district contests 3 county contests 5 municipal contests 3 supervisory district contests one bond issue e Contest types N of M Straight Party Ranked Recall Judicial Voting Devices Utilized FFVT and OVT Procedures Select the previously exported fnGE01_050930 database delete all existing tally results from the database In Generate Ballot data o Add candidates to the N of M race to get a total o
71. r Microsoft Agent o Spanish Lernout amp Hauspie TruVoice engine 2 0 licensed under Microsoft Agent Support Equipment The following equipment is COTS equipment that was used in this test but may be replaced by equivalent equipment that conforms to the specifications provided in the vendors TDP e BarCode Scanner Hand Held Products Inc Model 3800PDF 12 Serial Number X 48 07189 for scanning voter paper record barcode e Dell Latitude D500 Service Tag Number BFSN31 Ballot preparation and Tally e HP Onmebook XE3 Serial Number TW11625720 VID Card Generation e e Machines T2042 Serial Number QIP2CC0203013 OVT Computer e Dell Dimension 2400 Serial Number BNVC351 OVT Computer e HP Onmebook XE3 Serial Number TW11625720 VID Card Generation e Lexmark Printer T520 Serial Number 9909TM6 e Access 2003 to enter language translations and to maintain test records Documentation This computer was used until it had a catastrophic failure of the hard drive on 2 26 06 At that time it was replaced by the Dell Dimension 2400 that follows this entry in the equipment list CIBER Version 1 2 April 10 2006 Proprietary 5 The documents with a document date of On File Previous Release were not changed and therefore not resubmitted with this release Document dates for TDP Sections are the CIBER file date TDP Document Name Rev
72. rce code changes for the programs that reside on the OVT work station CIBER conducted functional testing of the EMS software reviewed all documents in the Technical Data Package that the vendor had modified and inspected the changes to the source code Following functional testing of the components and the recommendation by Wyle Laboratories that the FFVT 1 2 0 met the 2002 voting standards CIBER conducted system tests to verify the integrated operation of all system components in simulated elections 1 3 DOCUMENT OVERVIEW This document consists of five main sections Introduction Qualification Test Background System Identification System Overview and Qualification Test Results The Qualification Background gives general information about the qualification test process The System Identification Section gives information about the AI Technology software and supporting hardware The System Overview describes the software and the Qualification Test Results Section provides a summary of the results of the testing process Wyle Laboratories performed qualification testing of the FFVT Voting machine and prepared a separate report describing that testing Wyle report number 52667 01 This document describes the testing performed by CIBER which includes testing the integrated operation of the full system CIBER Version 1 2 April 10 2006 Proprietary 3 2 QUALIFICATION TEST BACKGROUND The primary purpose of Software Qualification Testin
73. rder frx 1 frmCandidateRotation frm 38 frmCandidateRotation frx 1 FrmChangePass frm 10 frmContestGroup frm 23 frmContestGroup frx dl frmContestPosition frm 107 frmContestPosition frx 1 frmCounty frm 26 frmCounty frx 1 frmDateTime frm 9 frmDistrict frm 23 frmDistrict frx 1 frmEntry frm 8 CIBER Version 1 2 April 10 2006 Proprietary KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB KB 12 8 2004 3 23 2005 9 7 2005 11 16 2004 12 8 2004 12 10 2 7 29 2 10 18 2 7 29 2 7 29 2 3 16 2 7 29 2 4 4 2 7 29 2 7 29 2 4 4 2 7 29 2 7 29 2 7 29 2 1 11 2 10 25 2 10 14 2 10 17 2 4 4 2 4 4 2 9 30 2 11 1 2 12 2 2 9 28 2 12 2 2 3 16 2 12 2 2 7 29 2 12 2 2 7 26 2 12 2 2 7 26 2 12 2 2 7 29 2 12 2 2 12 2 2 7 26 2 12 2 2 10 17 2 12 2 2 7 26 2 12 2 2 12 2 2 10 17 2 12 2 2 0 O O O O O O O 0 00 00 SO O O OO O O O O Or O D 0 0 0 0 0 00O 0 0 0 0 0 0 0 O 00 O L O 04 05 04 05 05 05 05 05 05 05 05 05 05 05 02 05 05 05 05 05 04 04 05 04 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 05 04 02 01 08 04 01 04 01 04 04 04 04 04 04 04 04 04 04 04 04 10 0 54 123 06 06 09 11 06 09 06 04 06 05 06 08 06 08 06 05i 0
74. s described in their report number 54667 01 CIBER conducted the source code review of the FFVT firmware See Section 5 1 2 for discussion of the source code review After Wyle determined that they could recommend the FFVT for Federal Certification CIBER conducted system tests that included the FFVT model that Wyle was recommending for certification See Section 5 1 3 for discussion of system testing The OVT contained only COTS hardware components that were included in the previous certification effort and therefore required no hardware testing for this qualification test The Canon Model DR 5020 that is included in the submitted configuration was previously tested by CIBER and subsequently certified by the FEC as compliant to the 2002 standard 5 3 NOTES During testing the OVT paused scanning when a ballot failed to feed correctly The error message that printed indicated a scanner cover not closed error had occurred The tester checked the cover and then re fed the ballot that failed to read The program flashed a second message warning that the scanning activity should be restarted and immediately continued scanning and counting ballots The tester allowed the scanning to complete and verified that all ballots were counted correctly This test was also conducted by restarting the scanning activity which also resulted in correct results The vendor added text to the Optical VOTE TRAKKER Operating Manual clarifying the error and the
75. sessment and Test specifications have not been 2 7 2d maintenance provided ITA shall design and execute test cases for full Functional Testing Vol Il Sec e Ballot interpretation logic Test specifications have not been 2 7 2e provided ITA shall design and execute test cases for full Functional Testing Vol ll Sec f Exception handling Test specifications have not been 2 7 2f provided ITA shall design and execute test cases for full Functional Testing Vol ll Sec g Security and Test specifications have not been 2 7 29 provided ITA shall design and execute test cases for full Functional Testing Vol Il Sec h Production of audit trails and Test specifications have not been 2 7 2h statistical data provided ITA shall design and execute test cases for full Functional Testing Vol ll Sec System Test and Verification Test specifications have not been 2 7 2 Specification Qualification Test provided ITA shall design and execute Specifications The specifications shall identify procedures for assessing and demonstrating the suitability of the software for elections use test cases for full Functional Testing CIBER Version 1 2 April 10 2006 Proprietary 18 2002 Reqt Description Ciber Comments Avante Comments Voll Sec 2 2 5 1 System Audit System Audit Purpose and Context The requirements for all system types both precinct and central count are described in generic language Beca
76. sic Noted amp Corrected Anomalies 4 2 3a Module does not have a single purpose 4 2 Tal Module header needs a purpose 4 2 7a2 Module header needs comments for units called 4 2 7a3 Module header needs comments for input and outputs 4 2 7a4 Module header needs comments for file referenced 4 2 7a5 Module header needs comments for global variables 4 2 7a6 Module header needs revision history 4 2 7b Variables objects and data types need comments upon declaration 4 2 7c Code needs in line comments 5 4 2f Case statement with no default case 5 4 2g Possible vote counter overflow 5 4 2h Indenting is inconsistant 5 4 2k Lines is more than 80 columns in width 5 4 2p Message not self explanatory or error message like the status message 5 4 2r Over 5 levels of indenting 5 4 2s Variables must be initialized where declared where permitted 5 4 2u Constants should be defined enumerated or have comments explaining its meaning Component Language Full Face VOTE TRAKKER 1 2 Visual Basic Noted amp Corrected Anomalies 4 2 3b Not legal name 4 2 Tal Module header needs a purpose 4 2 7a2 Module header needs comments for units called 4 2 7a3 Module header needs comments for input and outputs 4 2 7a4 Module header needs comments for file referenced 4 2 7a5 Module header needs comments for global variables 4 2 7a6 Module header needs r
77. tem Setup The system is delivered by AI Technology with the software pre installed for the user The system is not to be attached to any office network telecommunication links or the internet e Reports The EMS provides reports of contests candidates and jurisdiction organization The EMS Tally program reports election results at multiple jurisdictional levels precinct polling place jurisdiction and voting sources absentee provisional normal It provides for viewing and printing of FFVT logs The EMS Event Log application provides for viewing and printing of all event logs that are maintained by each EMS application and the FFVTs The FFVT provides required poll reports and the OVT reports vote totals at the subprecinct level and for totals at that OVT All components maintain logs which can be displayed and transferred to the EMS for viewing and printing e Election The system supports general elections and open and closed primary elections The system supports multiple geographic entities with consolidation at multiple sites and or a single central site The system supports a variety of contest types including N of M contests ranking contests recall questions judicial contests and cumulative contests e Election Setup utilizes the EMS as follows o The election official uses the Manage Ballot Data application to create a database in which a new election will be defined or to select an existing database for editi
78. ther documents thought that was removed in this version and a vid card used Proprietary Rev B 16 2002 Description Ciber Comments Avante Comments Reqt Vol Il Sec Software Design and Specification R2 0 Maintenance and Repair Manual 2 5 10 Appendices has been received and references have been verified The vendor may provide R1 0 References to section 9 6 which descriptive material and data is not yet provided note this is at the supplementing the various discretion of he vendor and as such will sections of the body of the be validated when section 9 6 is Software Specifications The provided with referenced information content and arrangement of Note Appendix J and other appendixes shall be at the appendices satisfy this also discretion of the vendor Vol Il Sec System Security Specification R2 0 System Security Specification 2 6 2 Access Control Measures 5 2 5 Rev A Section 6 2 added references to OVT Operating Manual The vendor shall provide a R1 0 The use of BID number for the detailed description of all system OVT is not clear The statement does access control measures and not indicate what the bid number must mandatory procedures designed to match to be valid This is explained in permit access to system states in the OVT operating manual but needs to accordance with the access policy be added here or a reference to the and to prevent all other types of OVT manual
79. translated and to enter the translated phrases o Generating of paper ballots at the OVT and of all reports utilizes Microsoft Word 5 QUALIFICATION TEST RESULTS 5 1 ELECTION MANAGEMENT SOFTWARE AND ELECTION SYSTEM This section describes the testing of the EMS software including ballot preparation software and central tally software The tests included TDP review source code review and functional test of the EMS This section describes the functional and system tests conducted by CIBER Section 5 2 summarizes the hardware testing conducted by Wyle Laboratories and Section 5 4 presents the ITA recommendation 5 1 1 Technical data package TDP review summary The TDP contains requirements design configuration management quality assurance and system operations information The AI Technology TDP consists of the documents listed in Section 3 CIBER reviewed the TDP documents associated with the EMS and OVT for accuracy completeness and compliance to the 2002 Voting System Standards See Wyle report 52667 01 for a discussion of the FFVT TDP review The review results were recorded in a worksheet that provided the pass fail compliance to each applicable Voting System Standard The Vendor corrected each non conformance observation and resubmitted the associated documents for re review This process continued until the TDP was in compliance with all TDP Standards Functional testing also identified text in the TDP that conflicted with actua
80. upervisory district contests one bond issue e Contest types N of M Straight Party Ranked Recall e Candidate rotation by District e Regular ballots Absentee paper ballots Provisional Ballots After Hour Provisional Ballots e English only text e Multi line write in Voting Devices Utilized OVT FFVT Procedures e Create election database using the Manage Ballot Data application e Generate a Ballot using Generate Ballot Data as follows o Define levels as State County City Precinct and Ward o Enter voting districts as Statewide Alabama Assembly District AD1 AD7 AD32 County Madison Jefferson City Madison Birmingham Clanton Senate Districts 1 and 2 Supervisory Districts 1 3 5 Enter 5 political parties Enter 4 precincts one with two subprecincts Save Template of precinct district party configuration Enter registered voter counts Enter contests and setup contest groups Add candidates to contests and order candidates Enter 2 qualified write in candidates Enter bond issue Setup 3 rotation groups for assembly districts and Cities set up precinct rotation for contests not in those districts o Set system settings for voting machines voting strategy voter paper receipt format 000000000 e Preview ballots using the voting test application and modify ballots and system settings as necessary e Use Generate VID application to generate maximum number of Voter Identification
81. use the actual implementation of specific characteristics may vary from system to system it is the responsibility of the vendor to describe each system s characteristics in sufficient detail that the ITAs and system users can evaluate the adequacy of the system s audit trail This description shall be incorporated in the System Operating Manual which is part of the Technical Data P ackage TDP ITA Note documentation requirement user manual design documents R3 0 verified information has been added Information about system audit Event Log is included in System Administrator Manual Version 5 2 6 Rev A Optical Vote Trakker Manual and Tally and Reporting Manual Voll Sec 441 Software Standards P re election Audit Records During election definition and ballot preparation the system shall audit the preparation of the baseline ballot formats and modifications to them a description of these modifications and corresponding dates The log shall include a The allowable number of selections for an office or issue b The combinations of voting patterns permitted or required by the jurisdiction c The inclusion or exclusion of offices or issues as the result of multiple districting with in the polling place d Any other characteristics that may be peculiar to the jurisdiction the election or the polling place s location e Manual data maintained by election personnel f Samples of all final ba
82. would recommend its certification CIBER conducted end to end System Tests to validate the integrated operation of the EMS FFVT and OVT The test cases exercised the overall ballot preparation voting and tabulation software as it would be used in actual elections allowing the new functions to be validated as part of the overall operation of the application Approach The ITA prepared a software test plan that identified the objectives of the test and the test cases that were required to validate those objectives The planned test cases were expanded and modified during the testing to ensure all identifiable potential deficiencies exceptions were tested and that all modifications were validated All exceptions were reported to the vendor and tracked by CIBER to ensure they were resolved The test cases that were executed are summarized the tables that follow The results of functional and system testing verifies that the AI Technology VOTE TRAKKER 1 2 0 voting system conforms to the FEC 2002 Voting System Standards and implements the features specified in the vendor s TDP CIBER Version 1 2 April 10 2006 Proprietary 41 Test Case fnGE01 050930 Configuration General Election e 4Precincts 1 split Definition 1 State 2 counties 3 e 5 political parties municipalities 2 polling places absentee e 1 statewide contest 3 assembly district contests 2 senate ballots district contests 3 county contests 5 municipal contests 3 s
83. y vendor personnel CIBER Version 1 2 April 10 2006 Page 10 11 refers to manuals for the following areas System hardware and networking Election and software application engineers Implementation manager It states that they are part of the TDP but could not determine which manuals they are or if they were omitted from the TDP Proprietary 22 CIBER Version 1 2 April 10 2006 2002 Description Ciber Comments Avante Comments Reqt Vol ll Sec Configuration Management Plan R3 0 verifed this has been deleted Removed Instand Runoff This change is 2 11 2a Configuration Identification R1 0 This section identifies a instant included in Rev A of Section 11 of runoff module That reference should Version 5 2 5 The vendor shall provide a be removed if incorrect or the module description of the procedures and included in all applicable sections of naming conventions used to the TDP and associated manuals address the specific requirements of Volume I Section 8 4 8 3 Configuration Identification of the Standards These requirements pertain to a Classifying configuration items into categories and subcategories File note Should read Volume Section 8 3 Vol ll Sec c Procedures and training R4 0 Verified Appendix Vote Trakker The Maintenance and Repair Manual was 2 11 7 G materials for using the tools CM Plan included in the CD that Tom recevied from R3 0 1 Maintenance an

Software Qualification Test Plan

Contents

Download Pdf Manuals

Related Search

Related Contents