Home

PDF file - Deep Blue - University of Michigan

1. BY Cc DIET F G Size Allocated Percent Files Description 2 Container Files 241 4MB 241 4MB 51 2 3 Compressed Archives and disk images _3 zip 169 3 MB 1693MB 35 9 2 Compressed zipped Folder 4 tar 72 1MB 721MB 15 3 1 TAR File 5 Mail Files 153 0 MB 153 0 MB 32 5 18 Email messages and files of email clients 6 pst 746MB 746MB 15 5 1 Outlook Data File 7 mbox 73 1MB 73 1MB 15 5 8 MBOX File 8 eml 53MB 53MB 1 1 7 E mail Message _9 msg 0 0 MB 00MB 0 0 2 Outlook Item 10 Graphic Files 39 2 MB 39 2MB 68 3 8 Files containing pictures images or mouse cursors 11 arw 14 3MB 143MB 3 0 1 ARW File 12 orf 116MB 116MB 25 1 ORF File 13 psd 50MB 50MB 11 1 IrfanView PSD File _14 bmp 41MB 41MB 09 1 IrfanView BMP File Compare Age of Files The Age of files option generates a bar graph representation of the age range for content identifying material produced within the year from 1 to 5 years ago from 5 to 10 years and then older than 10 C BHL unprocessed 87209_0001 on OS Last Change Date Review Duplicate Content The Duplicate content option uses MD5 checksum comparison to identify duplicate material and then produce a CSV report The Bentley Library is not going to do mass de duplication due to the difficult and labor intensive analysis required to identify the record version of content To maintain original order and prevent the potential loss of record versions of content you may allow dupli
2. El Export Logical Image AD1 de Add to Custom Content Image ADI You will then be prompted to select a destination for the extracted files to maintain original order choose the same directory that holds the original disk image file Select the destination folder di di unprocessed 4 87209 0001 Ji files b de dc_scripts IN FTK Imager will immediately begin to extract files from the disk image AutoPro User Manual v 3 0 13 Export Files Copying files C BHL unprocessed 87209_000 1 flles Session 1 Track 01 SPOTLIGHT When the operation is successfully completed you will receive a notice Export Results 8 folder s and 47 file s exported successfully SL 2337048576 bytes copied You may now remove the disk image or evidence item from FTK Imager select the image file and either click the remove evidence item from the toolbar or right click and select the option from the menu You are now ready to proceed to the next disk image A AccessData FTK Imager 3 0 1 1467 E File View Mode Help Evidence Tree RE 7 EEE fife SPOTLIGH amp Remove Evidence Item Once you are done extracting content from the disk image file s you may enter C into the AutoPro interface to COMPLETE this step If you need to quit and resume file extraction at a later date enter Q and recommence this procedure the next time you log into AutoPro Upon completing the step AutoPro will
3. bhidesign css File bhllayout css Pane bhiprint css index html g intro_webarchives html G printlogo jpg UMwebarchives html Home 6 webarchives html Research AutoPro User Manual v 3 0 24 While viewing a file you may search for text in the ribbon s text box zoom in out for images only and adjust the document size so that it fits entirely in the view are or is fit to the area s width IrfanView When IrfanView is selected AutoPro will open the application in thumbnail view with the main processing directory selected in the left hand folder pane It will be necessary for you to navigate down into the directory hierarchy to the folder s that contain image galleries IrfanView Thumbnails or ka File Options View Stop Exit 3 bhl root bhl root m Temporaryltems snapshot C backup deepblue_deposits Once you have selected a folder with images IrfanView will automatically load thumbnails of all image files into the viewing pane You may then click on a thumbnail to take a closer look at the full scale or use the navigation pane to browse to other folders IrfanView Thumbnails or File Options View Stop Exit ZAunprocessed test 201 1113_0001 images raw som Documents W i ch A FS Cow 4 Im Inkscape Inkscape may be used to view vector image files such as SVG Al WMF etc that
4. it is therefore highly recommended that you 1 take notes throughout this procedure to help with separations and gaining intellectual control of content and 2 establish the intellectual arrangement before completing any arrangement or packaging of files for deposit While digital processing should be done as efficiently as possible it is very important that you do not rush through this step If your collection lacks an appropriately detailed finding aid and descriptive metadata researchers may not be able to discover valuable information therein You may quit this procedure at any time and return to complete it at a later date AutoPro User Manual v 3 0 20 AutoPro will display an overview of goals for this procedure some of these actions may be inapplicable i e no content needs to be separated or no physical arrangement of files is necessary Note that if you have an archived website in the deposit you will create a list of one or more resources that will be converted into hyperlinks to content for a BHL access page that AutoPro will automatically create in the next procedure Bentley Historical Library Digital Curation Division AutomatedProcessor vw H F c 2013 APPRAISAL ARRANGEMENT and PACKAGING FOR DEPOSIT This procedure will permit you to 1 Conduct an appraisal and review of content 2 Record manual conversions of files if necessary 3 Separate unneeded materials via right click menu options see manual for
5. s digital desktop Given the steep increase in born digital and digitized content accessioned by the library in recent years archivists have sought more efficient and standardized processing procedures The Andrew W Mellon Foundation funded MeMail Project 2010 2011 provided the library with resources to establish a workflow and corresponding policies for the ingest and processing of archival email but a similar solution was needed for mixed digital content i e Office documents PDFs audio and video files images etc Archivists in the library s Digital Curation Division have advanced the work ofthe MeMail Project in developing the AutomatedProcessor or AutoPro a series of inter dependent Windows shell scripts that automate key steps in preparing digital content for long term preservation and access Digital Processing as a Concept and Approach at the Bentley Library Archival processing in the digital era requires traditional steps such as appraisal arrangement and description in addition to procedures that ensure the authenticity integrity and security of content Digital processing therefore corresponds to the generate AIP function of the Open Archival Information System OAIS Reference Model s Ingest entity After a Submission Information Package SIP has been assigned an accession record digital processing permits archivists to assume intellectual control establish the integrity of materials and perform preservati
6. A decision was made to package each of the audio recordings i e both the original file and preservation copy created by the library into separate ZIP files so that users could access a single event without having to download a very large file that contained recordings of all the programs The materials were placed in Deep Blue in accordance with the intellectual arrangement and deposit plan AutoPro User Manual v 3 0 32 Program Director s Files Leadership Assembly Toward a Fair Michigan 2013 N 2 8MB Program Director s Files Litigation Toward a Fair Michigan 2013 1 5MB Program Director s Files Organizational Documents Toward a Fair Michigan 2013 270 9KB Program Director s Files Press Toward a Fair Michigan 2013 752 2KB Program Director s Files Programs Audio Files Benton Harbor Debate Toward a Fair Michigan 2013 N 902 8MB Program Director s Files Programs Audio Files Ellis Cose Colloquium Toward a Fair Michigan Cose Ellis 2013 h 763 6MB Program Director s Files Programs Audio Files Grosse Pointe Unitarian Debate Toward a Fair Michigan 2013 N 735 5MB Packaging Content with AutoPro zipOneFolder bat To zip a single folder and all files and subfolders therein into a single uncompressed ZIP file right click on the folder and select the batch file AutoPro zipOneFolder bat from the Send to context menu Cut gt unprocessed 8720
7. cannot be opened in IrfanView of Quick View Plus AutoPro will open Inkscape and you will then need to click on the open file icon or use File gt Open in the navigation menu to access vector image files AutoPro User Manual v 3 0 25 CAENLetteringCS2 ai Inkscape Ea File Edit View Layer Object Path Text Filters Extensions Help Gas AGiee SC Daa ex J 2 fia LE se EB Et looo H vioo00 woo A mE iat Ed I Poser A e ie Qa a w 1 2 a O ne D gi T i 1 x 4 i t gt gt E E m 3 5 4 oll H Fill N A un E A Xx 2934 18 pa 100 CAENLetteringCS2 click to sele d 12 en a o 100 a di ingCS Ir Alt click to select und z Z 12 VLC Media Player AutoPro will open VLC Media Player after which you may use the Media menu item to open a single file or an entire folder in which case all audio video files within the folder will be added to a playlist VLC media player lee Playback Audio E Open File Cero r Advanced Open File Ctrl Shift 0 EG Open Folder Ctrl F Open Disc Ctrl D a Open Network Stream Eri nai Open Capture Device trl L Tools View Help Video Play controls are located at the bottom of the Media Player window in addition to Play Pause and Stop buttons the archivist may fast forward or reverse progress by adjusting the s
8. continue until all the files on the list have been reviewed with the addition of any extensions recorded in a log file Step 5 Format Conversion The creation of preservation copies for content in at risk formats is one of the Bentley Historical Library s primary preservation strategies AutoPro searches for at risk formats using the matrix developed by the Division of Digital Curation and creates temporary files listing all files associated with the following formats and media types conversion software noted in italics e Raster images BMP PSD PCD PCT and TGA to TIF Image Magick e Raw digital camera image files 3FR ARW CR2 DCR MRW NEF ORF PEF RAF RAW X3F to JPG Image Magick AutoPro User Manual v 3 0 19 e Vector images Al WMF EMF to SVG Inkscape e PostScript files PS to PDF A Ghostscript e Encapsulated PostScript files EPS to PDF A Ghostscript e Portable Document Format PDF to PDF A Ghostscript e Audio files WMA RA AU SND to WAV ffmpeg e Video files FLV WMV RV RM MTS to MP4 with H 264 encoding ffmpeg e DVD encoded video VIDEO_TS folders to MP4 with H 264 encoding Handbrake e Office Documents DOC PPT XLS to Open Office XML MS Office File Convertor e Email PST MSG EML etc to MBOX Aid4Mail After compiling listing of each format media type AutoPro calls subroutines that kick off the various conversion applications using pr
9. extent of the deposit they can be useful in reviewing a large collection of unstructured data AutoPro User Manual v 3 0 21 Review Directory Hierarchy Reviewing the directory hierarchy may aid in the appraisal of the deposit and help you to understand the breadth and scope of content transferred to the library Folder naming conventions and directory structure may also reveal organizational principles if any and areas of particular interest The Directory Hierarchy displays the structure of folders within the deposit dirTree_87209_0001 txt Note File Edit Format View Help Folder PATH listing for volume 05 Volume serial number is 0857 4B60 Cc BHL U NPROCESSED 87209_0001 orrespondence Inbox Sent eeting Materials 2010 2011 View Relative Size of Directories The Relative size of directories will produce a bar graph comparing the sizes of the folders in the main processing directory To examine the relative size of the contents of subfolders simply complete the folder path in the AutoPro interface or enter C to complete the step Compare File Extensions within Deposit The File extensions option produces a list of file extensions grouped by format type images video Office files etc with the number of files for each extension included This information can be useful in understanding the dominant types of materials as well as for noting unique file types AutoPro User Manual v 3 0 22
10. in an Excel spreadsheet that will open automatically using one row for each file Consult with Digital Curation if you need help automating this step If there are a large number of files you may quit this action and finish at a later date Create a List of Resources for an HTML Access Page Archived websites captured by the library with HTTrack or Teleport Pro will be placed in Deep Blue with an HTML document page that describes the content and provides access points to significant information Once this option is selected you will be asked to provide the full path to the folder that holds the archived website If the material is in a domain folder i e www soe umich edu you should create a folder to hold the site i e SOE Website 2008 Bentley Historical Library Digital Curation Division AutomatedProcessor u 8 9 c 2613 IDENTIFY RESOURCES TO BE LINKED TO IM THE BHL ACCESS PAGE At the propt copy and paste the full path to the folder that contains the archived Website gt NOTE Each archived version should be in its own folder this folder should be located immediately within the deposit directory Be sure to make any changes or corrections to the folder name before entering it here Folder C BHL sunprocessed 98564_H6061 SNRE 2008 You will then have an opportunity to record the date of any prior processing and the name of the processor if not applicable hit enter AutoPro User Manual v 3 0 29
11. more information 4 Arrange content via Windows Explorer only if necessary G Package content in ZIP files via right click menu options see manual for more information HOTE For each archived Website you MUST create a list of one or more resources that will be linked to from a BHL access page Press any key to continue m After reading through the instructions you may hit any key and then choose from the following options Bentley Historical Library Digital Curation Division AutomatedProcessor v 9 7 Cc 2013 APPRAISAL ARRANGEMENT and PACKAGING FOR DEPOSIT Please select from the following options CHARACTERIZE DEPOSIT REVIEW CONT EMT 1 Directory hierarchy GO Quick View Plus Relative size of directories U ULC Media Player audio video File extensions formats IRF IrfanUiew raster images Age of Files INK Inkscape vector images Duplicate content OTHER M Record MANUAL conversions L create a LIST of resources for HIML access page C COMPLETE procedure QUIT QUIT AutoPro and resume appraisal and arrangement at a later time Enter an appraisal option _ Characterize Deposit Understanding the Extent and Diversity of Content Items within the Characterize Deposit employ TreeSize Professional and Windows utilities to provide visualizations of various aspects of the deposit While these procedures may not be necessary if you have a clear understanding of the nature and
12. on the collection and material at hand we may package content at the series subseries folder or item level The size of files individually or as bundled in one or more ZIP files To avoid excessive download times for users ZIP packages should not exceed 2 GB if possible Of course very large audio or video files will be deposited as is There is no one way to ready materials for deposit into Deep Blue but the following scenarios will suggest the possible ways in which content may be packaged A deposit with a large number of small Office files related to a single function might be packaged into a single ZIP file which might represent an entire series subseries or folder If the deposit contains records in separate folders that are related to diverse functions each of these subfolders may be packaged as a separate ZIP file In some cases i e a large number of uncompressed TIFF images it may be necessary to divide a single folder into multiple ZIP file packages so that the materials can be uploaded and downloaded efficiently In this case the ZIP file names should contain the original folder name as well as an indication of the breakdown alphabetical or chronological meeting_minutes January April zip or committee_reports_A M zip Very large files video or audio for example or high value content that needs to be described or linked to at the item level will be deposited individually please note tha
13. rerun a virus scan to verify that the newly extracted contents do not contain a virus or malware The program will then return to the Extraction procedure to search for other archive files Archive File Extraction with 7 Zip AutoPro will search for a variety of common archive file formats and if any are found extract the contents of each into a newly created folder that bears the name of the original archive file To avoid name collisions and identify content extracted by the Bentley Historical Library a suffix consisting of bhl_ and the eight character CRC32 hash of the archive file will be applied to the name of this new directory AutoPro User Manual v 3 0 14 If an archive file cannot be extracted with 7 Zip you will have an opportunity to manually extract content using the Windows Compressed zipped folder utility If the archive file is corrupted it may be impossible to retrieve content consult with Mike if issues arise Step 3 Personally Identifiable Information PII Scan Personally identifiable information PII includes Social Security numbers SSN credit card numbers bank account information and passwords any of which may be used to steal an individual s identity or perpetrate fraud in some manner To ensure that the Bentley Historical Library is aware of the presence of PII AutoPro uses Identity Finder DLP Endpoint to search for such material Identity Finder uses regular expressions as well as validat
14. www 7 zip or FTK Imager is digital forensics software produced and freely distributed by AccessData For more information see the FTK Imager manual release notes and manual at http www accessdata com support product downloads DROID is a file identification tool developed by the National Archives U K For more information see http droid sourceforge net TrID is a freely distributed utility that identifies file types based upon a library of over 4 800 binary signatures For more information see http markO net soft trid e html PRONOM is an on line information system about data file formats For more information see http www nationalarchives gov uk help PRONOM fag htm For more information on the Library of Congress s Sustainability of Digital Formats and the FCLA s format recommendations see http www digitalpreservation gov formats index shtml and http fclaweb fcla edu fda format landing page respectively For an overview of sustainable formats and conversion strategies at the Bentley Historical Library see http deepblue lib umich edu handle 2027 42 93307 AutoPro User Manual v 3 0 2 encoding Aid4Mail various email formats to MBOX and Microsoft Office File Converter Office files to Open Office XML These preservation versions are stored alongside the original and denoted by a suffix consisting of _bhl and where possible the CRC32 hash of the original file i e oralHistoryProjec
15. 1 Processing Location z unprocessed 87250_0001 Institution Bentley Historical Library university of Michigan Timestamp 2012 10 23115 24 37 a a en Initiated 1 VIRUS SCAN procedure 1 of 12 Processing archivist Shallcross Michael Timestamp 2012 10 23T15 25 04 Executing virus scan with Microsoft Forefront Endpoint Protection 2010 Total infected files found 0 completed 1 VIRUS SCAN procedure 1 of 12 Processing archivist Shallcross Michael Timestamp 2012 10 23T15 27 08 CEEWwWwwwwwwwrwWwWwWwWwWwWwWwWwWwWwWwWwWwWwWWWwWwWwWwWwWwWrwWwWwWwWWwWwWwWwWwWwWwWwWwwwwWw In addition all preservation events that is activities performed by the archivist that impact the provenance authenticity and or integrity of the content are recorded in a PREMIS PREservation Metadata Implementation Strategies spreadsheet A B E D E G H I J K L M eventType eventldeventldentifierV eventDateTime eventDetail eventOutcome linkingAgenti linkingAg linkingAg linkingAg linkingAgenti linkingAgentRole virus scan UUID d76cfe99 9706 4f 2012 07 27T15 50 2 identification a action complete MARC21 Code MiU H Executor tool Microsoft Ant software decompression of file UUID 84c36d87 d675 4 2012 07 27T15 54 3 decompression action complete MARC21 Code MiU H Executor tool 7 Zip 9 20 software personally identifiabl UUID f119bf41 1b56 47 2012 07 27T15 58 2 identify and rei action complete MARC21 Code MiU H Executor tool Identity Finde softwar
16. 9 KB EJ function 1 elem Finally if the identity match is in a significant record that must be preserved but cannot be redacted it may be necessary to impose an access restriction Please consult with Nancy Mike or a division head to determine the appropriate restriction Step 4 Identify Missing File Extensions To aid in the creation of preservation copies and facilitate the eventual use of content by patrons AutoPro will help the processor identify missing or incorrect file extensions To run this procedure you must make sure that DROID is configured so that it will NOT calculate MD5 checksums If checksums are calculated this procedure may take a very long time to complete and the resulting spreadsheet of information will not be parsed correctly by AutoPro If you don t know or are unsure assume that DROID is set to calculate checksums and answer Y After DROID opens it will check for any updates if any are available approve their installation Next click on the Tools item in the top navigation menu and select Preferences Make sure that the box next to Generate MDS hash for each file is NOT checked Click OK and then close the DROID window AutoPro User Manual v 3 0 17 Tools File Edit Run Filter Repor P3 G Check for signature updates Ctrl U New Open e Export j pload signature file Ctrl Shift U Untitled 1 x 7 Resource Extension Size H Profile Defaults Signatur
17. 9_000 Copy em _ AutoPro bat AutoPro getSubFolderNames bat AutoPro separations extensionRemoval b AutoPro separations onlySelecteditems b AutoPro zipMulti AutoPro zipOneFolder bat v Share with Bu Create shortcut Delete leFilesFolders bat A Name Rename di Correspondence Properties BI Desktop create shortcut ud Meeting Materials 7_T6_ _ _____ mm t Documents This operation may be performed on a top level folder within the processing directory i e 87209_0001 or a subfolder thereof After selecting the batch file anew CMD EXE window will open and you will be asked to supply a descriptive filename without the ZIP extension or simply retain the folder name for the ZIP file The folder itself will be retained in the ZIP file do whatever makes the most sense to you AutoPro User Manual v 3 0 33 Bentley Historical Library Digital Curation Division AutomatedFrocessor u 8 8 cc 2013 gt PACKAGING FOLDER G BHL unprocessedx 78564_B9661 finding alds At the prompt enter a filename without zip extension or to QUIT The ZIP file will be saved to C BHL unprocessed 8564_B061 MOTE Hit Enter to use the folder name as the filename Filename Z Enter m After you verify that the filename is correct AutoPro will package the entire folder in a ZIP file preserving the relative path to the folder and generating a ma
18. BENTLEY HISTORICAL LIBRARY ct u nn nun Cagli Wr NIE RETE BEE FE SL o PPEP E BEI BIRRE ai e tei ME EHE ER NN _L__ ME Cosa ie tt Cai Bentley Historical Library Digital AutomatedProcessor v BEE BEE BEE a BEE FEM HE EN EN FEM S S BE NN FE BEE caliente FEM i Tia ie T HE BEE BEE a N NN FEM M NS HE EN NN NR 5 fc BEE BEE BEE a en JE FE N a nn BEE a nz BEE BEI E Calia Curation Division 613 An AutoPro User Manual and General Reference Version 3 0 October 29 2013 Prepared by Michael Shallcross Division of Digital Curation I ERO CII TIO Wisco einen 1 IHSEILUIENAL Contex ee Inne Sanaa teases beens hu einen 1 Digital Processing as a Concept and Approach at the Bentley Library 1 Overview of the Automated Processing Workflow iii 1 Notes on the Windows Command Prompt aaa teaser 4 2 HMitlate lt a Processing SESSION alici olii 6 AUCH AUPITO ansehe Be tata 6 BepositConirmatibiasaa ilaele lei 7 Processoridentificationiacl A 7 Identification or Archived WebSites an nase en 8 Finale 8 3 Main Ment ana Processins Options a a lecca 9 A Guide to Individual Procedures a ne ua 11 STEH VIPS Seal een DREH Tele 11 STER 2 EC EREL Oo 6 he 11 Disk Image Extraction With PIK Imager aio 12 Archive FIE Extraction With 7 210 ee eier 14 Step 3 Personally Identifiable Information PI SCAN 15 Step 4 Identity Missing File Extens
19. Next enter the full path to a single resource HTML or other file you wish to include as a hyperlink in the BHL access page that AutoPro will create After you verify that the path is correct you will enter a title that identifies the resource for users You will then be permitted to add additional resources if multiple links to content are required Arrangement The Bentley Historical Library strives to respect provenance and maintain the original order of content in order to preserve important contextual information found in the structure of directories and the associations of different files Given the structure that many record creators impose on their files and the importance of original order additional arrangement will be unnecessary in most cases At the same time a basic assumption in our digital processing workflow is that folders in the top level of a processing directory represent key functions or activities of the record creator You may therefore need to create a top level folder in order to organize files or subfolders into series avoid using spaces and or non alphanumeric characters in the names It may also be necessary to impose some organizational principle if the files were copied pell mell from their source location with no structure While manual arrangement may be performed via Windows Explorer you can also arrange content during the packaging process described below Be sure to complete the intellectual arrangemen
20. Pro User Manual v 3 0 39 AutoPro will take this information and create an HTML access page BHL start here html that is stored directly within the ZIP package This file will contain descriptive and administrative metadata related to the site as well as hyperlinks to individual resources identified in Step 6 Reviewing an Item You have the opportunity to review items when you resume step 7 after previously quitting or when you select option R Review Items created for this deposit from the Main Item Options screen During the review process AutoPro will display an item ID generated for internal use by AutoPro title description and status either completed or NOT completed for all the items you have created Please note that ALL items must be formally completed for AutoPro to correctly generate metadata records From this screen you may ADD additional content to an existing item if not complete VIEW the contents of an item or COMPLETE the packaging and description of an item When you select any of these options AutoPro will ask for the item ID number e If you elect to ADD content you will follow instruction outlined in the section Associate Bitstreams with the Item e If you elect to view content AutoPro will display the filename and description for all content associated with the item before returning you to the Item Review screen AutoPro User Manual v 3 0 40 e f you elect to co
21. Upload signature file Ctrl Shift U Preferences 7 Resource Extension ti Profile Defaults Signature Updates Export Defaults Ctrl Shift P Untitled 1 x Binary Signature File DROID_SignatureFile_V65 Container Signature File container signature 20 120828 Analyse contents of archive files zip tar gzip Generate MDS hash for each file After you confirm the correct settings DROID will run and produce a report this step may take a long time for large collections DROID also has a tendency to hang on very small deposits contact Mike if this occurs You may need to enter CTRL C to interrupt the procedure and then answer N when you are asked if you want to terminate the batch process Step 8 Transfer to Long Term Storage This and the following workflow steps will only be completed by Mike Nancy or other staff with access to the Deep Blue deposit folder and the BHL Dark Archive Be sure that you are connected to the appropriate repositories and logged in to the UMROOT domain with your Windows AD password before beginning Transfer to Deep Blue Deposit Folder Upon initiation of this step AutoPro will check if the materials will be deposited in Deep Blue if so AutoPro will also ask if the deposit requires additional email processing via procedures developed by MLibrary LIT Finally you will enter the drive letter where the deepblue_deposits folder lives AutoPro will t
22. als but archivists may group unorganized content in directories or package content in ZIP files to simplify the management and storage with such actions recorded in log files Archivists also develop a plan as to how content will be deposited in Deep Blue in a manner that is both convenient to end users and in accordance with the intellectual arrangement of material in the finding aid Once the arrangement is established AutoPro calls DROID to extract technical metadata and generate an MDS checksum for all content including files in ZIP archives Archivists then use the AutoPro interface to apply descriptive and administrative metadata to materials This step produces a Dublin Core XML file and Excel spreadsheet used to deposit material in Deep Blue the University of Michigan s DSpace repository Finally AutoPro employs Baglt to transfer a copy of all material and metadata to a i ImageMagick http www imagemagick org script index php is an open source raster image editor Ghostscript http www ghostscript com is an open source interpreter for the PostScript language and PDF documents that may be used to convert the latter documents to PDF A Inkscape http inkscape org is an open source vector graphics editor ffmpeg http ffmpeg org for Windows builds http ffmpeg zeranoe com builds is freely available software used for audio and video recording and conversion Aid4Mail http www aid4mail com is a proprietary email c
23. ape should be used for vector images and VLC Media Player for audio and video content The following will provide a brief overview of how to operate these various applications after opening them via AutoPro Quick View Plus The QVP interface is divided into three main parts in addition to the navigation menu and ribbon at the top of the application window The right portion of the interface holds the Viewing Environment while the left hand side is divided between the Folder Pane on the top and the File Pane on the bottom After QVP opens use the mouse or arrow keys right and left arrows may be used to expand collapse subfolders to navigate to the appropriate directory in the Folder Pane Once the appropriate folder has been selected a list of its contents both subfolders and files will be displayed in the File Pane You may use the mouse or the tab key to move to the File Pane then whatever file is highlighted will appear in the Viewing Environment Please note that very large files especially email may take longer to open EX intro_webarchives html Quick View Plus File Edit View Document Go Window Help w amp e e SWI Ge BEH lt Fit to Window Width text only 7 S CDL Web Arda Bentley v 3 0 web Folder E Case Stu Pane O Conferen RZ 3 Crawl Issi Viewing Environment E Faculty H Going Put H O MARC rec e BENTLEY HISTORICAL LIBRARY UNIVERSITY OF MICHIGAN EJ banner gif
24. cate content to remain in the deposit AutoPro User Manual v 3 0 23 B C D E 1 Name Path Size Last Change Last Access 2 todo txt 2 Files 22 4 KB 8 27 2012 5 58 AM 8 27 2012 5 58 AM 3 todo txt C Users Mike Desktop test FITS openplanets fido ae 19806 11 2 KB 8 27 2012 5 58 AM 8 27 2012 5 58 AM 4 todo txt C Users Mike Desktop test FITS fido openplanets fido ae19306 11 2 KB 8 27 2012 5 58 AM 8 27 2012 5 58 AM 5 ARL SPEC Kit zip 2 Files 30 0 KB 10 3 2012 1 06 PM multiple 6 ARL SPEC Kit zip C Users Mike Desktop test 9999 0001 Back Up Executive Committee Meeting Materials Correspondence 15 0 KB 10 3 2012 1 06 PM 11 1 2012 12 46 PM 7 ARL SPEC Kit zip C Users Mike Desktop test 9999 0001 Executive Committee Meeting Materials Correspondence 15 0 KB 10 3 2012 1 06 PM 2 6 2012 9 53 AM 8 Updated Agenda eml 2 Files 36 6 KB multiple multiple 9 Updated Agenda eml C Users Mike Desktop test 9999_0001 Back Up Executive Committee Meeting Materials Correspondence L 18 3 KB HH 11 1 2012 12 46 PM 10 Updated Agenda eml C Users Mike Desktop test 9999_0001 Executive Committee Meeting Materials Correspondence Literature 18 3 KB 10 3 2012 1 06 PM 10 28 2011 1 37 PM 11 Agenda for News an 2 Files 37 2 KB multiple multiple 12 Agenda for News_and C Users Mike Desktop test 9999 0001 Back Up Executive Committee Meeting Materials Correspondence L 18 6 KB HH 11 1 2012 12 46 PM 13 Agenda for News_and_ C Users Mike Desktop test 9999 0001 Executive Comm
25. ctory name and start the process again being sure to inform Mike or Nancy This simple step can save a lot of time and energy down the line Call number 00132 Bimu C221 2 Language The matenals are in English Repository Bentley Historical Library University of Michigan 1150 Beal Ave You will also identify the creator and collection name these can be changed later in the process so don t worry if you re still determining the exact provenance of the collection or its title Processor Identification AutoPro will record the identity of the processing archivist each time someone logs in to work on the deposit to fully document preservation events and provide a full audit trail of digital processing activities AutoPro User Manual v 3 0 7 On subsequent sessions AutoPro will retrieve the name of the last processing archivist from the main log file and ask you to confirm it see below If you are taking over from another processor respond with N and enter your own Identification of Archived Websites AutoPro will then ask if the deposit contains an archived website Digital Curation will inform you if it does This only applies to complete or partial websites captured with HTTrack Teleport Pro or similar software This initial identification will ensure that specialized steps are taken later in the processing workflow Final Steps Once deposit information and your identity have been verified AutoPro wil
26. d txt All files with this extension will be removed from this directory txt Is this extension entered correctly T N AutoPro will ask you to verify your choice enter N to identify a different extension or Yto proceed with the separation Once AutoPro has an affirmative all files with that extension will be removed to the separations directory At the conclusion of the Appraisal and Arrangement procedure AutoPro will generate a manifest of all files that have been separated from the deposit with figures on the number of files and volume and store a copy of this file with other log files Record Manual Conversions When this option is selected a new screen will open with instructions for recording the original filename the filename of the new preservation access version and the software used to perform the conversion AutoPro User Manual v 3 0 28 Bentley Historical Library Digital Curation Division AutomatedProcessor v 8 9 c gt 2613 ANUAL FORMAT CONVERS 1 OH his step will help record the manual conversion of files At the prompt hit Enter to open a spreadsheet or O to QUIT and return to main menu row of the spreadsheet record 1 The original filename 2 The filename of the new preservation version 3 The software with version number used to perform the conversion Options Q QUIT and resume later S spreadsheet is completed and SAVED CQ 2 GDI am You will record this information
27. d in the Appraisal and Arrangement workflow step they may be extracted at that time If disk image files are detected AutoPro will open a text file with the full paths to each file save locally or refer to it in its location in the decompress folder in the deposit s tmp directory File Edit Format View Help C BHL unprocessed 87209_0001 files test adl C BHL unprocessed 87209_0001 T7 les Ethics Skits 2006 iso After opening FTK Imager 1 click the Add Evidence icon or go to File gt Add Evidence Item 2 check the radio dial next to Image File and 3 click Next Select Source Please Select the Source Evidence Type C Physical Drive Logical Drive Fee C Contents of a Folder logical fileJevel analysis only excludes deleted unallocated etc You may then copy and paste the path to a disk image file in the new window s text box or browse to its location before clicking Finish AutoPro User Manual v 3 0 12 Select File Evidence Source Selection Please enter the source path FTK Imager will now show the disk image in the upper left Evidence Tree panel Expand the disk image file by clicking on the next to its name right click on directory immediately within and select the Export Files option Evidence Tree File List amp Bthics Skits 2006 iso Name A Ur a i Export Files Pea Export File Hash List
28. e message digest calcul UUID 82262b07 9155 4c 2012 07 27T16 09 3 calculation of N action complete MARC21 Code MiU H Executor tool md5deep 4 0 software From left to right this document records A the event type B the event identifier type here a Universal Unique Identifier or UUID C an event identifier value D a time stamp E a description of the event F the outcome G the agent identifier type here the MARC21 institution code H the value or identifier for agent here MiU H the Bentley s MARC21 identifier I the type of agent and then information on the software agent employed J L At the conclusion of this and subsequent procedures you will have the opportunity to enter A to ADVANCE to the procedure M to return to the MAIN MENU or Q to QUIT AutoPro If you choose to quit you will resume processing at the point you left off AutoPro User Manual v 3 0 10 4 Guide to Individual Procedures The following pages will provide an explanation of and guidance for running the individual procedures in the AutoPro workflow Be sure to take your time While processing should be conducted in a timely and efficient manner there is no race Do not try to get through the workflow as fast as you can instead pay careful attention to detail especially in those steps that require user interaction review of content and metadata entry Also despite extensive testing and troubleshooting unique files or deposi
29. e Insert Page Layout Formulas Data Review View Acrok F Cut el E di Cu Calibri cu Aa gt e 9 Wrap Kunde Copy r a r 3 Format Painter zz 2 a er es zj Merge Clipboard Font Alignment F10 hi Fe A B Cc D E G H 1 Deep Blue Donor Donor ID Donation Copyright Abstract 2 2027 42 9 Sue Smith 10982 2012 Copyright Prominent Grand Rapids attorney 3 A If you return to this procedure after having previously started and quit you will have the opportunity to edit this deposit level metadata if necessary Check for Pre Completed Metadata Spreadsheet If this is the first time you are describing the deposit AutoPro will check to see if you have a pre completed descriptive metadata spreadsheet AutoPro User Manual v 3 0 36 This spreadsheet must be completed in accordance with conventions established for depositing content into Deep Blue Some metadata massaging may be required consult with Nancy or Mike if you have questions If you have completed this spreadsheet AutoPro will prompt you to indicate ifthere are any access restrictions and then move on to technical metadata extraction see below Creating a New Item The first time you add descriptive metadata to a deposit you will be immediately taken to the Main Item Options screen From this point you may either create a NEW Item or QUIT and resume description at a later date After you have added metadata to an Item you will ret
30. e Updates Export Defaults Binary Signature File DROID_SignatureFile_V65 Container Signature File container signature 20 120828 Y Analyse contents of archive files zip tar gzip Generate MDS hash for each file AutoPro will then ask you to confirm that these settings have been saved and DROID is closed AutoPro will launch DROID to produce a spreadsheet that identifies files with missing or mismatched file extensions Once this operation is complete AutoPro will generate a DROID report and inform you of the number of files if any identified as having mismatched extensions A text file of these files will then open each line will contain the full path PRONOM unique identifier PUID and mime type if the latter two were identified J candidates 98566 000L ut Notep Free File Edit Format View Help C BHL unprocessed 98566_0001 SNRE 2008 BHL doc fmt 43 image jpeg Advanced user note File extension identification is not required if there is an extremely large number of files with mismatched extensions or you have reason to believe the mismatches were incorrectly identified you may finish the procedure immediately You may also delete lines from the file if you do not wish the associated files to have extensions identified for instance Javascript and CSS files from archived web pages will be incorrectly identified as having an extension mismatch AutoPro User Manual v 3 0 18 If you conti
31. e archivists through the workflow AutoPro tracks the current processing status generates log files for all operations and records PREMIS preservation metadata that will be stored alongside the processed content in a preservation environment Archivists must approve the successful completion of each step and may stop at any point in the workflow and resume their work at a later time After content has been accessioned and deposited in the Bentley Library s secure interim repository a processing archivist receives a local copy and then starts AutoPro to add basic collection level metadata and run a virus scan the University of Michigan employs Microsoft Forefront Endpoint Protection on all work stations AutoPro next searches the deposit for disk image and archive files ISO AFF ZIP TAR RAR etc if any are found a script employs 7 Zip to extract the contents to a directory named after the archive file with the original file paths preserved After verifying the extraction s success AutoPro moves the archive file to a separations directory and records the operations in a log file The newly extracted content is then searched for additional archive files from which the contents are extracted if necessary Some disk image formats AFF AD1 EO1 etc will require the processor to employ FTK Imager to manually extract content AutoPro then runs DROID to search for files with missing or mismatched extensions and the archivist
32. eccccassececauseceeauneees 34 Completing the Appraisal and Arrangement Procedures ccccseecccssssccesesceececeeecseeueceeeesseeeeeees 35 Step 7 Add Descriptive and Administrative Metadata ie 35 Add Deposit Level Metadata ei 35 Check for Pre Completed Metadata Spreadsheet 36 Creatine a New EI Giara 37 Kem LeveiMetiddl Arap orn E E E E 37 Associate Bitstreams with the Item 38 Create an Item for Archived WebSIteS i 39 REVIEW Nella 40 Finalize Description and Technical Metadata Extraction 41 Step 8 Transfer to Long Term Storage ieri 42 Transfer to Deep Blue Deposit FOlder 42 Spiel 43 5 VSL SION FLO cali 45 This work is licensed under the Creative Commons Attribution NonCommercial ShareAlike 3 0 Unported License To view a copy of this license visit http creativecommons org licenses by nc sa 3 0 Qoen AutoPro User Manual v 3 0 ii 1 Introduction Institutional Context Established in 1935 by the University of Michigan Regents the Bentley Historical Library serves as the official archives of the university and documents the history of the state of Michigan and the activities of its people organizations and voluntary associations The library has successfully managed and preserved digital content since the 1997 accession of former University President James J Duderstadt
33. edure Bentley Historical Library Digital Curation Division AutomatedProcessor v 8 9 Cc 2013 MAIN MENU OF PROCEDURES VIRUS SCAN FILE EXTRACTION IDENTIFY MISSING FILE EXTENSIONS FORMAT COMUERS TOM PII SCAN APPRAISAL ARRANGEMENT and PACKAGING FOR DEPOSIT ADD DESCRIPTIVE and ADMIMISTRATIVE METADATA TRANSFER to LONG TERM STORAGE CLEAN UP Ae ST fm Quit the BHL Automated Batch Processing Sequence AutoPro may be completed in one session or may be stopped at any point Quitting an automated procedure mid operation may result in loss of data Please complete a procedure before exiting the sequence CURRENT STATISTICS DEPOSIT 9869132_00B2 FILES 12 VOLUME 31676657 bytes PROCESSING STATUS DEPOSIT 86132 _B G2 Cas of Thu 10210220135 Completed 1 VIRUS SCAN procedure 1 of 7 Completed 4 FORMAT CONUERSIOM procedure 4 of 7 Completed 5 PII SCAN procedure 5 of 9 Sl lidia 6 APPRAISAL ARRANGEMENT and PACKAGING FOR DEPOSIT procedure 6 o Enter an option AutoPro User Manual v 3 0 9 At each stage of the workflow AutoPro will record the procedure name processing archivist initiation and completion timestamps and additional information in the main batch processing log file see example below File Edit Format View Help wW E E E FF FF E F FF W W Ww F amp F F EEEF EEE Initiating the Bentley Historical Library Automated Batch Processing Sequence Digital Deposit ID 87250_000
34. ement main menu to COMPLETE the procedure Step 7 Add Descriptive and Administrative Metadata This step in the processing workflow requires you to add descriptive and administrative metadata for items that will be deposited into Deep Blue You will be guided by the deposit plan you developed in Step 6 The procedure will result in a spreadsheet to batch upload content to Deep Blue a Dublin Core XML manifest of materials and a modified EAD record of administrative metadata for the deposit as a whole Please note that you may quit at any point when given the option and resume at a later date Upon initiation of this step AutoPro will remind you to finish the intellectual arrangement and description that should have begun with the Appraisal Arrangement and Packaging of content see Step 6 By completing an initial draft of the finding aid or catalog record you have a thorough understanding of the relationships between different parts of the deposit and if present previous accessions to the collection Add Deposit Level Metadata AutoPro will require you to enter descriptive and administrative metadata about the deposit in an Excel spreadsheet This information includes e Deep Blue collection handle e Donor AutoPro User Manual v 3 0 35 e Donor ID number e Year of donation e Copyright statement e Abstract describing the creator and the contents of the deposit not the collection as a whole Pe TE File Hom
35. eset parameters to create preservation copies Each conversion involves the following steps e Check to see that a preservation copy has not already been created e Generation of the preservation copy e Validation of conversion success e f successful the conversion is recorded in a log file These sustainable versions of content are stored alongside the original bitstream and are differentiated by the addition of a suffix to the filename bhl_ plus the 8 character CRC32 hash of the original file Thus a standard PDF file named BHL LogoBlack pdf will yield a preservation copy named BHL LogoBlack bhl f14fe8e6 pdf For the conversion of PDF files to PDF A AutoPro employs JHOVE to verify if the original file meets the PDF A 1 a or 1 b specifications The conversion process is largely automated with the exception of the email and DVD encoded video routines In each case AutoPro will guide you through operations with detailed instructions Please consult with Mike if any issues arise Step 6 Appraisal Arrangement and Packaging for Deposit This procedure provides you with an opportunity to gain full intellectual control of the content so that it can be meaningfully described in a finding aid and packaged in a manner that will facilitate its long term preservation management and access Take time to review the files Familiarity with the content is essential for the production of rich metadata and informative finding aids
36. hen check to see what drive letter the dark archive is associated with AutoPro User Manual v 3 0 42 Bentley Historical Library Digital Curation Division AutomatedProcessor u 0 9 cc 2013 TRANSFER TO DARK ARCHIVE MOTE This procedure may only be completed by staff with full access to the bhl root and bhl archive storage locations At the prompt enter the drive letter to which bhl archive the BHL Dark Archive has been mapped i e Z or 73 Do NOT include a colon Options QUIT QUIT and resume transfer at another date Drive letter i e A Y or Z etc QUIT Drive letter Y AutoPro will then package the deposit according to the Baglt specification and then move the material to the dark archive where the bag will be validated Please note that large deposits may take a long time Step 9 Clean Up In the final step of the digital processing workflow AutoPro will delete the processing directory temporary files and logs with an additional option of deleting separations Initiated 9 CLEAN UP This procedure will clean the processing directory after content has been successfully transferred to Deep Blue and or the BHL Dark Archive BE SURE TO CLOSE ANY WINDOWS EXPLORER WINDOWS OR APPLICATIONS THAT MAY BE USING FILES IN THE TARGETED FOLDERS The procedure will package log files and delete the following directories C BHLunprocessed 98567_H061 G BHL tmp 98567_66061 G iXxBHL l
37. ion of information via Luhn algorithms for credit card numbers and dictionaries of known SSN number patterns to reduce the occurrence of false positive results AutoPro immediately initiates Identity Finder the scan may take a long period of time especially if the deposit includes large PDFs or email accounts AutoPro is configured to skip the scans of image audio and video files to reduce processing time At the scan s conclusion you will be asked if Identity Finder produced any search results The identity matches will be displayed in an Identity Finder Search Summary window The following example has five matches the number would read O if no matches had been identified Identity Finder Search Summary Identity Finder has completed its search and found Identity Matches that require your attention Locations containing at least one match Total number of matches across alllocations 5 Please choose how you would like to proceed Signin with Profe Password Reminder AutoPro User Manual v 3 0 15 For no matches you will simply enter N at the AutoPro prompt if there are matches enter Y click the Advanced option and then follow the step by step instructions in AutoPro If there are identity matches you will need to review each one to determine if the match is legitimate or merely represents a false positive All files containing identity matches will appear in the Location Pa
38. ions ai 17 SLED S FOMA Conversion ee ee de 19 Step 6 Appraisal Arrangement and Packaging for Deposit ri 20 Characterize Deposit Understanding the Extent and Diversity of Content cceeccseeeseeeeeeees 21 Review Directory Hierarchy cso El nn 22 View Relative Size Of Dir CtOries ccccccssssssseeccccccceeessececccessaueesseeececeessueuseeeceeeessuausseeeeeesenaeas 22 Compare File Extensions within Deposit i 22 Compare Age OFFIS iniia 23 REVIEW Duplicate Contenti alleate iaia 23 ManualliviReview Contelit alia ahi lara aaa 24 QUICKMIEWP US lle aa 24 HENVIC W eon are ea 25 Neapel 25 VEC Media vecia 26 AutoPro User Manual v 3 0 i Separations Removing Superfluous CONtent iii 26 Resource 1 AutoPro separations onlySelectedltemS bat 27 Resource 2 AutoPro separations extensionRemoval bat 27 Record Manual Conversions 28 Create a List of Resources for an HTML Access Page rei 29 PAV EVEN SENG aio 30 Packaging Content for Deposit and Defining Deep Blue Items ccc cceeccceeeceeecseeeeseeeneseeenees 30 Determining How to Package Content i 31 Example of a Packaging and Deposit Plan 32 Packaging Content with AutoPro zipOneFolder bat ei 33 Packaging Content with AutoPro zipMultipleFilesFOIders DAt cccccccseeecccees
39. ipped folder J Properties Desktop create shortcut d Agendas Documents jhovetest txt 10 26 2012 3 45 P Text E nent _ d SendTo_20121029 zip out txt 10 17 20121 30 PM Text Document tect bat amp test ba output txt 1 45P Text L ment PERA AREE es eg DVD RW Drive D pdfs txt 10 26 2012 2 57 P ext ment Ga bhl archive bhl archive m st umich EL podcast csv 10 30 2012 2 06 PM Microsoft Excel e re Serge TER 1 1 38 P ET Er G2 bhl root bhl root m storage umich edu Z CA ne test2 txt 10 11 2012 5 09 P Text L ment 1 _ youtube txt 10 10 2012 11 37 Text Document 4 KB If creating a new file you must supply a descriptive filename without the ZIP extension AutoPro will package the items in a ZIP file preserving the relative path to the folder and generating a manifest of all AutoPro User Manual v 3 0 34 files therein place it at the top level of the deposit directory and delete the original content If adding to an existing file copy and paste the full path to the file in the window AutoPro will add all the material to the specified file preserving relative paths update the ZIP manifest file and delete the original content Completing the Appraisal and Arrangement Procedures Once you have completed all the procedures in this workflow step appraisal and review of content separations arrangement and packaging select option C from the Appraisal and Arrang
40. it ID the collection ID plus a four digit suffix 1 Right click on the processing directory i e 87209_0001 2 Select AutoPro bat from the Send to context menu ITILIGUE MI NIGI Yy AutoPro getsubFolderNames bat N Can gt AutoPro separations extensionRemoval bat 5 C BHL unprd EEE AutoPro separations onlySelecteditems bat vasi amp AutoPro zipMultipleFilesFolders bat clude in library v ee gt AutoPro zipOneFolder bat FR Rename dj Compressed zipped folder i Properties MU Desktop create shortcut ide 87209 0001 Documents DVD RW Drive D ay G2 bhl archive bhl archive m storage umich edu bhl root bhl root m storage umich edu Z T The AutoPro application will now open hit any key to proceed AutoPro User Manual v 3 0 Deposit Confirmation You will then be prompted to enter and confirm basic information for the deposit including the collection ID creator and title The collection ID will be the string of numbers prior to the underscore in the BHL digital deposit number for example deposit number 00132 0002 has a Collection ID of 00132 If this is a new collection the ID should have been correctly entered by Mike or Nancy However if the deposit represents an addition to an existing collection check the online finding the ID will be the first string of digits in the Call Number If the collection ID is wrong quit AutoPro change the deposit dire
41. ittee Meeting Materials Correspondence Literature 18 6 KB 10 3 2012 1 06 PM 10 28 2011 1 36 PM 14 ccn doc 2 Files 38 0 KB 10 3 2012 1 06 PM multiple 15 ccn doc C Users Mike Desktop test 9999 0001 Executive Committee Meeting Materials Agendas 19 0 KB 10 3 2012 1 06 PM 11 1 2011 1 02 PM 16 ccn doc C Users Mike Desktop test 9999 0001 Back Up Executive Committee Meeting Materials Agendas 19 0 KB 10 3 2012 1 06 PM 11 1 2012 12 46 PM 17 CCE assignment doc 2 Files 49 0 KB 10 3 2012 1 06 PM multiple 18 CCE_assignment doc C Users Mike Desktop test 9999_0001 Back Up Executive Committee Meeting Materials Agendas 24 5 KB 10 3 2012 1 06 PM 11 1 2012 12 46 PM 19 CCE_assignment doc C Users Mike Desktop test 9999_0001 Executive Committee Meeting Materials Agendas 24 5 KB 10 3 2012 1 06PM 11 1 2011 1 02 PM At the same time if the duplicate content report reveals a high concentration of duplicate content or entire folders that were used to backup material you may separate this content with the methods described below Manually Review Content AutoPro also brings together various tools that can be used to review materials of these Quick View Plus QVP is ideal for browsing through a wide variety of Office files images PDFs some email formats including PST and MBOX and other common file types IrfanView s thumbnail view allows for quick browsing of large image galleries although QVP is able to view many of the same files while Inksc
42. ive This series contains materials related to the other two series in the collection Online Board 2006 ZIP archive file download item Online Correspondence 2006 ZIP archive file download item Online Leadership Assembly 2006 ZIP archive file download item Online Litigation 2006 ZIP archive file download item Online Organizational Documents 2005 ZIP archive file download item Online Press 2005 2007 ZIP archive file download item Programs Audio files Online Benton Harbor debate 2006 ZIP archive file download item Online Ellis Cose Colloquium 2006 ZIP archive file download item Online Grosse Pointe Unitarian debate 2006 ZIP archive file download item Online Mackinac Center Lansing debate 2005 ZIP archive file download item Online Michigan Black Republican Council talk 2005 ZIP archive file download item Online Union Missionary Baptist Church debate 2006 ZIP archive file download item Online Wayne State University debate and summation 2006 ZIP archive file download item The large size of the deposit 11 5 GB required the processing archivist to package the content into multiple zip files Rather than create arbitrary divisions within the content each subfolder was packaged in a single ZIP file with the exception of the Programs folder This directory included logistical information about TAFM s programs as well as some fairly large audio recordings of public events
43. l set anumber of variables and if this is the first processing session for a deposit create directories for log files temporary files a backup restore point and separations in addition to generating an initial manifest with MD5 checksums of the content AutoPro User Manual v 3 0 3 Main Menu and Processing Options After verifying the processing directory and processor name AutoPro will open its main menu This screen is divided into four parts 1 Main Menu of Procedures This section lists all eleven steps in the AutoPro workflow Steps should be completed in the order provided but some content types such as archived websites may require variations 2 Statistics This section provides up to date information on the number of files and size thereof in bytes in the processing directory 3 Processing Status This section lists all the procedures completed within the processing workflow Upon first opening the application it reports that a new batch processing sequence has been initiated 4 Option Entry This section permits the user to enter the number of the procedure to run next All newly initiated workflows must begin with 1 Virus scan If you are returning to a previously started workflow you should enter the next procedure in numerical order after the most recently completed step as listed in the Processing Status section After selecting a procedure AutoPro will immediately take you to the selected proc
44. lected extension from a given folder and all subfolders within that directory Use this option with care If used at an upper level of the AutoPro User Manual v 3 0 27 processing directory hierarchy all files of the chosen extension will be removed from each subfolder in that branch of the deposit To use this option right click on a folder and then click on the batch file AutoPro separations extensionRemoval bat from the Send to context menu i gt AutoPro bat Cut Copy ee gt AutoPro separations only gt elected _ na AutoPro zipMultipleFilesFolders bat ary v Share with ele AutoPro zipOneFolder bat Base dj Compressed zipped folder i Properties WU Desktop create shortcut di 20 10 Lies mar ETES Documents di 2011 10 30 2012 2 34 PM File folder H SendTo_20121029 zip A new CMD EXE window will open and prompt you to enter the extension for the files you wish to remove Be sure to enter the appropriate extension as used in Windows and to precede it with a period i e ini Bentley Historical Library Digital Curation Division AutomatedProcessor v 9 7 c 2012 SEPARATE ALL FILES OF A SPECIFIC FILE EXTENSION At the prompt enter the file extension with leading period you wish to remove from this directory USE CAUTION The program will remove every file with this extension to the separations directory Example ini Enter file extension with a leading perio
45. lider on the progress bar He 1 00x 00 34 06 15 Previous next in playlist If there are multiple files in a playlist you may click the arrow keys to move to the next previous item in a playlist Separations Removing Superfluous Content The appraisal and review process may reveal content that should be separated from the collection prior to its deposit in a long term repository This may include certain file types and content deemed to be AutoPro User Manual v 3 0 26 superfluous or outside the collecting scope of the library When content is moved to the separations directory AutoPro will recreate the folder structure of the deposit so that separated materials retain their original position and context Upon initiation of the Appraisal and Arrangement procedure AutoPro will search for and separate a number of common files generated by operating systems These include thumbs db and LNK file shortcuts on Windows and DS_STORE and resource fork _ files produced by Macs If you would like to move additional files to the deposit s separations directory you have two options both of which are batch files found in the Send to section of the right click context menu Please note that each option is available when you are reviewing content with Quick View Plus or aWindows Explorer window Resource 1 AutoPro separations onlySelectedItems bat Use the batch file AutoPro separations onlySelecteditems bat to remo
46. may then append correct file extensions using information generated by the TrID File Identifier utility and collected from the PRONOM format registry If the archivist determines that an extension should be added or corrected AutoPro will document the action in a log file In transforming the SIP to an AIP the Bentley Library relies upon file format conversion as a primary preservation strategy Based upon the Library of Congress s work on the Sustainability of Digital Formats and documentation from the Florida Center for Library Automation and other peer institutions the library has identified a number of at risk i e proprietary or potentially obsolete file formats and developed conversion pathways to sustainable formats with various open source and freeware tools AutoPro searches for these at risk formats based upon extension and then employs the following tools with digital media and target format in parentheses ImageMagick raster images to TIFF Ghostscript PS EPS and PDF to PDF A JHOVE verifies if the original PDF meets PDF A specifications Inkscape vector images to SVG ffmpeg audio to WAV video to MP4 with H 264 See UM ITS FAQ pages at http safecomputing umich edu antivirus fag php Microsoft antivirus information may be found at http www microsoft com en us server cloud system center endpoint protection 2012 aspx 7 Zip is an open source file archiving application For more information see http
47. mplete packaging and description of the item AutoPro will AutoPro will record file sizes checksums and mime types and then generate an XML manifest and EAD for the item You will then be taken to the Main Item Option menu where you willthen have an opportunity to create a new item review other items quit or finalize the packaging and description From the Review Item menu you may also create a NEW item FINALIZE the deposit or QUIT and resume at a later date Finalize Description and Technical Metadata Extraction When you enter option F FINALIZE from the Main Item Option menu AutoPro will complete the generation of metadata files and then prepare to extract technical metadata from material with DROID This application will also generate MD5 checksums for content including materials within ZIP files you will thus be prompted to confirm that DROID is correctly configured If you don t know answer No so that you can be sure that this feature is enabled After DROID opens it will check for any updates if any are available approve their installation Next click on the Tools item in the top navigation menu and select Preferences Make sure that the box next to Generate MD5 hash for each file is checked Click OK and then close the DROID window AutoPro User Manual v 3 0 41 dep DROID v6 01 File Edit Run Filter Repor OPE New Open Save Export 4 Gi Check for signature updates Ctrl U
48. n and then confirm your choice Please note that this option will permanently delete the files please consult with Nancy Mike or division heads if you have any uncertainty 2 Ifthe identity match is in a plain text file or MS Office Open XML document i e DOCX PPTX or XLSX you may use the SCRUB action to redact the PII from file This option replaces the PII with a string of X s Please note that if a preservation copy of a MS Office 1997 2003 file has AutoPro User Manual v 3 0 16 been created you may redact the PII from the preservation copy in OOXML and then SHRED the original Please consult with a division head before taking this step 3 For false positives check the boxes next to the affected files click on the IGNORE action and then select This Item Location from the options god Main Identities Locations Configuration Tools Identity Finder Guest Profile TT O Fob Ee E F I serpe Start Filter Collapse Status Shred Secure y Results All Rows Window Search Display Actions ee zj D Location Date ve Size D Identity Match Preview This Identity Match wa Firefox Password http grooveshark c N A 6 bytes Egfree12 1 Manage Ignore List M Firefox Password www pandora com N A 12 by Egfree12 1 T wa Firefox Passw www jam software de N A 21by EJ 2 329dcb 1 resta kW B C Users Mike Document iquery_002 js 11 1 22
49. ne on the left hand side of the Identity Finder interface Please note that some of these files may contain multiple matches If you click on a file or an identity match the potentially sensitive information will be displayed in the right hand Preview Pane e Main Identities Locations i Fr E sel 0 0 Fob gr Od Start Stop Filter Collapse Status Shred Scrub Secure Quarantine Recycle Ignore s Results All Rows Window Search Display Actions Date Size C Identity Match O E Firefox Password http grooveshark c N A 6 bytes EJ feei O E Firefox Password www pandora com N A 12 by Egfree12 O Firefox Passw uiian jam software de N A 21 by ssword 229 KB return elem nodeName toLowe rCase input amp amp elem type submit function elem var name elem nodeName toLowe rCasel F rah rn frome 5 jquery 002 js Date Modified 11 1 2011 Encrypted with EFS JScript Script File Size 229 KB Read only Owner BHL DC1 Mike There are three possible actions you can take with content that has been found to contain an identity match Shred Scrub Secure Quarantine Recycle Ignore ia 3 Actions 1 If the content is in non unique or routine business documents such as business expense reports or P Card logs you may use the SHRED option to securely delete content Check the boxes next to the target filenames click the Shred actio
50. nifest of all files therein place it at the top level of the deposit directory and delete the original folder If your deposit includes an archived website AutoPro will check to see if the folder in question holds a website Bentley Historical Library Digital Curation Division AutomatedProcessor v 0 8 tc 2013 gt PACKAGING FOLDER C BHL unprocessed 78567_H661 uw greatday com Is this an archived website z Hb If so AutoPro will automatically use the folder name as a title for the ZIP file Packaging Content with AutoPro zipMultipleFilesFolders bat The other option to package content is to place multiple files and or folders into a single uncompressed ZIP file Please note that you will be limited to about 110 items per packaging attempt it may therefore require numerous operations to package a large number of individual files folders To initiate this procedure select the desired files and or folders using the Shift or Ctrl keys and the left mouse button as needed Once the appropriate content is selected right click on one of the items and choose the batch file AutoPro zipMultipleFilesFolders bat from the Send to context menu gt S AutoPro bat a faa AutoPro getSubFolderNames bat gt AutoPro separations extensionRemoval bat Copy a gt unprocessed 87209 unprocesse e Create shortcut dele Delete J 2 Rename a on dj Compressed z
51. nue AutoPro will loop through this list of files using the TrID file identification utility and the PRONOM file format registry to propose appropriate file extensions Based upon these results and if necessary your review of the file you may choose to adopt one of the proposed extensions Enter the three digit extension without a period AutoPro will apply it to the file and record the operation in a log file Bentley Historical Library Digital Curation Division AutomatedProcessor v 8 9 lt c 2013 MANUAL FILE EXTENSION IDENTIFICATION Filename C BHLunprocessed 8566_H661 SNRE 2608 BHL doc TrID output probability of File type assocation based on binary signature 38 4 JFG JFIF EXIF JPEG Bitmap 30 7 JPG JFIF JPEG Bitmap 23 6 JPG JPEG Bitmap 7 6 MP3 MPS audio PRONOM format registry information Mime type imagejpeg Suggested extension s gt from the PRONOM format registry gt Jpg gt jpe jpeg OPTIONS Enter an appropriate 3 character file extension based upon the TrID and PROHOM output Hit ENTER to do nothing and leave the file as is recommended if file cannot be identified by TrID FRO NOM or by archivist s review Enter 0 to QUIT and exit AutoPro Extension tor ENTER to do nothing If you are unsure of what the extension should be simply hit Enter AutoPro will leave the file unchanged and proceed to the next one on its list AutoPro will
52. ogs 98567_H0bi Delete Separations directory too CY z W Ya After you make your selection AutoPro will package the log files so that a copy may be deposited in the Bentley s IFS space for quick reference TRANSFER LOG FILES Upload C BHL logs 98567_6661 6001 2ip to BHL IFS space gt hhl Private htmlprocessing_log_f iles 98567 MOTE Consult with Mike if problems arise Enter SU when AAd1 zip has been successfully uploaded Once you have indicated that the ZIP file of logs has been uploaded AutoPro will request a final verification that you are ready to delete the working copies of materials AutoPro User Manual v 3 0 43 If you prefer to wait you may quit the program and return at a later time Following the deletion of content the digital processing workflow is concluded you may press any key and AutoPro will close AutoPro User Manual v 3 0 44 5 Version History Version Date Prepared By 2013 10 29 Michael Shallcross 2013 04 09 Michael Shallcross 2012 10 29 Michael Shallcross AutoPro User Manual v 3 0 45
53. on Another great feature is the Command History familiar to Linux Mac terminal users use the up and down arrow keys to browse through information previously entered into the CMD EXE console This feature will be particularly useful when compiling administrative and descriptive metadatal Resources AutoPro relies on a number of CMD EXE utilities and Windows batch file syntax to move content through its work flow If you d like to learn more about using the CMD EXE console and batch files in general the following sites can provide some basic information e An A Z Index of the Windows CMD Command Line http ss64 com nt e DOS commands and their usage in batch files http www robvanderwoude com batchcommands php e Command line Reference A Z http www microsoft com resources documentation windows xp all proddocs en us ntcmds mspx mfr true AutoPro User Manual v 3 0 5 2 Initiate a Processing Session While you may be able to process a deposit of digital content in one session you may need several sessions over a series of days to completely process and package content so that it is ready for storage in a long term repository Follow the steps below to initiate all processing sessions for a given deposit Launching AutoPro Go to the Unprocessed directory and locate the appropriate deposit directory i e the folder holding the unprocessed material you ll be working on it will be identified by a unique depos
54. on events i e scans for viruses and personally identifiable information conversion to preservation formats recording of descriptive and technical metadata etc that transform the SIP into an Archival Information Package AIP Bentley archivists initially developed a manual workflow with more than 40 discrete steps that required the operation of numerous stand alone applications and saving tool output to various log files In addition to being highly labor intensive and introducing numerous opportunities for operator error this approach was daunting for staff without technical expertise Given these challenges the Division of Digital Curation developed AutoPro to 1 Make digital processing more efficient by automating key workflow steps 2 Reduce technical barriers and thereby permit archivists to focus their energies on the traditional archival functions of appraisal arrangement and description Overview of the Automated Processing Workflow AutoPro is comprised of 24 Windows CMD EXE and Visual Basic scripts that move content through a nine step workflow and thereby simplify the operation of more than 20 applications and command line utilities The Windows Command Prompt and Explorer windows function as the main interfaces a feature that may be unique to staff more familiar with Graphical User Interfaces see section 1 4 for tips AutoPro User Manual v 3 0 1 on using the CMD EXE console In addition to providing a framework to guid
55. onality of the Windows CMD EXE console also referred to as the command prompt The properties for the CMD EXE console on your work station should have been configured when AutoPro was installed but you may want to check to make sure that the following options are set Open a CMD EXE console window enter CMD EXE into the Start Menu s search box right click on the border of the window and select Defaults from the context menu y C Windows system32 cmd en i ia Microsoft Windows Uersion Copyright c 2009 Microsof Move Restore C Users Mike gt Size Minimize Properties When the Console Windows Properties window opens make sure that the boxes for the following items are checked under the Options tab n N Medium Large Command History Buffer Size 50 Mumber of Buffers 4 Insert Mode Discard Old Duplicates W AutoComplete e QuickEdit Mode allows you to highlight text with the mouse and then hit Enter to copy it to the clipboard Baglt is part of an open source set of transfer tools developed by the Library of Congress For more information see http sourceforge net projects loc xferutils AutoPro User Manual v 3 0 4 e Insert Mode allows you to paste text from the clipboard by right clicking where you would like to insert text e AutoComplete allows you to hit the Tab key to complete the entry of folder and file names when entering path informati
56. onversion program Microsoft File Convertor http www microsoft com en us download details aspx id 11454 is part of the freely available Office Migration Planning Manager Identity Finder Data Loss Prevention DLP Endpoint http www identityfinder com us Business IdentityFinder EnterpriseClient is proprietary software that can identify potentially sensitive information TreeSize Professional is a proprietary hard disk space and file manager and Quick View Plus is a file viewing utility For more information see http www jam software com treesize and https avantstar com respectively Explore the Bentley Historical Library s archival community in Deep Blue at http deepblue lib umich edu handle 2027 42 65133 AutoPro User Manual v 3 0 3 secure dark archives At the conclusion of processing AutoPro deletes the working directory restore point and temporary files and the archivist records the completed digital deposit in the Bentley s collections management database This basic workflow and the component software is subject to change as the Division of Digital Curation actively tracks the development of standards and professional best practices It is furthermore recognized that unique features of digital deposits and material may require additional steps to process and record metadata Notes on the Windows Command Prompt This section of the User s Manual provides additional information on features and functi
57. posit QUIT packaging and resume at a later date RZF Q m Create an Item for Archived Websites If you indicated earlier that the deposit includes an archived website AutoPro will ask if you are describing the web content If so you will be prompted to enter the year the website was captured before continuing with additional item level metadata ITEM METADATA This step will allow you to enter descriptive and administrative metadata for an individual item Is this item an archived website lt Y z ND y Date website was captured CY 2009 H TE The item title should reflect the different sections of the content s intellectual arrangement series subseries file etc gt Enter D when DONE Enter a single level section and hit Enter Title component or D for DOME Your description of the item should note the precise date of the capture if known as well as important features and functions of the archived website The description from a School of Natural Resources and Environment website may be used as a template Archived version of the School of Natural Resources and Environment website as of April 20 2004 Documents the academic programs accomplishments resources events and people at the School of Natural Resources and Environment Content includes important news and announcements information for alumni and content geared towards students such as course descriptions and funding opportunities Auto
58. t before you arrange or package any files Packaging Content for Deposit and Defining Deep Blue Items To ensure that content is packaged correctly the first time you MUST e Complete your intellectual arrangement AutoPro User Manual v 3 0 30 Prepare a draft of your finding aid and submit it for review Discuss with Digital Curation how the content will be deposited and presented in Deep Blue Develop a deposit plan to guide your packaging Use Windows Task Manager to kill Quick View Plus if you used it to review content Even after you close the program QVP continues to run in the background and use the files and or folders you accessed with it Finally be sure to complete all packaging before proceeding to the next procedure There are two options for packaging content both of which are Windows batch files available via the right click Send to menu Determining How to Package Content The processing archivist in consultation with Digital Curation must determine when it is appropriate to deposit individual files or to package multiple files or subfolders into a ZIP file This decision will depend upon a number of factors The intellectual arrangement of the material in the finding aid Ideally we would like to package content at a level where we can provide clear descriptive metadata We also want to provide access via links in EAD finding aids to a discrete and well defined chunk of content Depending
59. t a single deposit may include any or all of the above packaging options AutoPro User Manual v 3 0 31 Example of a Packaging and Deposit Plan To better understand how content may be deposited consider the following example from the Toward A Fair Michigan Records The former program director of Toward A Fair Michigan TAFM transferred her work files to the library on a single optical disk in 2010 The files were arranged in a single directory with multiple subfolders that reflected different aspects of her work as program director In processing the collection the archivist determined that this group of files taken as a whole represented a distinct series Program Director s Files The subfolders in the main directory were determined to represent a level 2 hierarchy as they dealt with specific functions of her role i e Board Materials Correspondence Litigation Press Programs etc With these considerations in mind the TAFM packaging deposit strategy led to the following representation in EAD Container Location Title Program Director s Files series The Program Director s Files series is comprised of born digital records created and maintained by TAFM Program Director Carol Allan and includes incoming and outgoing correspondence press releases organizational bylaws and logistical information for public events The series also features audio and video recordings of public forums on the Michigan Civil Rights Initiat
60. t conditions may result in an unrecoverable error or application failure If you have any issues with AutoPro please immediately share them with Mike so that a workaround can be developed in an efficient and timely manner Step 1 Virus Scan AutoPro runs Microsoft Antimalware Service Command Line Utility a component of Microsoft s Forefront Endpoint Protection 2010 on each file in the deposit If a virus or malware is detected the antivirus software will delete the file and record the deletion in a log file If one or more infected file cannot be removed you will be alerted by AutoPro and be permitted to view a listing of such files If this occurs please consult with Mike upon review of the antivirus log it may be necessary to manually delete the remaining infected files Step 2 File Extraction This procedure consists of the identification of disk image such as E01 IMG AFF etc and archive files such as ZIP TAR RAR etc extraction of content with its original directory structure with FTK Imager and 7 Zip respectively AutoPro User Manual v 3 0 11 Disk Image Extraction with FTK Imager Upon initiation of the procedure AutoPro will search for common disk image file format If none are found the resource will continue on to search for common archive files see below Please note that given the wide variety of disk image file formats this step may not detect all formats If additional disk images are detecte
61. t_bhl Ofbc2cc7 wav AutoPro also creates a log of all file conversions including the original and new filenames timestamp and conversion software In order to protect the identities of record creators and limit its exposure to risk the Bentley Historical Library has established policies in regard to personally identifiable information PII such as credit card numbers and U S Social Security numbers AutoPro thus employs Identity Finder DLP Endpoint to scan for PII Archivists then use the Identity Finder interface to verify search results and if true positive hits are found redact the PII from Open Office XML and plain text files or assign appropriate access restrictions to the content A record of identity matches and corresponding archival intervention is maintained with the log files Archivists then proceed to a more in depth appraisal and arrangement of content AutoPro loads data visualizations such as the distribution of file extensions date range of content relative size of directories etc produced by TreeSize Professional to better characterize and launches Quick View Plus a file viewing program to rapidly review a wide range of file types for description in finding aids While reviewing content with Quick View Plus or the Windows Explorer archivists use a batch file in the right click context menu to remove superfluous files or folders to a separations directory Every effort is made to retain the original order of materi
62. urn to this screen and will have additional options to either REVIEW existing items which in turn will allow you to add additional content to an item or complete the description thereof or FINALIZE the deposit s packaging see below Item Level Metadata The first step in creating a new item involves the entry of descriptive metadata You will first enter the title which should reflect the intellectual arrangement of the material i e series sub series folder etc AutoPro will prompt you to enter one title component i e a level of the intellectual arrangement at a time enter a D when have completed entering all components of the title AutoPro User Manual v 3 0 37 You will then enter additional information about the item including a description i e scope and content note the inclusive dates for when the content was created or originally used the number of years access must be restricted 0 if the content is open and the content type information required by Deep Blue administrators After entering this information you will be given an opportunity to correct any errors Associate Bitstreams with the Item After the item metadata has been saved AutoPro will prompt you to enter information about associated files or bitstreams in a spreadsheet To identify related files i e the original version as well as its preservation and or access copy we use a concept of bitstream groups All
63. ve only those items you have selected to the separations directory As the following example illustrates this option may be used ona single file or folder gt AutoPro bat w m Cut amp AutoPro getSubFolderNames bat AutoPro separations extensionRemoval bat L unprocessed 87209 Copy ry Sharewith v ee AutoPro zipMultipleFilesFolders bat a A sua dl AutoPro zipOneFolder bat er Jh Compressed zipped folder d Inb z Properties WU Desktop create shortcut L Sent EI E Documents d SendTo_20121029 zip This option may also be used on multiple files and or folders selected by clicking on the left mouse button while holding down the Shift key to separate an entire range of files or the Ctrl key to choose a select number of files and or folders for separation i 20100916PriceTagsConferenc amp copied b dirs txt Open __ duration 7 Zip gt _ U Scan with Microsoft Forefront Endpoint Protection 2010 Quick Compress gt t Combine supported files in Acrobat KM Identity Finder 3 gt Gi AutoPro bat amp AutoPro getSubFolderNames bat as _AutoPro separations extensionRemoval bat Create shortcut Please note that when selecting multiple files and or folders for separation all content must reside in the same parent directory Resource 2 AutoPro separations extensionRemoval bat In the second option the archivist may remove all files of a se
64. versions of a file belong to the same group additional files associated with the item will belong to other groups In addition a bitstream group may only have one file in it You will also note the inclusive dates for when the content was Originally used created and provide a brief description For files in the same bitstream group the description should clearly identify the relationship between the files A B L E F 1 Full Path to File Bitstream Group Date Range File Description 2 C BHL un process 1 2009 Original version of surveys and surv 3 C BHL un process 1 2009 Preservation copy of surveys and re A C BHL un process 2 2010 Follow up surveys from 2010 AutoPro User Manual v 3 0 38 After you indicate that the spreadsheet has been saved AutoPro will record file sizes checksums and mime types and then generate an XML manifest and EAD for the item You will then be taken to the Main Item Option menu where you will then have an opportunity to create a new item review other items quit or finalize the packaging and description gt ITEM IDENTIFICATION AND DESCRIPTION In most cases an Item will consist of a single file Multiple files may he associated with an Item only if all the files represent a single coherent intellectual unit MOTE Be sure to discuss the organization of complex deposits with Digital Curation Options create a MEW Item REVIEW Items created for this deposit FINALIZE packaging for entire de

PDF file - Deep Blue - University of Michigan

Contents

Download Pdf Manuals

Related Search

Related Contents