Computing Services for Diamond Users
Contents
1. diamond Diamond Light Source Ltd Rutherford Appleton Laboratory Chilton Didcot Oxfordshire OX11 0QX United Kingdom www diamond ac uk SCI SAD 0008 Computing Services for Diamond Users Prepared by Nick Rees Diamond Light Source Ltd Issue 6 Date of issue 16 May 2014 Abstract This document provides an outline of the services provided to Diamond Users who have been allocated time Page 1 of 40 Issue 1 12 UE RL DOS ONE PM Mr MU PN CENT 3 2 GIDSSdly ncc ILL Mt iu MR MM 3 3 Data access from Diamond Linux workstation eeeeseeeseeses 8 4 Data Access from Diamond Windows workstatiOn ccceceeeeeeeeeeeeeeeeeeeeeaeeeeeeeeeeaeeeas 10 D From trie visitors Te BWOFIK axuvsendstsipnixukt Dada suco ndis ano x patus toD R dns UR Nuke 11 42 1 10 D3ta access MON EINU canna D ces 11 Usmo sampa SMOCIENT 11 Usmo ios iai a a A a 13 4 1 2 Data access MON W INGOWS i scree 16 hYrsiducud weiem TTE 18 NECI 19 Wis ate Feoye ag oy lt 1d Ga qe Ir C n ee 19 Eata E EA mer tc m 21 opata ACCESS TTORTTOEmOIO MOS ausente 24 zm pq 24 7 WO 2 Mare Pa H H 29 LATA NN K E ES ep er WM EN ee ae 29 7 ANNEX B Using Diamond printers 31 ERN rM IEEE LED A 31 1 1 2 Connecting from a Windows computer2. cccccsseeeceecceeeeeeececeeeeceeeceseeseeeesseaaeeeeeseaeeeesessaaess 31 1 1 3 Connecting to the print server from a Linux computer or a Mac esseeseeeeeeses 32 1 1 4 Information on printers prints jobs plus CUPS user manual eesseeesessssss 33 8 Using ICAT to retrieve data from the data archive 33 J 33 18 2 REWEVNO D Sacs 35 1 8 3 Getting data that has been retrieved PreviOuSly cccccsssceccseseeeceeseeeceenseecseseeessaseeessaaeees 38 1 8 4 Searching for a VISIt ccccccsececsceseececssececcseeecsaueeecscuseseceuseceesaeeecsouseeesouseeeesaseeessneesesecasees 39 1 8 5 oggi CUL eirio acer vet alien re Mer ac at ufo ug Dd ui eerte edes ute beg D ate 40 Page 2 of 40 Issue 1 12 INTRODUCTION 1 Purpose This document describes the services provided to users who have been given beam time at Diamond but who are not Diamond staff or have any other Diamond connection 2 Glossary CDROM DLS DVD Fedid ftp Janet PBS ssh sftp scp USB eSATA Compact Disk Read Only Media Diamond Light Source Ltd Digital Video Disk or Digital Versatile Disk Federal Identifier A computer username unique across CLRC and DLS systems File Transfer Protocol Joint Academic Network Principal Beamline Scientist Secure shell a method of logging in securely across the network Secure File Transfer Protocol an ecrypted form of ftp Secure copy
3. a methor of copying files securely across the network Universal Serial Bus External SATA a way of connecting external disks no longer widely used but still available on some beamlines Page 3 of 40 Issue 1 12 COMPUTING SERVICES PROVIDED FOR DIAMOND USERS 1 Federal IDs and passwords A Federal ID usually abbreviated to FedID is a Diamond computer username which together with an associated password enables the user to access the Usernet Proposal Manager website and all beamline computing systems including data storage New users must complete a Registration form which ensures that the User Office hold relevant personal details The following steps happen once the registration form is submitted User is automatically allocated a Fed ID User must acknowledge receipt of FedlD and acceptance of Diamond IT policy User is allocated a password by the User Office together with details about the Quest Password Manager system If arriving at Diamond without their FedID and or password the user will not be able to take the safety test on line and is likely to experience significant delays in starting their experiment as staff in the User Office cannot immediately reset passwords For your own benefit and to help the User Office team please ensure that you remember to bring your FedID and password when you come to Diamond for beamtime 2 Quest Password Manager By registering a set of personal questions and answers y
4. Name Al Date Modified Macintosh HD dat 14 April 2011 21 53 71 diskOs3 Bl amp sna dat 26 March 2007 13 53 diskOs4 E 9na dat 26 March 2007 13 55 E iDisk gt E 2006 7 December 2006 16 10 a 2007 20 May 2011 08 04 Saat gt GE 2008 31 March 2011 11 27 Es ape lay gt B 2008 2 17 November 2009 09 06 E 2rb13439 s re Y i 2009 31 March 2011 11 01 Y PLACES Gg 0 0 22 December 2008 12 00 Fal Desktop Bg mx19 13 10 June 2009 13 49 tina ig nt491 1 28 June 2009 16 45 J Applications ij nt492 1 19 March 2009 12 13 Koc ig nt1173 1 4 June 2009 10 52 ig nt1174 1 10 March 2009 13 07 b ij nt1175 1 17 September 2009 09 33 Add G8 nt1176 1 4 September 2009 14 28 V Yesterday ig nt1237 1 27 October 2009 08 12 C Past Week i nt1239 1 30 October 2009 03 32 5 All Images Ga spo 2 February 2010 12 58 5 All Movies ij spo 1 14 July 2009 13 58 3 All Documents A sp684 2 28 August 2009 11 30 Bg sp752 2 10 June 2009 15 32 lof 144 selected 3 42 TB available Figure 17 navigate to visit data directory Data can then be dragged and dropped on to your machine Using sftp The instructions for command line data access via sftp from Linux 4 1 1 page 13 apply to Mac as well Alternatively a graphical tools for sftp access can be used The following example is using Cyberduck Open the software and choose SFTP as protocol Page 21 of 40 Issue 1 12 Get a donation key i E FTP File Transfer Protocol FTP SSL E
5. access to data from a Diamond Windows workstation On beamline Windows workstations the network drive might already be connected as the X drive 5 From the visitors network The beamline data directories are available read only on the visitor network The files are visible on the server data diamond ac uk and can be accessed using either smb Windows Explorer or sftp You can connect to the visitor network either using the wireless network or using the dedicated wired connections on every beamline 4 1 1 Data access from Linux Using samba smbclient The Nautilus Linux file manager and the Konqueror browser can be used to browse the data directories by typing sinot data diamond ac uk Page 11 of 40 Issue 1 12 into the address bar If you try to connect to the directories you will be asked for username and password Note that you will have to enter your username as CLRC yourFedID see Figure 4 for Konqueror and Figure 5 for Nautilus f smb data diamond ac uk i15 Konqueror Location Edit View Go Bookmarks Tools Settings Window Help A 4 AgOB DASS sogna m e Location Authorisation Dialogue Please enter authentication information for Serer data diamond ac uk Share i15 Username CLRC npr s Password 11 Items 0 Files 11 Folders Figure 4 Connecting from Linux using Konqueror Page 12 of 40 Issue 1 12 File Browser i18 on data diamond ac uk File Edit View Go Bookmarks Help gt A
6. ox e Select one or more files in the file view the tick box at the top of the table will select all files in the view e Click on Download ie det gem he bee Demo dem TOPCAT web Hn tmm i Ld om en mae hic ima im EU EX a emi i ii F D disc i em Ty I i gm xm i Sti 4 ma CEL SE ER EIN im m Ram Re sm mall ambas i mm amp Drm Ps Sab pies le eidebdgo Tiap Papen nem aa pat E iil M LL a 1 pasaq um E ae Em mx FRE tke FAUX TIER Umen iat mar ma amada SEs I Sed LUSLASIER De Dem wena eel RT ere Enter a file name for the compressed file that will be prepared and presented for download to your local machine Page 36 of 40 Issue 1 12 Se he eee ee HENCE Se free ee ee EEEE AE You will see the following message when the retrieve from archive begins This retrieves it from the archive and prepares it for download to your local machine This may take a long time You do not have to be logged in for this process to complete e Progress of the download will be shown in the My Downloads tab This is found in Page 37 of 40 Issue 1 12 the same line as the My Data button e f you wait in an active session you will be presented with a save dialogue box where you can download the file to the local machi
7. 0 3 CX Back Forward Up Stop Reload Home Computer Location smb data diamond ac uk i18 amp 100 amp Information Y Authentication Required You must log in to access data diamond ac uk i18 Lisername npr s 118 on data diamond E x directory smb share Domain C LRC ry Password Remember password for this session Save password in keyring Figure 5 Connecting from Linux using Nautilus 18 selected You can also use the command line utility smbclient to connect to the data directores smbclient U CLRC yourFed D L data diamond a cuk will show you the list of available directories on data diamond ac uk Note you need to put quotes around CLROWourFedlD Connect with the following command smbclient U CLRCVyourFedID data diamond ac uk typing help in an smbclient session will give you a command overview typing help command will give help on that command Using sftp Linux Mac data diamond ac uk can also be directly reached via sftp from the visitors network i e data can also be transferred using sftp Using sftp you can also write to the directories given you have write access to them e g your home directory Under Linux you can simply use the sftp command on the command line or use the sftp features of Konqueror or Nautilus sftp data diamond ac uk in Konqueror File gt Open Location gt sftp data diamond ac uk in Nautilus You will likely have to l
8. 2008 080628 151043 csv 15K8 Text File with Comma S 29806281511 blGuser 72008 080628 223621 csv LIKE Text File with Comma S 2008 06 28 22 38 rweee r bl user 72008080628 224019 c5v LIKE Text File with Comma S 20866282241 mewt bl user T 2008_080628_224307 csv 14K Text File wth Comma S 2006 96 28 22 43 mwrw t bl6user 2008 080627 225336 2008 090627 225458 2008 080627 225605 csv cw csv x p fv dis selected contaireng 27 items 2009 Folder Page 9 of 40 Issue 1 12 Ese dis Eile Eda View Places Help Location Gam View Ge posmas Toss Settings Windas Hels TREI Aa i xn L2 Ss ET E Liao ES iedobi 2009 Ele Edi View Places t 2009 01 06 10 118 n Gtx aj ine f i l tb E 4 008 Folder 1 o KE et 2305 01 06 10 18 E3mi767 1 ORG Fal 1509 01 15 01 05 Sprei 767 5 JAOKS Folder 3009 01 12 11 404 E te 0 n514 A Items Files 4 Folders Figure 2 access to data from a Diamond Linux workstation using Nautilus or Konqueror These directories are automatically available on all Diamond Linux workstations Note data wil only be readable to users registered for this visit to access your data this way you will have to be logged in to a workstation using your Fedld 4 Data Access from Diamond Windows workstation On any Diamond Windows workstation the beamline data directories are available as BEAMLINE storage BEAMLINE Experimental data wi
9. beamline data directories plus your Diamond home directory see Figure 10 T data diamond ac uk File Edit View Favorites Tools Help sack d pa Search j Folders Comments Bl data To data 103 data I04 data T06 data I11 data I15 data I16 data I18 data I2 data I 4 data Home Directories ey Printers and Faxes Shows installed printers and Fax 13 objects RH Figure 10 View after connecting to data diamond ac uk from Windows Page 16 of 40 Issue 1 12 oelect the directory you need For beamline directories this will then show you the data directories on that beamline see Figure 11 for an example data diamond ac ukib16 File Edit View Favorites Tools Help sack eJ a Fa Search Folders mei x 3 Bak Address Size Type Date Modified File Folder 1012 2007 10 05 File Folder OF fos 2008 13 54 File Folder OS 03 2008 19 31 File Folder Soe 2007 12 56 File Folder 11 12 2007 17 39 File Falder peruana 09 32 jlost found File Folder B6 7 2007 15 57 scripts File Folder 2 n2r2008 13 34 C software File Falder 12 12 2007 13 48 var File Folder 14 02 2008 09 25 10 objects 0 bytes a Local intranet Figure 11 An example Beamline directory Experimental data will be in the directory data This directory contains a subdirectory named after the year of the experiment e g 2007 which in turn should contain a directory named after your proposal number see Figure 12 for an example The
10. HO soci comaining 10 heme Msci scraich 08 Folder 2009000 11 29 rarxra root Smp OB Folder 2005 01 20 11 29 POPKA mot tmp usr var gt Y S selected containing 27 tems bi Folder bit Konqueror 066 6010 Gld SHS ems 2007 12 19 10 05 2008 1722 11 44 2008 D4 28 12 00 A KB Folder 2008 05 27 10 41 4 0KB Folder 2007 12 11 17 38 310KB Felder 2000 01 16 20 50 IGOKE Locked Folder 2007 07 06 15 57 4 008 Folder 2009 01 14 10 35 AKB Folder 2005 01 06 16 19 IOKE Folder 2008 01406 16 47 JY dis selected containing 27 dems Location dit View Go Bookmarks Tools Settings Window Help 9 9 0 0 0 l4 SIDA w e D Location E3Ne dwpigdga ET hm heme Permissions Owner 007 40KB8 Folder 2007 07 16 14 04 MAN gda 20 40K Folder 2009 01 07 01 01 PANTNNT K bl user 4 0 KB Folder 2009 01 14 01 08 eran bnh 538 EH m451 overflow 40K8 Folder 2006 09 14 170 rwewarx bl user mages 4 0K8 Folder 2008 12 04 19 12 rwxrwuwx bl user 40K8 Folder 2006 06 15 17 11 Merce gda T 2006_080827_225336 csv 5148 Text File with Comma 5 2006 06 27 22 53 eter bluser T 2008 080527 225458 csv 5128 Text File with Comma 5 20606272255 mwener bluser T 2008 080527 225605 csv 5178 Text File with Comma S 2006 06 27 22 56 TW TW C blbuser 72008 080527 225655 csv 15K8 Tex File with Comma 5 200806272257 mwmwr bl6user T 2008_080628_150912 csv 15KH Text File wth Comma S 2006 06 28 15 09 rwtmet blbuser T
11. beam time Outside this time they will be unable to log in unless authorised by the PBS of the beamline for which they had time Users will be granted authorisation to log in to machines on their allocated beamline during the days they have been allocated time to set up their experiment and take beam Outside this time they will be unable to log in unless authorised by the PBS of the beamline for which they had time Users may bring their personal computing equipment to Diamond However it will normally only be allowed to be connected to the Diamond Visitor network The only Diamond services available from the Diamond Visitor network will be access to their Diamond data access to Diamond printers and access to the external internet Access to other Diamond services will only be available as an external untrusted system A Beamline Visitor Instrumentation Network BVIN will be made available to host users equipment that needs access to the beamline network for instance specialist sample envrionments PVs published by the beamline are also visible on this network Users must indicate at least 2 weeks in advance if they require the use of this network Page 5 of 40 Issue 1 12 17 Under exceptional circumstances the PBS of the beamline can make a case for beamline user s personal computing equipment to be connected to a Diamond network other than the visitors network This case must be made at least 2 weeks in advance of the visit and DLS
12. can be opened like Nautilus gnome Konqueror KDE kdf51254 File Edit View Places Help accounts atw bin blstoragel_diags cifs keytabs_i02 clustat_on_all storage e v Open Location x Location sftp data diamond ac uk Desktop diamond computing dnstable controls dnstable new documentation downloads kdf51254 339 items Free space 351 9 GB BP sftp data diamond ac uk home kdf51254 Konqueror Location Edit View Go Bookmarks Tools Settings Window Help 8 9 0 010 R SIDE m 2 Y E Location sttp data diamond ac uk home kdtS1254 lt q LEO Name v Size File Type Modified Permissions ESaccounts 40KB Folder 2007 09 12 15 10 mwxrwxrx E3atw 4 0KB Folder 2006 09 11 11 49 rwxrwxr x shin 40 KH Folder 2009 01 20 OR 5 marmara i3blstoragel diags 4 0 KB Folder 2007 06 11 16 41 rwxrwxr x EXcifs keytabs j02 storage 4 0 KB Folder 2007 03 29 16 51 rmwxrwxr x E3clustat on ali 4 0KB Folder 2007 02 13 08 38 twxrwxr x E3CSO4R 40KB Folder 2008 11 24 16 24 rwxrwxr x Escscrtracks 4 0 KB Folder 2008 02 08 15 22 waranta E3dell_open manage mibs 40KH Folder 2007 09 28 16 05 rwxrwxt x EXDesktop 4 0 KB Folder 2009 01 12 09 39 rwxr xt x E3diamond computing 40KB Folder 2008 10 16 11 24 nwxrwxr x EXdnstable controls 40 KB Folder 2009 01 14 11 33 nwxrwxt x ESdnstable_new 4 0 KB Folder 2009 01 14 10 17 maranta E3documemation 4 0KB Folder 2008 04 07 10 34 rwxrw
13. ct Username Password from the Authentication Type box e Enter your FEDID password and click log in Page 33 of 40 Issue 1 12 e Once logged in you will see the My Data home page e You can navigate back to this point by using the tabs below the Diamond logo pe put yes Him Bestes xn gem iti Nr gR sb nimm coa pma RON Ea mo EET itt m n od foe pa mar E jam ter M zm ai i irm i Page 34 of 40 Issue 1 12 1 8 2 Retrieving Data e Double click on the visit you wish to retrieve data from in the My Data window e Click on the visit and select the files individually or tick the box at the top of the table to select all data sets e Click the View button and the download window will appear Page 35 of 40 Issue 1 12 TOFTA mh ae Ee m wi ds i rs a imp ey Pe Ag Fai 2 GER OE am a vu FER mar um ua i marna EM mias 707 ARS pa amt E Pd a E E el ee piijpidi 1j Ee zH A IET Gui Eee B m bcd i Be 4 iia eG PG a a um erm Pidpa Admam dura snm ong indt nent 0 a toile d La IL EE en SSS ee erc E iim eee ZU ee ane Coe ee ee OO m Pee ee E U TE Emri i um Ier bam kanp mini reer urm iem m DL
14. cups restart The printers should now appear try lpstat p 2 per user does not require root access As user in your home directory add a file cupsrc with the entry ServerName print diamond ac uk That user should now see all printers when doing 1pstat p Note that this is not system wide and each user would have to do it him herself Page 32 of 40 Issue 1 12 A default printer can be set with the command lpoptions d printer e g lpoptions d dh g 51 bw 1 to set the default printer to the black and white printer in the ground floor print room in Diamond House Note that this is user specific unless the command is called as root 1 1 4 Information on printers prints jobs plus CUPS user manual Connect to print diamond ac uk 631 from any web browser for access to a list of printers and print jobs can also be cancelled or restarted The CUPS user manual with information on command line print tools and options can also be found there 8 Using ICAT to retrieve data from the data archive 1 8 1 Logging In e Open a web browser and navigate to https icat diamond ac uk e You will see an Authentication Type pop up box If this is not shown please check if your browser disables pop ups e f you still do not see the login Authentication type dialogue box click login in the top right hand corner a m ud m o amp Soemono i Po Eh ES Bineri r Pea Fed i bp repie Fierro Prd oat e Sele
15. e File Folder 06 01 2009 16 19 var File Folder 06 01 2009 16 47 b16 storage b16 data b16 storage b16 data 2009 File Edit View Favorites Tools Help File Edit View Favorites Tools Help Q Back i J Search gt Folders gt X 9 E Q Back 9r 2 J Search Folders S e x W Address Wb16 storagelbi6ldata JB Go are t a eje pate toed tuus storaged File Folder 21 08 2008 10 10 0 i Folder 06 01 2009 10 18 2007 File Folder 16 07 2007 13 04 Onto File Folder 06 01 2009 10 18 232008 File Folder 07 01 2009 01 01 Cmt767 1 File Folder 14 01 2009 01 08 File Folder 14 01 2009 01 08 ntsi4 1 File Folder 12 01 2009 11 04 FH_nt451_overflow File Folder 14 09 2008 16 01 images File Folder 04 12 2008 19 12 runs File Folder 18 06 2008 16 11 2008_080627_225336 csv 1KB Microsoft Office Exc 27 06 2008 21 54 E2008 080627 225458 csv 1KB Microsoft Office Exc 27 06 2008 21 55 2008 080627 225605 csv 1KB Microsoft Office Exc 27 06 2008 21 56 2008 080627 225659 csv 2KB Microsoft Office Exc 27 06 2008 21 57 2008 080628 150912 csv 2KB Microsoft Office Exc 28 06 2008 14 09 aL 2008 D80628 151043 csv 2KB Microsoft Office Exc 28 06 2008 14 11 2008 080628 223621 csv 3KB Microsoft Office Exc 28 06 2008 21 38 E2008 e60628 224019 csv 3KB Microsoft Office Exc 28 06 2008 21 41 2008 080628 224307 csv 2KB Microsoft Office Exc 28 06 2008 21 43 Figure 3
16. er than the bus can provide when working flat out This can lead to file system corruption and data loss The Experimental Hall Coordinators EHCs have a small number of external hard drives for loan These are formatted as either ext3 or NTFS For smaller data volumes it may be sufficient to copy it to a portable storage medium DVD USB flash drive USB hard drive from a beamline workstation All beamline workstations have CD DVD writers Blank media may be obtained from either the User Office or the EHCs 3 Data access from Diamond Linux workstation All data taken during an experiment is saved in a directory dls BEAMLINE data YEAR EXPERIMENT VISIT i e for experiment mt9343 visit 1 in 2014 that is proposal no 9343 in village mt 1 visit in 2014 on B16 data will be stored in dls b16 data 2009 mt9343 1 qqs4347 2o0ws040 dis b16 data 2014 Figure qqsd3472lwsO4Q cd dla bib j qqs43472lws040 b16 ls m eldata epics access el qqsds3d472bustd0 b15 cd data na data aa5434728us040 data ls 9 2008 2009 2010 2011 2013 2014 Oa qqsdz4728us040 data cd 2014 Diamon qqsdz4728us040 2014 ls m cm4959 1 inB007 1 mt9104 2 mtS934z 1 mtSB53 1 nt4819 2 nt5045 1 lom4969 2 mt4984 1 mtS9140 1 mt9502 1 mtSB857 1 nt4819 3 nt5B71 3 worksta ee9666 1 mt 946 2 mtSl48 1 mtS9571 1 nts3800 2 nt5038 1 nth5B71 4 tion qqs434720u2040 2014 M the comma nd line interfac e Altern atively any file browser e g Kon
17. formation about FedIDs and passwords minor cleanup and extra info about esata and data dispensers Revision of Annex A especially data copying instructions added Data Dispensers added instructions for Mac Revised section 3 Revised section 5 to remove out of date instructions for attaching external media to linux workstations Added information on supported interface types revised quota limits and added URI for password management suitable for printed version updated most sections and added section on NX implemented changes from documentation review ANNEX A Guide to SOME common computing tasks DATA ACCESS AND DATA TRANSFER The recommended and fully supported way to make a copy of experimental data taken during a visit especially for large data volumes is copying the data to a USB or eSATA disk whilst Page of 40 Issue 1 12 it is being acquired using the designated Data Dispenser hardware available on each beamline Beamline data dispensers can handle either ext3 ext4 or NTFS formatted hard drives Note drives formatted with FAT file systems are NOT recommended due to inherent limitations Drives formatted with HFS Apple are NOT supported Note also that the Thunderbolt interconnect is NOT supported Users can bring their own disks It is recommended that these are the desktop variety with external power supply rather than bus powered This is because bus powered disks often require more pow
18. ge 30 of 40 Issue 1 12 7 ANNEX B Using Diamond printers Diamond s print server is called print diamond ac uk It is available from the Visitor Network i e visitors bringing their own computers and connecting them to Diamond s Visitor Network should be able to install Diamond printers following these instructions 1 1 1 Printer names All printer names in Diamond follow a naming scheme Additionally all printers should have a name tag A printer name will contain the building the room the printer is in the printer type and a printer number Thus Diamond House printers will be called dh floor room type no in the synchrotron building s floor room zone type no and beamline printers beamline room type no e g dh g 51 bw 1 Diamond House black white printer room G 51 ground floor s 1 13 202 co 1 Synchrotron building colour printer room 1 13 zone 02 1 floor b i16 cc4 col 1 Beamline 106 colour printer in CC4 1 1 2 Connecting from a Windows computer The quickest way to find all shared printers is to directly connect to the print server using Windows Explorer Open an Explorer window and enter print diamond ac uk In the address bar see Figure 23 That will bring up a list of all available printers Select the required printer right click on it and choose Connect see Figure 23 y S prink diamond ac uk File Edit View Favorites Tools Help Q Bak EJ F pe Search Folders oy 3 XX e fal Addre
19. his is described in more detail on page 13 in section 4 1 1 the instructions given for Linux Linux sftp access apply to Windows sftp access as well Page 18 of 40 Issue 1 12 4 1 3 Data access from Mac Using samba CIFS In the main menu Finder select Go Connect to Server Forward Enclosing Folder ii Computer Home Fa Desktop Network E iDisk Applications i Documents gt E Utilities Recent Folders Go to Folder Connect to Server Figur e 13 data access from Mac connect to server Enter smb data diamond ac uk as the server to connect to and press Connect Server Address smb data diamond ac uk E Favorite Servers T Remove Figure 14 connect to server data diamond ac uk Enter CLRC FedlD as your id and your FedID password as your password Page 19 of 40 Issue 1 12 Enter your name and password for the server data diamond ac uk Connect as Guest 9 Registered User Password eesssassas _ Remember this password in my keychain Ges Comet Figure 15 enter your FedlD and password This will give you a share with the list of beamlines select the correct beamline and click OK Select the volumes you want to mount on data diamond ac uk Figure 16 select beamline as volume to mount Your data directory can be found as in a subdirectory data year proposal number Page 20 of 40 Issue 1 12 iQ PE
20. home institutes can be slow SSH remote transfers are usually around 10MB s but remote sites may have further limitations Transfer to the Visitor network is limited by firewalls so is less suitable for large data volumes For further information please refer to the Usernet Transferring Data pages which can be found under the While You are Here section Page 6 of 40 4 DOCUMENT HISTORY vO 1 v0 2 v0 3 v1 0 v1 1 v1 2 v1 3 v1 4 v1 5 v1 6 v1 v1 8 v1 9 v1 10 v1 11 v1 12 2 Aug 2006 16 Dec 2006 11 Jan 2007 12 Jan 2007 1 Nov 2007 25 Mar 2008 12 Aug 2008 21 Jan 2009 23 Mar 2010 9 Nov 2010 5 April 2011 16 August 2011 21 October 2011 7 March 2012 22 April 2014 14 May 2014 Issue 1 12 First draft NPR Second draft after review by IT DAG Linux system administrators and user office Third draft Added incoming ssh from DLS visitor network and update glossary First released version Numbered items and added PBS designate to point 11 Added ability to access data directly from data diamond ac uk Added instructions for copying data to the visitor network and printing plus a couple of minor corrections from the User Office Simplified instructions and added a few more pictures changed instructions re data copying from being visitor network to general data copying instructions visitor and beamline network merged in data transfer speeds Added more in
21. ing a DLS workstation The hardware that you are using is more powerful than a standard desktop and has fast access to the beamline data areas This is an ideal environment for you to perform data processing 4 1 5 4 1 6 21 4 1 7 224 1 14 1 3FTP DLS provide an FIP service for remote transfer at ftp fedecks diamond ac uk This system requires client software that can use explicit encryption or ftpes This is to protect your credentials The server is compatible with clients that can negotiate parallel transfers Page 29 of 40 Issue 1 12 Other greg nipc e Clients greglinlipez lftp qqsds4 z fedecks diamand ac uk are Password ftp qqs4s472BRPedecks diamond ac ukt cd i data 2014 availa cd ok cwd iz dataz2 14 ble ftp qqsds4 2Bfedecks diamond ac ukz i2 0 data 20145 ls druxrus 13 37500 108452 4086 Apr 15 05 36 cmd974 1 and druxrwus 12 3 500 108038 4096 Apr 22 05 55 cmed9 4 27 compa drwxrws 4 500 108451 4096 May s 25 10 spdib 4 druxrus 7500 108435 4086 Dec 28 00 10 sp4374 1 tible drwxrws Arog 1098471 4096 Apr 28 23 10 sp 961 2 such drwxrwz 4 500 108205 4096 Mar 31 15 40 sp915 1 drwxrwz 3 500 109546 4096 Mau 11 23 10 sp917 0 2 as druxrws 37500 109126 4096 Mar 31 15 40 sp3196 1 Filezill drwxrwz 4 500 108487 4096 Mau 5 23 10 sp97 97 1 aa lftp gqos4a472lfedecks diamond ac ukt i20ydata 20l4 gt D d Illustration 8 Secure ftoes server accessed using lftp client Pa
22. k an existing session if you already have one for a Diamond NX server or click the New button Page 24 of 40 Issue 1 12 HoMachine Recent connections NOM ACHINE lt Click here to create a new connection gt Illustration 1 choose an existing connection or create a new one Page 25 of 40 Issue 1 12 NoMachine New connection NOMACHINE Protocol Select the protocol used to connect All protocols use ci TOR aphy to protect your communication MX is the native protocol used by NoMachine and is optimized for multimedia data SSH is an industry standard for accessing computing resources from external networks Illustration 2 Select the SSH protocol to connect to v3 5 servers NoMachine New connection NOMACHINE Protocol Insert the hostname or IP and port where you want to connect gt C hest SPES nx diamond ac uk Port a d The port was cho sen automatically based on the default for the protocol If the remote computer was configured to listen on a different port please insert it above Illustration 3 Set the hostname the port number will be 22 by default if you have selected the SSH protocol in the last screen Page 26 of 40 HoMachine New connection Protocol Authentication How do you want to authenticate on the host L Use the system login This is the default Choose this method to Use the Now achine logi an 5 Ea Sei Verspet fi 15 4 n Use a i i ee Bid 7 ul
23. l as key crabe Hoa wa O Lo wt F Issue 1 12 NOMACHINE Illustration 4 Select the NoMachine login option to connect to a v3 5 server Use defaults for the following screens Page 27 of 40 Issue 1 12 HoMachine Connection to nx diamond ac uk Verify host authenticity NOMACHINE The authenticity of host nx diamond ac uk 172 23 146 10 can t be established The RSA key fingerprint is b5 12 21 a3 8a 62 Oc Be c7 94 29 9b 8e fa ff fa Are you sure you want to continue connecting Illustration 5 the first time you connect you will need to accept the host key NoMachine Connection to nx dlamond ac uk Connection to nx diamond ac uk NOMACHINE Please type your username and password to login Waele 0054134172 Password ee ee SE EE SEE SE Save this password in the connection file ii Login using a guest account Illustration 6 login with your FedID and password It is recommended that you do NOT save these credentials in the connection file Page 28 of 40 Issue 1 12 HoMachine Connection to nx diamond ac uk Connection to nx diamond ac uk NOMACHINE E ve Create a new KDE Create a new XDM Create a new VNC virtual desktop virtual desktop virtual desktop Create a new RDP Create a new CDE Virtual desktop virtual desktop Illustration 7 Select the type of connection You should now have a full DLS desktop environment with access to your home directory and data directories as if you were us
24. ll be in the directory data This directory contains a subdirectory named after the year of the experiment e g 2009 which in turn should contain a directory named after your proposal number see Figure 3 for an example The subdirectory named after your proposal contains your experimental data For example for experiment mt767 visit 1 in 2009 that is proposal no 767 in village mt 1 visit in 2009 on B16 data will be stored on b16 storage b1 6 data 2009 mt 767 1 Note data will only be readable to users registered for this visit to access your data this way you will have to be logged in to the workstation using your Fedld or at least connect to the share using your Fedld Page 10 of 40 Issue 1 12 P b16 storage 9 bi6 storage b16 File Edit View Favorites Tools Help File Edit View Favorites Tools Help ae Q Back P P Search Folders i 2 X 19 m O Back P P Search Folders y 3 X 19 m Address Y Mb16 storage Go Address 2 b16 storagelb16 ae Comments of bl b16 storage Fle Folder 10 12 2007 10 05 E kdf51254 Home Directories File Folder 22 12 2008 11 44 a Printers and Faxes Shows installed printers and Fax docs File Folder 08 04 2008 11 00 epics File Folder 27 05 2008 09 41 etc File Folder 11 12 2007 17 39 logs File Folder 16 01 2009 20 50 5 lost found File Folder 06 07 2007 15 57 scripts File Folder 14 01 2009 10 35 softwar
25. n key y i s F _ Se gie OQ 7 Open Connection Quick Conner Action Refresh Edit Disconnect Filename a uc Re rJpur pcull voles u dis OB 02 08 2011 14 56 b Lj attic OB 02 08 2011 14 56 Lc bl6 OB 02 08 2011 14 56 Lc b18 OB 02 08 2011 14 56 7 b22 4 0 KB 17 08 2009 17 45 E bin 4 0 KB 17 08 2009 17 45 T je data 4 0 KB 04 01 2011 09 47 e 2009 4 0 KB 07 03 2011 13 31 v 2010 4 0 KB 07 03 2011 14 38 ac 0 0 8B 25 12 2010 21 06 d m cmi1905 1 33 B 07 03 2011 13 55 a d cm1905 2 33 B 07 03 2011 13 55 3 cmi1905 3 33 B 07 03 2011 13 55 b d cm1905 4 33 B 07 03 2011 13 55 L3 cm1905 5 33 B 07 03 2011 13 55 m nt1872 2313 35 B 07 03 2011 14 07 P acd nt1938 32 B 07 03 2011 13 55 b Lc nt1938 1 33 B 07 03 2011 13 55 b el nt5572 1 33 B 07 03 2011 13 55 bh ee r 270 nm noocnaqv 770 7 rE 110 Files a 4 Figure 20 nagivate to visit directory in Cyberduck 6 Data Access from remote host Remote access to DLS is provided using NoMachine software The client software is available free from www nomachine com for Linux Microsoft and Apple clients Unless your data sets are small it is recommended that you analyse your data at DLS rather than trying to transfer it over the WAN The following instructions and screenshots were created using v4 2 21 of the NoMachine software 4 1 4 NX otart the NX client nxplayer this should give you a Recent connections screen Either choose double clic
26. ne 5 diamond sss 1 8 3 Getting data that has been retrieved previously e f you close the session and want to get data that has already been retrieved navigate to the My Downloads tab in the same row as My Data e Click the Download or Re download button on the right hand side of the line showing the retrieved visit e his will open up the save dialogue box and there you can download your data from the archive to your local machine Page 38 of 40 Issue 1 12 1 8 4 Searching for a visit e If the visit is not shown in the My Data view there is a Search utility available e Click on Search in the tabs that you see My Data in e Expand the Facilities Search field e n Visit Id type the visit number all in capitals and click search e Do not click on any Beamline if something is selected press CTRL and click on the one selected Sresetss brat Page 39 of 40 Issue 1 12 1 8 5 Logging Out To log out of icat select the Logout button in the top right of the screen Pa BM wee TM A7 mrp umi Pun n i i noU B i a diamond ee PE GERNE Pu nar acta Gul amm a d mg Em ds memes c oi Dr M vA MITAN HM cree oye ae Ll d a ir MEIHE PP PEHE NIRE Py HAEE eet aed Page 40 of 40
27. ntly one month after it has been taken Currently DLS has no provision for keeping data for a longer period than this and users are responsible for keeping copies of their data Data taken on previous visits is made available from the archive at https icat diamond ac uk Users will be able to make copies of their data by a Copying the data to a personal disk using designated Data Dispenser hardware available on each beamline b Copying the data to DVD or CDROM using designated machines c Using outgoing ftp to copy the data to an external location d Using outgoing scp to copy the data to an external location e Using incoming scp to copy the data to a laptop connected to the Diamond visitor network f Mounting the Diamond data areas exported from data diamond ac uk onto a machine on the visitor network and doing a normal file copy to their visitor machine Connection of users storage devices to other than designated connection points isnot allowed Users will be granted authorisation to log in on the console of machines on the DLS office network for a limited period around their allocated beam time up to one month before and at least one month after Outside this time they will be unable to log in unless authorised by the PBS or their designate of the beamline for which they had time Users will be granted authorisation to log in to machines on the DLS office network via ssh for a limited period around their allocated
28. og in using your Fedld maybe as CLRC Fedld as described above if you are not logged in to your computer with your Fedld Note this will probably drop you into your home directory from where you Page 13 of 40 Issue 1 12 wil have to navigate into the data directory you want to go to the directory will be called dls BEAMLINE data YEAH EXPERIMENT VISIT as described in section Error Reference source not found Using the command line sftp connect by entering sftp yourFed id data diamond ac uk in a terminal window in sftp help will show all available commands qqs4347 20ws040 dis b16 data 2014 Figure 6 sfto access Quit sftp e l cd path Change remote directory to path data di chgrp grp path Change group of file path to arp amond chmod mode path Change permissions of file path to mode k chown own path Change owner of file path to oun deo df hi path Display statistics for current directory or from filesystem containing path Linux exit Quit sftp get P remote path Llocal path Download file comma help Display this help text nd line led path Change local directory to path first lls ls options path Display local directory listing lmkdir path Create local directory figure ln oldpath newpath Sumlink remote file F pwd Print local working directory pwd le LaflneSt path Display remote directory listing will lumask
29. ou can change your password to make it more memorable or reset if you have forgotten it This will save time for you and can ensure that you start your beamtime without delay Please register with Quest Password Manager http access stfc ac uk so that you can quickly reset your password if forgotten 3 All usage of Diamond Computers by Diamond Users must be in accordance with all current laws and must be ethical and not interfere with the business mission professional reputation or security of Diamond In particular users must abide by the following policies a Janet Acceptable Use Policy b STFC Information Systems Privacy and Security Policy 4 All users will initially be allocated 512 MByte of user space This will appear as their home directory as well as their Windows U drive and My Documents folder 5 he 512 MByte allocation will not be deleted and will be backed up If data is present in this area for a minimum of 24 hours it should be recoverable 6 Extensions beyond 512 MByte may be provided if resources permit and the extension is approved by the PBS of the beamline for which the user has allocated time 7 Users will have access to science data collected on their beamline during their allocated time from any Diamond machine Page 4 of 40 10 11 12 13 14 15 16 Issue 1 12 Science data collected on beamlines will only be guaranteed available from DLS for a limited period prese
30. queror Nautilus can be used to navigate to the relevant directory Nautilus gnome Konqueror KDE Page 8 of 40 Issue 1 12 ind m V a OSOE Location dit view Go Bookmarks Tods Settings Window Help 0 609010 Gl4 I A Anga m 2 D utm a e e Nam Site Modified Permissions Owner b22 E 40KB Folder 200712191629 mwarwx mol le Of Folder 2000 01 20 11 20 ravxrx root Eit 0B Folder 2000 01 20 11 79 haere toot zXmi misc 08 Folder 2009 01 20 11 29 PEIKA root bes scratch 08 Folder 2009 01 20 11 29 TA XEX mot cs scratch i02 202 08 Folder 2009 01 20 11 29 TAXIACE K mot 03 08 Folder 2009 01 20 11 20 rxrxrx tool ee e OR Folder 20090120 11 29 rararx root OF 08 Folder 2009 01 20 11 20 rxexrx root i051 E306 1 08 Folder 2006 61 20 11 29 rarxrx toot ES 08 Folder 2009 01 20 11 20 rxtxtx root 15 O8 Folder 2009 01 20 11 29 PANKIX foot e 6 OO Folder 2009 01 20 11 29 r xrxrx root e 8 08 Folder 2009 0120 11 29 rrr root r zt 08 Folder 2009 91 20 11 29 FX KI X root E322 08 Folder 2009 01 20 11 29 trt foot Of Folder 2099 01 20 11 20 Xr mot e ocal 08 Folder 2009 01 20 11 29 rarxrx toot hneStatus 08 Folder 2009 01 20 11 20 fre root 08 Folder 2009 01 20 11 29 MOREA toot Sm scraich OR Folder 7009 01 20 11 29 TACITA root E03 08 Folder 2009 01 20 11 20 E root Sdoos physics OR Folder 2008 01 20 11 29 AXI root 5 68 Folder 2000 01 20 11 20 rxextx toot eA AQ 08 Folder 20090120 11 29 PAPRA root dev C
31. ss print diamond ac uk EJ co Name 4 Cormments o y SA b ii3 cc3 cal 1 Colour Laser Jet bA bib cc1 col 1 Colour Laser Jet ba b i06 cc4 col 1 Colour Laser Jet Numam 4 Colour LaserJet HA b il5 cc3 n Colour LaserJet EA bilg cct c Connect Colour LaserJet Ex amp b il8 ccl c Create shortcut Colour LaserJet badh 1 03 col on A2 colour printer 5mdh i 19 bw Properties A4 mona duplex Eg dh 1 19 bw 2 visitors Printer Es dh 1 19 cal 1 435 44 color duplex a dh 1 19 col 2 Cannon Mopier Figure 23 Connecting to a printer on print diamond ac uk The driver will be automatically installed and the printer made available on the system Page 31 of 40 Issue 1 12 Alternatively open the Printers and Faxes dialogue box Start Settings Printers and Faxes Double click Add Printer to open the add printer wizard Choose A network printer or a printer attached to another computer in the second step which type of printer to use and click next In step three Specify a Printer select Connect to this printer or to browse for a printer select this option and click Next and type print diamond ac uk name of printer e g for printer dh g 51 col 1 type print diamond ac uk dh g 51 col 1 Click next Windows should now install the printer driver see Figure 24 Add Printer Wizard Add Printer Wizard j Local or Network Printer Specify a Printer in The wizard needs
32. subdirectory named after your proposal contains your experimental data Page 17 of 40 Issue 1 12 8s data diamond ac uk b16 data 2008 PME File Edit View Favorites Tools Help qe pack Neg 2 p Search Folders ER X iej mE Address data diamond ac ukibi6idatajz008 ir Size Type Date Modified File Folder OF 03 2008 13 54 File Folder OF 03 2008 13 54 C mt355 1 File Folder OSf02 2008 15 52 mt355 2 File Folder 08 02 2008 10 59 1 objects selected 0 bytes a Local intranet Figure 12 An example Beamline experimental data directory Please remember that the directories are read only You can copy this data to your own computer but you will not be able to write back to the file server This is also true for your home directory Alternative sftp data diamond ac uk can also be directly reached via sftp from the visitors network i e data can also be transferred using sftp Using sftp you can also write to the directories given you have write access to them e g your home directory On Windows a possible sftp client is WinSCP Diamond IT helpdesk might be able to help you set this up You will have to log in using your Fedld maybe as CLRC Fedld as described above Note this will probably drop you into your home directory home yourFedld from where you will have to navigate into the data directory you want to go to The data directory can be found as dis BEAMLINE data YEAR EXPERIMENT VISIT T
33. system administrators must be given root or administrator access to the computing equipment so it can be configured and scanned for potential problems If the equipment is deemed to be acceptable it will then be allowed to be connected to a designated Diamond network for a strictly limited period of time 18 Diamond will provide some public computing systems on the visitor network to allow visitors to access the internet even without need for a Fedld 3 DATA TRANSFER SPEEDS Users should consider the volume of data that is expected to be collected This will depend on length of experiment complexity and detector type For example a typical MX user may expect to collect some hundreds of gigabytes in certain circumstances Consideration should be given to transfer of data before the experiment has begun Leaving the copy process until the end of the experiment may result in long delays Diamond has developed a Data Dispenser which can be set up to transfer data continuously during an experiment to a user s own disks thereby reducing delays at the end of a visit due to data copying Below is an indication of typical transfer speeds for different copy processes and media eSATA disk Approximately 40 50MB s USB disk Approximately 25 35MB s SSH over LAN Approximately 20 40MB s depending on protocol SMB CIFS Windows Approximately 25 35MB s file share Hemote from Diamond Approximately 10 30MB s Users should remember that transfers to
34. to know which type of pnnber bo set up li you dort know the name or address of the printer you can search For a printer l iha meets your needs Select the opbon that describes the printer you want to use What printer do pou want bo connect to Local printer attached bo this computer W B3 maie deleet and iral ma Phi Pay grirfel p inter of a pinter altached to another computer Connect to this peirter or bo browse for a printer mx this option and cick Next C Find a printer in the directory Mame pinihdh a 51 coL 1 dh g 05 bw 1 dh g 53 bw 1 2 To sat up a network printer that is not attached to a print server URL 1j use Ehe Local printer aplion Example File sereer printers miprintes printer Back Het gt Cancel Back Hex gt Cancel Figure 24 Connecting to a Diamond printer using the Add Printer wizard 1 1 3 Connecting to the print server from a Linux computer or a Mac On Diamond Linux workstations all printers should be automatically available However in case they are not or you want to connect a non Diamond workstation e g a Mac from the Visitor network there are different ways of connecting to the print server 1 for all users on that computer requires root access You need to have a local cups server running on the computer Simply as root add the following line to etc cups cupsd conf BrowsePoll print diamond ac uk Then restart CUPS service
35. umask Set local umask to umask mkdir path Create remote directory show progress Toggle display of progress meter put P local path remote path Upload file you pud lisplau remote working directory OUI quit Quit sftp y rename oldpath newpath Rename remote file curren rm path Delete remote file rmdir path Remove remote directory t sumlink aldpath newpath Sumlink remote File direct version Show SFTP version command Execute command in local shell Ory Escape to local hell Synonym for help Is list its p conie EE ET FAC ERES qqs43472l9usO40 sftp data diamond ac uk n Connecting to data diamaond ac uk and cd sftp pud Remote working directory home qgsd444 2 change sftp cd dlas bib data z014 the sftp pud Remote working directory dlsz blbB data zU014 directory sftp ls cm4969 1 cm43B53 2 eeS8bbb 1 inbo u 1 mt4384 1 mt 345 mt3104 2 mt9140 1 mt3143 1 mt354z 1 mt85 2 1 mt357l 1 mt38b5z 1 mt8855 1 ntsB800 2 nt4313 2 nt4813 3 ntb asB8 1 ntb 4bh 1 ntbB8rl 3 ntb5b871 4 sftp l Figure 7 sftp accesss to data diam Issue 1 12 ond ac uk from Linux command line second figure Get will download a file or directory from the server into the directory you are currently in on your computer as shown by Ipwd Put can be used to upload files to the current working directory on the server provided you have write access to it Using one of the file browsers the connection
36. xplicit AUTH TLS E SFTP SSH File Transfer Protocol WebDAV Web based Distributed Authoring and Versioning WebDAV HTTP SSL gt MobileMe iDisk WebDAV EM 53 Amazon Simple Storage Service B Google Storage Eucalyptus Walrus 3 Rackspace Cloud Files Swift OpenStack Object Storage 9 Google Docs Windows Azure Cloud Storage storage clouddrive com https tinagistarage clouddrive com Unlimited File Storage amp Content Delivery svn cyberduck ch WebDAV HTTP svn cyberduck ch http ftina amp gsvn cyberduck ch trunk Figure 18 connect via SFTP using Cyberduck Connect to data diamond ac uk using your FedlD password Page 22 of 40 Issue 1 12 Geta donation key Mp Server data diamond ac uk URL sftp kdf51254data diamond ac uk 22 Username kdf51254 Anonymous Login Add to Keychain Cancel b More Options Rackspace Cloud Files starage clouddrive com https tinagistorage clouddrive com Unlimited File Storage amp Content Delivery svn cyberduck ch WebDAV HTTP syn cyberduck ch http tina amp isvn cyberduck ch trunk Figure 19 enter FedlD password in Cyberduck This will drop you into your home home FedID directory from where you will have to navigate to the beamline data directory You need to choose the top level directory and then find dls BEAMLINE data YEAR EXPERIMENT VISIT Page 23 of 40 Issue 1 12 aoe data diamond ac uk SFTP Get a donatio
37. xr x E3downlcads 8 0 KB Folder 2008 01 03 14 02 rworwxr x E3dset wohba 4 0 KB Folder 2007 06 11 11 02 max 4 0 KB Folder 2008 11 03 14 09 marmara 4 0 KB Folder 2007 08 08 12 39 nwxwarx 339 itens 274 Files 426 4 MB Total 65 Folders Figure 8 sftp to data using Nautilus or Konqueror on Linux then browse to the correct directory dls BEAMLINE data YEAR EXPERIMENT VISIT 4 1 2 Data access from Windows The beamline data directories are visible on the server data diamond ac uk and can be accessed using Windows Explorer Open an Explorer window e g My Computer or My Network Places and enter data diamond ac uk into the address bar Note that if you are not logged in to your workstation using your Fedld you Will have to log in before being able to see any files You will probably have to enter your username as CLRC yourFedID see Figure 9 note the CLRC and use your normal Diamond password Page 15 of 40 Issue 1 12 zr My Computer Sela ay File Edit View Favorites Tools Help aE oO Back d JJ Search ley Folders m E X Ei RG Er d My Computer d Se Connect to data diamond ac uk Wea ki Connecting to data diamond ac uk User name i ciRcinpr7s Password 909909099 Remember my password objects d My Computer E Figure 9 Logging in to data diamond ac uk on Windows Once you have logged in it will show all
Download Pdf Manuals
Related Search