Home

It comes in the night and sucks the essence from your

1. 59 Counter Table Layout o 59 Version Table Layout o 59 Base Files Table Layout o o 59 File Attributes sensy ii a Ge oe 78 Bacula Developer Notes General This document is intended mostly for developers and describes the the gen eral framework of making Bacula source changes Contributions Contributions from programmers are broken into two groups The first are contributions that are aids and not essential to Bacula In general these will be scripts or will go into and examples or contributions directory For these kinds of non essential contributions there is no obligation to do a copyright assignment as described below However a copyright assignment would still be appreciated The second class of contributions are those which will be integrated with Bacula and become an essential part Within this class of contributions there are two hurdles to surmount One is getting your patch accepted and two is dealing with copyright issues The following text describes some of the requirements for such code Patches Subject to the copyright assignment described below your patches should be sent in diff u format relative to the current contents of the Source Forge SVN which is the easiest to understand and integrate Please be sure to use the Bacula indenting standard see below If you have checked out the source with SVN you can get a diff using svn upd
2. lt file index gt lt stream id gt lt info gt The details are specified in the Daemon Protocol section of this document append abort session lt ticket number gt The open append session with ticket ticket number is aborted any blocks not yet written to permanent media are discarded Subsequent attempts to append data to the session will receive an error status of 3505 Session aborted append end session lt ticket number gt The open append ses sion with ticket ticket number is marked complete no further blocks may be appended The storage daemon will give prior ity to saving any buffered blocks from this session to permanent media as soon as possible append close session lt ticket number gt The append session with ticket ticket is closed This message does not receive an 3000 OK reply until all of the content of the session are stored on permanent media at which time said reply is given followed by a list of volumes from first to last which contain blocks from the session along with the first and last file and block on each containing session data and the volume session key identifying data from that session in lines with the following format Volume lt Volume id gt lt start file gt lt start block gt lt end file gt lt end block gt lt volume session id gt where Volume id is the volume label start file and start block are the file and block containing the first data from that session on the volume end f
3. The Unix File Attributes packet consists of the following lt File Index gt lt Type gt lt Filename gt lt File Attributes gt lt Link gt lt Extended Attributes gt where represents a byte containing a binary zero FileIndex is the sequential file index starting from one assigned by the File daemon Type is one of the following define FT_LNKSAVED define FT_REGE 1 hard link to file already saved 2 Regular file but empty define FT_REG 3 Regular file define FT_LNK 4 Soft Link define FT_DIR 5 Directory define FT_SPEC 6 Special file chr blk fifo sock define FT_NOACCESS 7 Not able to access define FT_NOFOLLOW 8 Could not follow link define FT_NOSTAT 9 Could not stat file define FT_NOCHG 10 Incremental option file not changed define FT_DIRNOCHG 11 Incremental option directory not changed define FT_ISARCH 12 Trying to save archive file define FT_NORECURSE 13 No recursion into directory define FT_NOFSCHG 14 Different file system prohibited define FT_NOOPEN 15 Could not open directory define FT_RAW 16 Raw block device define FT_FIFO 17 Raw fifo device Filename is the fully qualified filename File Attributes consists of the 13 fields of the stat buffer in ASCII base64 format separated by spaces These fields and their mean ings are shown below This stat packet is in Unix format and MUST be p
4. define STREAM_UNIX_ATTRIBUTES_ACCESS_ACL 15 Standard ACL attributes on UNIX define The DataSize STREAM_UNIX_ATTRIBUTES_DEFAULT_ACL 16 Default ACL attributes on UNIX is the size in bytes of the binary data record that follows the Session Record header The Storage Daemon has no idea of the actual contents of the binary data record For standard 69 Unix files the data record typically contains the file attributes or the file data For a sparse file the first 64 bits of the file data contains the storage address for the data block The Record Header is never split across two blocks If there is not enough room in a block for the full Record Header the block is padded to the end with zeros and the Record Header begins in the next block The data record on the other hand may be split across multiple blocks and even multiple physical volumes When a data record is split the second and possibly subsequent piece of the data is preceded by a new Record Header Thus each piece of data is always immediately preceded by a Record Header When reading a record if Bacula finds only part of the data in the first record it will automatically read the next record and concatenate the data record to form a full data record Version BB02 Block Header Each session or Job has its own private block As a consequence the SessionId and SessionTime are written once in each Block Header and not in the Record Header So the second an
5. It comes in the night and sucks the essence from your computers Kern Sibbald May 17 2007 This manual documents Bacula version 2 1 8 16 May 2007 Copyright 1999 2006 Free Software Foundation Europe e V Permission is granted to copy distribute and or modify this document under the terms of the GNU Free Documentation License Version 1 2 published by the Free Software Foundation with no Invariant Sections no Front Cover Texts and no Back Cover Texts A copy of the license is included in the section entitled GNU Free Documentation License Contents Bacula Developer Notes 0 00000025 eG 8 General vs a aa ee ea 8 Code Submissions and Projects 13 SVN Usage sisas backs ere wee ee RR ae es 14 Subversion Resources 0 pees 20 Platform Support coast ka ee eb ae e 33 General ansi ig heey a ne ar Go a felon Ata 33 Platform Requirements 0 00040 G 33 Daemon Protocol bee ee ee ee 35 General VV AA eae 35 Low Level Network Protocol 35 General Daemon Protocol 224 35 Protocol Used Between the Director and the Storage Daemon 36 Protocol Used Between the Director and the File Daemon 37 Save Protocol Between the File Daemon and the Storage Daemon 38 Director Services Daemon 000048 41 File Services Daemon e 42 Commands Received from the Director for a Backup 43 Commands R
6. JobBytes bigint Number of bytes saved in Job JobErrors integer Number of errors during Job JobMissingFiles integer Number of files not saved not yet used Poolld integer Link to Pool Record FileSetId integer Link to FileSet Record PurgedFiles tiny integer Set when all File records purged HasBase tiny integer Set when Base Job run The Job table contains one record for each Job run by Bacula Thus normally there will be one per day per machine added to the database Note the Jobld is used to index Job records in the database and it often is shown to the user in the Console program However care must be taken with its use as it is not unique from database to database For example the user may have a database for Client data saved on machine Rufus and another database for Client data saved on machine Roxie In this case the two database will each have Joblds that match those in another database For a unique reference to a Job see Job below The Name field of the Job record corresponds to the Name resource record given in the Director s configuration file Thus it is a generic name and it will be normal to find many Jobs or even all Jobs with the same Name The Job field contains a combination of the Name and the schedule time of the Job by the Director Thus for a given Director even with multiple Catalog databases the Job will contain a unique name that represents the Job For a given Storage daemon th
7. are certain short passages of text that are listed as Front Cover Texts or Back Cover Texts in the notice that says that the Document is released under this License A Front Cover Text may be at most 5 words and a Back Cover Text may be at most 25 words A Transparent copy of the Document means a machine readable copy represented in a format whose specification is available to the general public that is suitable for revising the document straightforwardly with generic text editors or for images composed of pixels generic paint programs or for drawings some widely available drawing editor and that is suitable for input to text formatters or for automatic translation to a variety of formats suitable for input to text formatters A copy made in an otherwise Transparent file format whose markup or absence of markup has been arranged to thwart or discourage subsequent modification by readers is not Transparent An image format is not Transparent if used for any substantial amount of text A copy that is not Transparent is called Opaque Examples of suitable formats for Transparent copies include plain ASCII without markup Texinfo input format LaTeX input format SGML or XML using a publicly available DTD and standard conforming simple HTML PostScript or PDF designed for human modification Examples of trans parent image formats include PNG XCF and JPG Opaque formats include proprietary formats that can be read and
8. s Cover Texts may be placed on covers that bracket the Document within the aggregate or the electronic equivalent of covers if the Document is in electronic form Otherwise they must appear on printed covers that bracket the whole aggregate 8 TRANSLATION Translation is considered a kind of modification so you may distribute trans lations of the Document under the terms of section 4 Replacing Invariant Sections with translations requires special permission from their copyright holders but you may include translations of some or all Invariant Sections in addition to the original versions of these Invariant Sections You may include a translation of this License and all the license notices in the Doc ument and any Warranty Disclaimers provided that you also include the original English version of this License and the original versions of those notices and disclaimers In case of a disagreement between the translation and the original version of this License or a notice or disclaimer the original version will prevail If a section in the Document is Entitled Acknowledgements Dedica tions or History the requirement section 4 to Preserve its Title sec tion 1 will typically require changing the actual title 9 TERMINATION You may not copy modify sublicense or distribute the Document except as expressly provided for under this License Any other attempt to copy modify sublicense or distribute the Doc
9. FileIndex INTEGER UNSIGNED NOT NULL DEFAULT O JobId INTEGER UNSIGNED NOT NULL REFERENCES Job PathId INTEGER UNSIGNED NOT NULL REFERENCES Path FilenameId INTEGER UNSIGNED NOT NULL REFERENCES Filename MarkId INTEGER UNSIGNED NOT NULL DEFAULT 0 LStat TINYBLOB NOT NULL MD5 TINYBLOB NOT NULL PRIMARY KEY Fileld INDEX JobId INDEX PathId INDEX Filenameld CREATE TABLE Job JobId INTEGER UNSIGNED NOT NULL AUTO_INCREMENT Job TINYBLOB NOT NULL Name TINYBLOB NOT NULL Type BINARY 1 NOT NULL Level BINARY 1 NOT NULL ClientId INTEGER NOT NULL REFERENCES Client JobStatus BINARY 1 NOT NULL SchedTime DATETIME NOT NULL StartTime DATETIME NOT NULL EndTime DATETIME NOT NULL 60 JobTDate BIGINT UNSIGNED NOT NULL VolSessionId INTEGER UNSIGNED NOT NULL DEFAULT 0 VolSessionTime INTEGER UNSIGNED NOT NULL DEFAULT O JobFiles INTEGER UNSIGNED NOT NULL DEFAULT O JobBytes BIGINT UNSIGNED NOT NULL JobErrors INTEGER UNSIGNED NOT NULL DEFAULT 0 JobMissingFiles INTEGER UNSIGNED NOT NULL DEFAULT O PoolId INTEGER UNSIGNED NOT NULL REFERENCES Pool FileSetId INTEGER UNSIGNED NOT NULL REFERENCES FileSet PurgedFiles TINYINT NOT NULL DEFAULT O HasBase TINYINT NOT NULL DEFAULT 0 PRIMARY KEY JobId INDEX Name 128 CREATE TABLE FileSet FileSetId INTEGER UNSIGNED NOT NULL AUTO_INCREMENT FileSet TINYBLOB NOT NULL MD5 TINYBLOB NOT NULL CreateTime DATETIME NOT NULL PRIMARY KEY FileSetId X CREATE TABLE JobMedia Jo
10. Tar file depkgs mtx autochanger control program tape drive info sqlite SQLite database program Tar file depkgs win32 pthreads Native win32 pthreads library dll zlib Native win32 zlib library wx wxWidgets source code Project bacula bacula main source directory containing configuration and installation files autoconf automatic configuration files not normally used by users intl programs used to translate platforms OS specific installation files redhat Red Hat installation solaris Sun installation freebsd FreeBSD installation irix Irix installation not tested 23 unknown cats console dird filed win32 findlib gnome console lib stored tconsole testprogs tools win32 baculafd compat filed findlib lib console wx console wx console Project regress regress bin build scripts tests tmp working Project docs docs developers home page manual manual fr manual de techlogs Project rescue rescue linux cdrom solaris freebsd Project gui l gui bacula web Default if system not identified translations of source strings source directory contains global header files SQL catalog database interface directory bacula user agent directory Director daemon Unix F
11. Yes No VolRetention bigint 64 bit seconds until expiration VolUseDuration bigint 64 bit seconds volume can be used MaxVolJobs integer maximum jobs to put on Volume MaxVolFiles integer maximume EOF marks to put on Volume The Volume table internally referred to as the Media table contains one entry for each volume that is each tape cassette 8mm DLT DAT or file on which information is or was backed up There is one Volume record created for each of the NumVols specified in the Pool resource record Pool Column Name Data Type Remark Poolld integer Primary Key Name Tinyblob Pool Name Num Vols Integer Number of Volumes in the Pool Max Vols Integer Maximum Volumes in the Pool UseOnce tinyint Use volume once UseCatalog tinyint Set to use catalog AcceptAnyVolume tinyint Accept any volume from Pool VolRetention bigint 64 bit seconds to retain volume VolUseDuration bigint 64 bit seconds volume can be used MaxVolJobs integer max jobs on volume MaxVolFiles integer max EOF marks to put on Volume MaxVolBytes bigint max bytes to write on Volume AutoPrune tinyint yes no for autopruning Recycle tinyint yes no for allowing auto recycling of Volume PoolType enum Backup Copy Cloned Archive Mi gration LabelFormat Tinyblob Label format The Pool table contains one entry for each media pool controlled by Bacula in this database One
12. move or archive data that is on a device to another device is very important 12 Why An ISP might want to backup to disk but after 30 days migrate the data to tape backup and delete it from disk Bacula should be able to handle this automatically It needs to know what was put where and when and what to migrate it is a bit like retention periods Doing so would allow space to be freed up for current backups while maintaining older data on tape drives Notes Migration could be triggered by Number of Jobs Number of Volumes Age of Jobs Highwater size keep total size Lowwater mark Bacula Code Submissions and Projects Getting code implemented in Bacula works roughly as follows Kern is the project manager but prefers not to be a gate keeper There are growing numbers of contributions very good Some contributions come in the form of relatively small patches which Kern reviews integrates documents tests and maintains All Bacula developers take full responsibility for writing the code post ing as patches so that I can review it as time permits integrating it at an appropriate time responding to my requests for tweaking it name changes document it in the code document it in the manual even though their mother tongue is not English test it develop and com mit regression scripts and answer in a timely fashion all bug reports even occassionally accepting additional bugs This
13. tled Dedications You must delete all sections Entitled Endorsements 6 COLLECTIONS OF DOCUMENTS You may make a collection consisting of the Document and other documents released under this License and replace the individual copies of this License in the various documents with a single copy that is included in the collection provided that you follow the rules of this License for verbatim copying of each of the documents in all other respects You may extract a single document from such a collection and distribute it individually under this License provided you insert a copy of this License into the extracted document and follow this License in all other respects regarding verbatim copying of that document 7 AGGREGATION WITH INDEPENDENT WORKS A compilation of the Document or its derivatives with other separate and independent documents or works in or on a volume of a storage or distri bution medium is called an aggregate if the copyright resulting from the compilation is not used to limit the legal rights of the compilation s users be yond what the individual works permit When the Document is included in an aggregate this License does not apply to the other works in the aggregate which are not themselves derivative works of the Document If the Cover Text requirement of section 3 is applicable to these copies of the Document then if the Document is less than one half of the entire aggregate the Document
14. Begin Session Label is similar to the Volume Label except that it contains additional information pertaining to the Session End Session Label The End Session Label is a special record placed by the Storage daemon on the storage medium as the last record of an append session job with a File daemon The End Session Record is distinguished by a FileIndex with a value of minus two 2 This record is useful for detecting the end of a particular session since no records with the same VolSessionId and VolSes sionTime will follow this record This record is not normally visible outside of the Storage daemon The End Session Label is similar to the Volume Label except that it contains additional information pertaining to the Session 66 Storage Daemon File Output Format The file storage and tape storage formats are identical except that tape records are by default blocked into blocks of 64 512 bytes except for the last block which is the actual number of bytes written rounded up to a multiple of 1024 whereas the last record of file storage is not rounded up The default block size of 64 512 bytes may be overrid den by the user some older tape drives only support block sizes of 32K Each Session written to tape is terminated with an End of File mark this will be removed later Sessions written to file are simply appended to the end of the file Overall Format A Bacula output file consists of Blocks of data Each block contains a block h
15. RFC 1321 in April 1992 and was placed into the public domain at that time Message digest algorithms such as MD5 are not deemed encryption technology and 105 are not subject to the export controls some governments impose on other data security products Obviously the responsibility for obeying the laws in the jurisdiction in which you reside is entirely your own but many com mon Web and Mail utilities use MD5 and I am unaware of any restrictions on their distribution and use The MD5 algorithm has been implemented in numerous computer languages including C Perl and Java if you re writing a program in such a language track down a suitable subroutine and incorporate it into your program The program described on this page is a command line implementation of MD5 intended for use in shell scripts and Perl programs it is much faster than computing an MD5 signature directly in Perl This md5 program was originally developed as part of a suite of tools intended to monitor large collections of files for example the contents of a Web site to detect cor ruption of files and inadvertent or perhaps malicious changes That task is now best accomplished with more comprehensive packages such as Tripwire but the command line md5 component continues to prove useful for veri fying correct delivery and installation of software packages comparing the contents of two different systems and checking for changes in specific files Option
16. Remark Filenameld integer Primary Key Name Blob Filename The Filename table shown above contains the name of each file backed up with the path removed If different directories or machines contain the same filename only one copy will be saved in this table Path Column Name Data Type Remark PathId integer Primary Key Path Blob Full Path The Path table contains shown above the path or directory names of all directories on the system or systems The filename and any MSDOS disk name are stripped off As with the filename only one copy of each directory name is kept regardless of how many machines or drives have the same directory These path names should be stored in Unix path name format Some simple testing on a Linux file system indicates that separating the filename and the path may be more complication than is warranted by the space savings For example this system has a total of 89 097 files 60 467 of which have unique filenames and there are 4 374 unique paths Finding all those files and doing two stats per file takes an average wall clock time of 1 min 35 seconds on a 400MHz machine running RedHat 6 1 Linux Finding all those files and putting them directly into a MySQL database with the path and filename defined as TEXT which is vari able length up to 65 535 characters takes 19 mins 31 seconds and creates a 27 6 MByte database Doing the same thing but inserting t
17. Set this to with tcp wrappers or without tcp wrappers TCPWRAPPERS with tcp wrappers Set this to to disable OpenSSL support with openssl yes to enable it or provide the path to the OpenSSL installation eg with openssl usr local OPENSSL with openssl BACULA SOURCE should be the full path to the Bacula source code that you wish to test It will be loaded configured compiled and installed with the make setup command which needs to be done only once each time you change the source code EMAIL should be your email addres Please remember to change this or I will get a flood of unwanted messages You may or may not want to see these emails In my case I don t need them so I direct it to the bit bucket SMTP yOSTde fineswhereyourS MT Pserveris SQLITE_DIR should be the full path to the sqlite package must be build before running a Bacula regression if you are us ing SQLite This variable is ignored if you are using MySQL or PostgreSQL To use PostgreSQL edit the Makefile and change or add WHICHDB with postgresql For MySQL use WHICHDB with mysql The advantage of using SQLite is that it is totally independent of any installation you may have running on your system and there is no special configuration or authorization that must be done to run it With both MySQL and PostgreSQL you must pre install the packages initialize them and ensure that you have authoriza
18. Storage Dae mon 36 Queued Job Messages 32 Record 64 Record Header 66 68 Regression Running the Disk Only 100 Requests SD Append 45 SD Read 46 Requirements Platform 33 Porting 84 Restore Commands Received from the Director for a 43 Running a Single Test 103 Running the Disk Only Regres sion 100 Running the Regression Script 97 Running the Tests by Hand 103 13 Save Protocol Between the File Daemon and the Storage Daemon 38 Script Running the Regression 97 SD Append Requests 45 SD Connections and Sessions 45 SD Data Structures 47 SD Design Introduction 44 SD Development Outline 44 SD Read Requests 46 See Also 107 Sequence of Creation of Records for a Save Job 51 Serialization 67 Server Becoming a 114 Services Catalog 49 Session 65 Session Label 71 Sessions SD Connections and 45 Setting the Configuration Param eters 97 Setting up your SQL engine 99 Smart Memory Allocation With Orphaned Buffer Detec tion 116 SMARTALLOC Installing 116 Squelching a 117 SMARTALLOC Details 119 Socket Blocking State Manipulating 94 SPAN class 45 47 SQLite Installing and Configuring 50 Squelching a SMARTALLOC 117 Statically Allocated Memory 108 Status Exit 107 Steps to Take for Porting 85 Storage Daemon Design 44 Storage Daemon File Output For mat 67 Storage Media Output Format 64 Stream 65 69 Structure Directory 104
19. Support Platform 33 Synopsis 105 Tables Database 51 TCP IP Network Protocol 112 Test Adding a New 104 Writing a Regression 103 Test and Demonstration Program 122 Testing Bacula Regression 96 Tests Other 101 Threads bnet and 112 TLS 90 TLS API Implimentation 91 TLS Configuration Directives 90 TLS Connection Negotiating 94 TLS Connection Manipulation 92 TLS Context Manipulation 92 TLS Introduction 90 TLS Post Connection Verifica tion 92 Type 77 Underhandedness Overlays and 122 Unix File Attributes 77 Utility Command Line Message Di gest 105 Verification TLS Post Connection 92 Version 2 Record Header 70 Version BB02 Block Header 70 VolSessionId 65 69 VolSessionTime 65 69 Volume Label 66 Volume Label Format 71 When SMARTALLOC is Disabled 121 Writing a Regression Test 103
20. and the closing brace is on a line by itself E g if abc some_code Just follow the convention in the code Originally I indented case clauses under a switch but now I prefer non indented cases switch code case A do something break case B again break default break Avoid using style comments except for temporary code or turning off debug code Standard C comments are preferred this also keeps the code closer to C Attempt to keep all lines less than 85 characters long so that the whole line of code is readable at one time This is not a rigid requirement 27 Always put a brief description at the top of any new file created de scribing what it does and including your name and the date it was first written Please don t forget any Copyrights and acknowledgments if it isn t 100 your code Also include the Bacula copyright notice that is in src c In general you should have two includes at the top of the an include for the particular directory the code is in for includes are needed but this should be rare In general except for self contained packages prototypes should all be put in protos h in each directory Always put space around assignment and comparison operators a 1 if b gt 2 cleanup but your can compress things in a for statement for i 0 i lt del num_ids i Don t overuse the inline if A full if is preferred excep
21. assertion in the free function will fail This catches incorrect program code that stores beyond the storage allocated for the buffer At free time the queue links are also validated and an assertion failure will occur if the program has destroyed them by storing before the start of the allocated storage In addition when a buffer is released with free its contents are imme diately destroyed by overwriting them with the hexadecimal pattern 0xAA alternating bits the one s complement of the initial value pattern This will usually trip up code that keeps a pointer to a buffer that s been freed and later attempts to reference data within the released buffer Incredibly this is legal in the standard Unix memory allocation package which permits programs to free buffers then raise them from the grave with realloc Such program logic should be fixed not accommodated and SMARTAL LOC brooks no such Lazarus buffer nonsense Some C libraries allow a zero size argument in calls to malloc Since this is far more likely to indicate a program error than a defensible programming stratagem SMARTALLOC disallows it with an assertion When the standard library realloc function is called to expand a buffer it attempts to expand the buffer in place if possible moving it only if nec essary Because SMARTALLOC must place its own private storage in the buffer and also to aid in error detection its version of realloc always moves
22. but enough so that I can understand it Look at the current code and you will see that I document more than most but am definitely not a fanatic I prefer simple linear code where possible Gotos are strongly discour aged except for handling an error to either bail out or to retry some code and such use of gotos can vastly simplify the program 25 Remember this is a C program that is migrating to a tiny subset of C so be conservative in your use of C features Do Not Use STL it is totally incomprehensible Avoid if Possible Using void because this generally means that one must using casting and in C casting is rather ugly It is OK to use void to pass structure address where the structure is not known to the routines accepting the packet typically callback routines However declaring void buf is a bad idea Please use the correct types whenever possible Using undefined storage specifications such as short int long long long size Theproblemwithalltheseisthatthenumbero f bytestheyallocatedependsonthecompi Returning a malloc ed buffer from a subroutine someone will forget to release it Heap allocation malloc unless needed it is expensive Use POOLyEMinstead Templates they can create portability problems Fancy or tricky C or C code unless you give a good explanation of why you used it Too much inheritance it can complicate the c
23. bytes in network byte order The data is read into sock gt msg and is sock gt msglen bytes If the sock gt msg is not large enough bnet_recv realloc the buffer It will return an error 2 if maxbytes is less than the record size sent It returns 113 Returns number of bytes read Returns O on end of file Returns 1 on hard end of file i e network connection close XA XX Returns 2 on error It should be noted that bnet_recv is a blocking read bnet_sig To send a signal from one daemon to another one uses the subroutine int bnet_sig BSOCK sock SIGNAL where SIGNAL is one of the following 1 BNET_EOF deprecated use BNET_EOD 2 BNET_EOD End of data stream new data may follow 3 BNET_EOD_POLL End of data and poll all in one 4 BNET_STATUS Request full status 5 BNET_TERMINATE Conversation terminated doing close 6 BNET_POLL Poll request I m hanging on a read 7 BNET_HEARTBEAT Heartbeat Response requested 8 BNET_HB RESPONSE Only response permitted to HB 9 BNET_PROMPT Prompt for UA bnet_strerror Returns a formated string corresponding to the last error that occurred bnet_close The connection with the server remains open until closed by the subroutine void bnet_close BSOCK sock Becoming a Server The bnet_open and bnet_close routines described above are used on the client side to establish a connection and terminate a connection with the server To bec
24. do is to turn on debug output A good place to start is with a debug level of 20 as in startit d20 The startit command starts all the daemons with the same debug level Alternatively you can start the appropriate daemon with the debug level you want If you really need more info a debug level of 60 is not bad and for just about everything a level of 200 Using a Debugger If you have a serious problem such as a segmentation fault it can usually be found quickly using a good multiple thread debugger such as gdb For example suppose you get a segmentation violation in bacula dir You might use the following to find the problem lt start the Storage and File daemons gt cd dird gdb bacula dir run f s c dird conf lt it dies with a segmentation fault gt where The f option is specified on the run command to inhibit dird from going into the background You may also want to add the s option to the run command to disable signals which can potentially interfere with the debugging As an alternative to using the debugger each Bacula daemon has a built in back trace feature when a serious error is encountered It calls the debugger on itself produces a back trace and emails the report to the developer For more details on this please see the chapter in the main Bacula manual entitled What To Do When Bacula Crashes Kaboom Memory Leaks Because Bacula runs routinely and unattended on client and server machines it may
25. is a sustainable way of going forward with Bacula and the direc tion that the project will be taking more and more For example in the past we have had some very dedicated programmers who did ma jor projects However these programmers due to outside obligations job responsibilities change of job school duties could not con tinue to maintain the code In those cases the code suffers from lack of maintenance sometimes I patch it sometimes not In the end the code gets dropped from the project there are two such contributions 13 that are heading in that direction When ever possible we would like to avoid this and ensure a continuation of the code and a shar ing of the development debugging documentation and maintenance responsibilities SVN Usage Please note that if you are familar with CVS SVN is very similar and better but there can be a few surprising differences The entire Bacula SourceForge net Subversion repository can be checked out through SVN with the following command svn checkout https bacula svn sourceforge net svnroot bacula bacula With the above command you will get everything which is a very large amount of data branches Branch 1 32a Branch 2 0 import vendor tags Release 1 1 Release 2 0 2 trunk bacula docs gui regress rescue Note you should NEVER commit code to any checkout that you have done of a tag All tags e g Release 1 1 Release 2 0
26. library can be written to support almost any architecture Currently Linux FreeBSD and Solaris are fully supported platforms which means that the code has been tested on those machines and passes a full set of regression tests In addition the Windows File daemon is supported on most versions of Windows and finally there are a number of other platforms where the File daemon client is known to run NetBSD OpenBSD Mac OSX SGI Requirements to become a Supported Platform As mentioned above in order to become a fully supported platform it must support POSIX Unix system calls In addition the following requirements must be met The principal developer currently Kern must have non root ssh access to a test machine running the platform The ideal requirements and minimum requirements for this ma chine are given below There must be a defined platform champion who is normally a system administrator for the machine that is available This per son need not be a developer programmer but must be familiar with system administration of the platform There must be at least one person designated who will run re gression tests prior to each release Releases occur approximately once every 6 months but can be more frequent It takes at most a day s effort to setup the regression scripts in the beginning and after that they can either be run daily or on demand before a release Running the regression scripts
27. old one on explicit permission from the previous publisher that added the old one The author s and publisher s of the Document do not by this License give permission to use their names for publicity for or to assert or imply endorsement of any Modified Version 5 COMBINING DOCUMENTS You may combine the Document with other documents released under this License under the terms defined in section 4 above for modified versions provided that you include in the combination all of the Invariant Sections of all of the original documents unmodified and list them all as Invariant Sections of your combined work in its license notice and that you preserve all their Warranty Disclaimers The combined work need only contain one copy of this License and multiple identical Invariant Sections may be replaced with a single copy If there are multiple Invariant Sections with the same name but different contents make the title of each such section unique by adding at the end of it in parentheses the name of the original author or publisher of that section if known or else a unique number Make the same adjustment to the section titles in the list of Invariant Sections in the license notice of the combined work In the combination you must combine any sections Entitled History in the various original documents forming one section Entitled History likewise combine any sections Entitled Acknowledgements and any sections Enti
28. on your machine To correct problems with detection of your system type or with routines and libraries you must edit the file lt bacula src gt autoconf configure in This is the source from which configure is built In general most of the changes for your system will be made in autoconf aclocal m4 in the routine BA_CHECK_OPSYS or in the routine BA_CHECK_OPSYS_DISTNAME I have already added the necessary code for most systems but if yours shows up as unknown you will need to make changes Then as mentioned above you will need to set a number of system dependent items in configure in in the case statement at approximately line 1050 depending on the Bacula release The items to in the case statement that corresponds to your sys tem are the following x DISTVER set to the version of your operating system Typically some form of uname obtains it x TAPEDRIVE the default tape drive Not too important as the user can set it as an option x PSCMD set to the ps command that will provide the PID in the first field and the program name in the second field If this is not set properly the bacula stop script will most likely not be able to stop Bacula in all cases x hostname command to return the base host name non qualified of your system This is generally the machine name Not too important as the user can correct this in his configuration file x CFLAGS set any special compiler flags needed Many sys tems need
29. or with sqlite on the configure line Bacula will use its minimal ist internal database This database is kept for build reasons but is no longer supported Bacula requires one of the three databases MySQL PostgreSQL or SQLite to run Filenames and Maximum Filename Length In general either MySQL PostgreSQL or SQLite permit storing ar bitrary long path names and file names in the catalog database In 49 practice there still may be one or two places in the Catalog interface code that restrict the maximum path length to 512 characters and the maximum file name length to 512 characters These restrictions are believed to have been removed Please note these restrictions apply only to the Catalog database and thus to your ability to list online the files saved during any job All information received and stored by the Storage daemon normally on tape allows and handles arbitrarily long path and filenames Installing and Configuring MySQL For the details of installing and configuring MySQL please see the Installing and Configuring MySQL chapter of this manual Installing and Configuring PostgreSQL For the details of installing and configuring PostgreSQL please see the Installing and Configuring PostgreSQL chapter of this manual Installing and Configuring SQLite For the details of installing and configuring SQLite please see the Installing and Configuring SQLite chapter of this manual Internal Bacula Catalog P
30. pass a buffer allocated by SMARTALLOC s allocation routines as it contains special information that the system free doesn t expect to be there The following code uses actuallymalloc to obtain the buffer passed to such a routine struct image obuf struct image actuallymalloc sizeof struct image dump_screen_to_image obuf putimage scrdump img obuf putimage releases obuf It s unlikely you ll need any of the actually calls except under very odd cir cumstances in four products and three years I ve only needed them once but they re there for the rare occasions that demand them Don t use them to subvert the error checking of SMARTALLOC if you want to disable or phaned buffer detection use the sm_static 1 mechanism described above That way you don t forfeit all the other advantages of SMARTALLOC as you do when using actuallymalloc and actuallyfree SMARTALLOC Details When you include smartall h and define SMARTALLOC the following standard system library functions are redefined with the define mechanism to call corresponding functions within smartall c instead For details of the redefinitions please refer to smartall h void malloc size_t size void calloc size_t nelem size_t elsize void realloc void ptr size_t size void free void ptr void cfree void ptr cfree is a historical artifact identical to free In addition to allocating storage in the s
31. pooled Memory In order to facility the handling of arbitrary length filenames and to effi ciently handle a high volume of dynamic memory usage we have imple mented routines between the C code and the malloc routines The first is called Pooled memory and is memory which once allocated and then re leased is not returned to the system memory pool but rather retained in a Bacula memory pool The next request to acquire pooled memory will re turn any free memory block In addition each memory block has its current size associated with the block allowing for easy checking if the buffer is of sufficient size This kind of memory would normally be used in high volume situations lots of malloc s and free s where the buffer length may have to frequently change to adapt to varying filename lengths The non pooled memory is handled by routines similar to those used for pooled memory allowing for easy size checking However non pooled mem ory is returned to the system rather than being saved in the Bacula pool This kind of memory would normally be used in low volume situations few malloc s and free s but where the size of the buffer might have to be adjusted frequently Types of Memory Pool Currently there are three memory pool types PM_NOPOOL non pooled memory PM_FNAME a filename pool PM_MESSAGE a message buffer pool PM_EMSG error message buffer pool Getting Memory To get memory one uses
32. project like Bacula it isn t al ways easy to ensure that the right headers are invoked in the right order there are a few kludges to make this happen i e in a few in clude files because of the chicken and egg problem certain references to typedefs had to be replaced with void Every file should include bacula h It pulls in just about everything with very few exceptions If you have system dependent ifdefing please do it in baconfig h The version number and date are kept in ver sion h Each of the subdirectories console cats dird filed findlib lib stored contains a single directory dependent include file generally the name of the directory which should be included just after the include of bacula h This file for example for the dird directory it is dird h contains either definitions of things generally needed in this directory or it includes the appropriate header files It always includes protos h See below Each subdirectory contains a header file named protos h which con tains the prototypes for subroutines exported by files in that directory protos h is always included by the main directory dependent include file Programming Standards For the most part all code should be written in C unless there is a burning reason to use C and then only the simplest C con structs will be used Note Bacula is slowly evolving to use more and more C Code should have some documentation not a lot
33. turn They are Statically allocated memory Dynamically allocated memory using malloc and free Non pooled memory Pooled memory Statically Allocated Memory Statically allocated memory is of the form char buffer MAXSTRING The use of this kind of memory is discouraged except when you are 100 sure that the strings to be used will be of a fixed length One example of where this is appropriate is for Bacula resource names which are currently limited to 127 characters MAX NAME LENGTH Although this maxi mum size may change particularly to accommodate Unicode it will remain a relatively small value Dynamically Allocated Memory Dynamically allocated memory is obtained using the standard malloc rou tines As in char buf buf malloc 256 This kind of memory can be released with free buf 108 It is recommended to use this kind of memory only when you are sure that you know the memory size needed and the memory will be used for short pe riods of time that is it would not be appropriate to use statically allocated memory An example might be to obtain a large memory buffer for reading and writing files When SmartAlloc is enabled the memory obtained by malloc will automatically be checked for buffer overwrite overflow dur ing the free call and all malloc ed memory that is not released prior to termination of the program will be reported as Orphaned memory Pooled and Non
34. void get_pool_memory pool 109 where pool is one of the above mentioned pool names The size of the memory returned will be determined by the system to be most appropriate for the application If you wish non pooled memory you may alternatively call void get_memory size_t size The buffer length will be set to the size specified and it will be assigned to the PM_NOPOOL pool no pooling Releasing Memory To free memory acquired by either of the above two calls use void free_pool_memory void buffer where buffer is the memory buffer returned when the memory was acquired If the memory was originally allocated as type PM_NOPOOL it will be released to the system otherwise it will be placed on the appropriate Bacula memory pool free chain to be used in a subsequent call for memory from that pool Determining the Memory Size To determine the memory buffer size use size_t sizeof_pool_memory void buffer Resizing Pool Memory To resize pool memory use void realloc_pool_memory void buffer The buffer will be reallocated and the contents of the original buffer will be preserved but the address of the buffer may change Automatic Size Adjustment To have the system check and if necessary adjust the size of your pooled memory buffer use void check_pool_memory_size void buffer size_t new size where new size is the buffer length needed Note if the buffer is already equal to or larger t
35. 2 should be considered read only You may commit code to the most recent item in branches in the above the most recent one is Branch 2 0 If you want to commit code to an older branch then please contact Kern first You may create your own tags and or branches but they should have a name clearly distinctive from Branch Release or Beta which are 14 official names used by the project If you create a tag then you should NEVER commit code to it for the same reason noted above it should serve as a marker for something you released If you create a branch then you are free to commit to it as you wish You may of course commit to the trunk In summary branches Branch nnn tags Release nnn Beta nnn are reserved names to be created only by the project manager or with his OK where the nnn is any sequence of numbers and periods e g 2 0 2 0 1 In addition all tags even those that you create are read only forever Typically tags represent release points either in the trunc or in a branch Coming back to getting source code If you only want the current Bacula source code you could use svn checkout https bacula svn sourceforge net svnroot bacula trunk bacula bac To view what is in the SVN point your browser at the following URL http bacula svn sourceforge net viewvc bacula Many of the Subversion svn commands are almost identical to those that you have used for cvs but some such as a che
36. CheckSum a checksum for the Block BlockNumber is the sequential block number on the tape VolSessionld a unique sequential number that is assigned 73 by the Storage Daemon to a particular Job This number is sequential since the start of execution of the daemon VolSessionTime the time date that the current execution of the Storage Daemon started It assures that the combination of VolSessionId and VolSessionTime is unique for all jobs written to the tape even if there was a machine crash between two writes Record Header Format BBO2 FileIndex a sequential file number within a job The Storage daemon enforces this index to be greater than zero and sequential Note however that the File daemon may send multiple Streams for the same FileIndex The Storage Daemon uses negative FileIndices to identify Session Start and End labels as well as the End of Volume labels Stream defined by the File daemon and is intended to be used to identify separate parts of the data saved for each file attributes file data The Storage Daemon has no idea of what a Stream is or what it contains DataSize the size in bytes of the binary data record that follows the Session Record header The Storage Daemon has no idea of the actual contents of the binary data record For standard Unix files the data record typically contains the file attributes or the file data For a sparse file the first 64 bits of the data contains the storage a
37. DEFAULT O VolUseDuration BIGINT UNSIGNED NOT NULL DEFAULT O MaxVolJobs INTEGER UNSIGNED NOT NULL DEFAULT 0O MaxVolFiles INTEGER UNSIGNED NOT NULL DEFAULT O MaxVolBytes BIGINT UNSIGNED NOT NULL DEFAULT O InChanger TINYINT NOT NULL DEFAULT O MediaAddressing TINYINT NOT NULL DEFAULT 0 VolReadTime BIGINT UNSIGNED NOT NULL DEFAULT O VolWriteTime BIGINT UNSIGNED NOT NULL DEFAULT O PRIMARY KEY Mediala INDEX PoolId CREATE TABLE Pool PoolId INTEGER UNSIGNED NOT NULL AUTO_INCREMENT Name TINYBLOB NOT NULL NumVols INTEGER UNSIGNED NOT NULL DEFAULT O MaxVols INTEGER UNSIGNED NOT NULL DEFAULT O UseOnce TINYINT NOT NULL UseCatalog TINYINT NOT NULL AcceptAnyVolume TINYINT DEFAULT O VolRetention BIGINT UNSIGNED NOT NULL VolUseDuration BIGINT UNSIGNED NOT NULL MaxVolJobs INTEGER UNSIGNED NOT NULL DEFAULT O MaxVolFiles INTEGER UNSIGNED NOT NULL DEFAULT O MaxVolBytes BIGINT UNSIGNED NOT NULL AutoPrune TINYINT DEFAULT 0 Recycle TINYINT DEFAULT 0 PoolType ENUM Backup Copy Cloned Archive Migration Scratch NOT NULL LabelFormat TINYBLOB Enabled TINYINT DEFAULT 1 ScratchPoolId INTEGER UNSIGNED DEFAULT O REFERENCES Pool RecyclePoolId INTEGER UNSIGNED DEFAULT O REFERENCES Pool UNIQUE Name 128 PRIMARY KEY PoolId CREATE TABLE Client ClientId INTEGER UNSIGNED NOT NULL AUTO_INCREMENT Name TINYBLOB NOT NULL Uname TINYBLOB NOT NULL full uname a of client AutoPrune TINYI
38. FLA which means you agree to the LICENSE in the main source directory It also means that you receive back the right to use the code that you have submitted Any developer who wants to contribute and is employed by a company should either list the employer as the owner of the code or get explicit permission from him to sign the copyright assignment This is because in many countries all work that an employee does whether on company time or in the employee s free time is considered to be Intellectual Property of the company Obtaining official approval or an FLA from the company will avoid misunderstandings between the employee the company and the Bacula project A good number of companies have already followed this procedure The Fiduciary License Agreement is posted on the Bacula web site at http www bacula org FLA bacula en pdf The instructions for filling out this agreement are also at http www bacula org page fsfe It should be filled out then sent to Free Software Foundation Europe Freedom Task Force Sumatrastrasse 25 8006 Zrich Switzerland Please note that the above address is different from the officially registered office mentioned in the document When you send in such a complete doc ument please notify me kern at sibbald dot com The Development Cycle As I noted in the 1 38 ReleaseNotes version 1 38 was different from prior versions because it had a lot more contributions I expect that this trend wil
39. NT DEFAULT 0 FileRetention BIGINT UNSIGNED NOT NULL JobRetention BIGINT UNSIGNED NOT NULL UNIQUE Name 128 PRIMARY KEY ClientId dE CREATE TABLE BaseFiles Baseld INTEGER UNSIGNED AUTO_INCREMENT BaseJobId INTEGER UNSIGNED NOT NULL REFERENCES Job JobId INTEGER UNSIGNED NOT NULL REFERENCES Job FileId INTEGER UNSIGNED NOT NULL REFERENCES File FileIndex INTEGER UNSIGNED PRIMARY KEY Baseld 62 CREATE TABLE UnsavedFiles UnsavedId INTEGER UNSIGNED AUTO_INCREMENT JobId INTEGER UNSIGNED NOT NULL REFERENCES Job PathId INTEGER UNSIGNED NOT NULL REFERENCES Path FilenameId INTEGER UNSIGNED NOT NULL REFERENCES Filename PRIMARY KEY UnsavedId CREATE TABLE Version VersionId INTEGER UNSIGNED NOT NULL oe Initialize Version INSERT INTO Version VersionId VALUES 7 CREATE TABLE Counters Counter TINYBLOB NOT NULL MinValue INTEGER MaxValue INTEGER CurrentValue INTEGER WrapCounter TINYBLOB NOT NULL PRIMARY KEY Counter 128 63 Storage Media Output Format General This document describes the media format written by the Storage daemon The Storage daemon reads and writes in units of blocks Blocks contain records Each block has a block header followed by records and each record has a record header followed by record data This chapter is intended to be a technical discussion of the Media For mat and as such is not targeted at end users but rather at developers and system administrators that wa
40. OK amp amp tls_remote_need BNET_TLS_OK sendit _ Authorization problem Remote server did not advertise required TLS support n auth_success false goto auth_done Verify that we are willing to meet the remote host s requirements if tls_remote_need gt tls_local_need amp amp tls_local_need BNET_TLS_OK amp amp tls_remote_need BNET_TLS_OK 4 sendit _ Authorization problem Remote server requires TLS n auth_success false goto auth_done 95 Bacula Regression Testing General This document is intended mostly for developers who wish to ensure that their changes to Bacula don t introduce bugs in the base code However you don t need to be a developer to run the regression scripts They are simply shell scripts that drive Bacula through bconsole and then typically compare the input and output with diff You can find the existing regression script in the Bacula SVN on the SourceForge SVN in the project tree named regress I strongly rec ommend that you check it out of the SVN because afterwards you can easily get updates made to them To get started we recommend that you create a directory named bac ula under which you will put the current source code and the current set of regression scripts Below we will describe how to set this up for the SVN trunk but you can also set it up for any branch such as the current Branch 2 0 where the currently released code resides The top
41. Sos SE eh Be a 114 Higher Level Conventions 115 Smart Memory Allocation With Orphaned Buffer Detection 116 Download smartall zip Zipped archive 123 GNU ree Documentation License 2004 1 1 APPLICABILITY AND DEFINITIONS 1 2 VERBATIM COPYING 0 3 3 COPYING IN QUANTITY 3 4 MODIFICATIONS o o 4 5 COMBINING DOCUMENTS 6 6 COLLECTIONS OF DOCUMENTS 7 7 AGGREGATION WITH INDEPENDENT WORKS 7 8 TRANSLATION 2 0 0006222 2b eee 7 9 TERMINATION Sry Gt ise Bee geal a aad Seg kee i 10 FUTURE REVISIONS OF THIS LICENSE ADDENDUM How to use this License for your documents List of Figures Smart Memory Allocation with Orphaned Buffer Detection 116 List of Tables Message Error Code Classes 1 2 ee ee 30 Filename Table Layout o e e 51 Path Table Layout e 52 File Table Layout ss o sanca acata a eca e 53 Job Table Layout 2 260 pn i ea OE ae A ee 53 JOD TYPES e hs Beech a oe Bee a eg 55 JODAStabuses ti a a a ni Gla ie At das 55 File Sets Table Layout o e e 55 JobMedia Table Layout o 56 Media Table Layout e 57 Pool Table Layout arsos p iis aek dn aa aa e o 57 Client Table Layout 0 0 020000 eee eee 58 Unsaved Files Table Layout o
42. U Free Documentation License Version 1 2 or any later version published by the Free Software Founda tion with no Invariant Sections no Front Cover Texts and no Back Cover Texts A copy of the license is included in the section entitled GNU Free Documentation License If you have Invariant Sections Front Cover Texts and Back Cover Texts replace the with Texts line with this with the Invariant Sections being LIST THEIR TITLES with the Front Cover Texts being LIST and with the Back Cover Texts being LIST If you have Invariant Sections without Cover Texts or some other combina tion of the three merge those two alternatives to suit the situation If your document contains nontrivial examples of program code we recom mend releasing these examples in parallel under your choice of free software license such as the GNU General Public License to permit their use in free software Index csignature 106 dinput_text 106 Download smartall zip Zipped archive 123 Acknowledgements 107 Adding a New Test 104 Additional Error information 113 all non root tape tests 102 all root tests 102 all tape and file tests 102 all_non root tests 101 Alloc Function 121 ALSO SEE 107 API Changes Bnet 93 API Implimentation TLS 91 Archive Download smartall zip Zipped 198 Download md5 zip Zipped 107 Attributes Unix File 77 Authentication Negotiation 95 Backup Commands Received fr
43. _file uint32_t SS Sg A AAA AS JobErrors uint32_t gt JobStatus uint32_t VerNum 11 gt fields deprecated Id 32 byte Bacula Identifier Bacula 1 0 immortal n LabelType in FileIndex field of Header EOM_LABEL 3 Label at EOM SOS_LABEL 4 Start of Session label EOS_LABEL 5 End of Session label VerNum 11 JobId Jobld write_btime Bacula time date this tape record written write_date Julian date tape this record written deprecated write_time Julian time tape this record written deprecated PoolName Pool Name PoolType Pool Type MediaType Media Type ClientName Name of File daemon or Client writing this session Not used for EOM_LABEL 83 Bacula Porting Notes This document is intended mostly for developers who wish to port Bacula to a system that is not officially supported It is hoped that Bacula clients will eventually run on every imaginable system that needs backing up perhaps even a Palm It is also hoped that the Bacula Directory and Storage daemons will run on every system capable of supporting them Porting Requirements In General the following holds true Bacula has been compiled and run on Linux RedHat FreeBSD and Solaris systems In addition clients exist on Win32 and Irix It requires GNU C to compile You can try with other com pilers but you are on your own The Irix client is built with the Irix complier but in general you will need GNU Your co
44. a identifier Bacula 0 9 mortal n LabelType Saved in the FileIndex of the Header record PRE_LABEL 1 Volume label on unwritten tape VOL_LABEL 2 Volume label after tape written EOM_LABEL 3 Label at EOM not currently implemented SOS_LABEL 4 Start of Session label format given below EOS_LABEL 5 End of Session label format given below 81 label_date Julian day tape labeled label_time Julian time tape labeled write_date Julian date tape first used data written write_time Julian time tape first used data written VolName Physical Volume name PrevVolName The VolName of the previous tape if this tape is a continuation of the previous one PoolName Pool Name PoolType Pool Type MediaType Media Type HostName Name of host that is first writing the tape LabelProg Name of the program that labeled the tape ProgVersion Version of the label program ProgDate Date Label program built Session Label Id 32 bytes RP RSaS rene nS SSS tsa See See SoS a aon nas Sa VerNum uint32_t ers Jobta A e ao Ria einer ine ne ee en re OS Bootie OB bytes AOS Poollype 128 bytes E Jobline s a T8 byee O D ClientName 128 bytes ier io ee OS FileSetName 128 bytes A ee oc oe SS a A BOSS Paes a ui JobFiles uint32_t ee ne aa JobBytes uint32_t SSS SH SS SSSR StS SaaS SS Sn SSS start_block uint32_t a aa eee end_block uint32_t start_file uint32_t end
45. a special flag to make pthreads work See cygwin for an example x LDFLAGS set any special loader flags See cygwin for an example x PTHREAD_LIB set for any special pthreads flags needed during linking See freebsd as an example lld set so that a long long int will be properly edited in a printf call x llu set so that a long long unsigned will be properly edited in a printf call x PFILES set to add any files that you may define is your platform subdirectory These files are used for installation of automatic system startup of Bacula daemons 86 To rebuild a new version of configure from a changed auto conf configure in you enter make configure in the top level Bacula source directory You must have done a configure prior to trying to rebuild the configure script or it will get into an infinite loop If the make configure gets into an infinite loop ctl c it then do configure no options are necessary and retry the make configure which should now work To rebuild configure you will need to have autoconf version 2 57 3 or higher loaded Older versions of autoconf will complain about unknown or bad options and won t work After you have a working configure script you may need to make a few system dependent changes to the way Bacula works Generally these are done in src baconfig h You can find a few examples of system dependent changes toward the end of this file For example on Irix sy
46. ain SVN repos itory and you are not experienced with SVN please ask Kern to do it If you follow the simple steps above it is unlikely that you will do any damage to the repository and if you do it is always possible for us to recover but it can be painful If you are only working in one subdirectory of say the bacula project for example the scripts directory you can do your commit from that subdirectory and only the changes in that directory and all its subdi rectories will be committed This can be helpful for translators If you are doing a French translation you will be working in docs manual fr and if you are always cd ed into that directory when doing your com mits your commit will effect only that directory As long as you are careful only to change files that you want changed you have little to worry about Subversion Resources Subversion svn Resources cvs2svn Statistics Total CVS Files 3286 Total CVS Revisions 28924 Total Unique Tags 63 Total Unique Branches 11 CVS Repos Size in KB 232421 Total SVN Commits 4116 First Revision Date Tue Apr 23 12 42 57 2002 Last Revision Date Tue Feb 6 06 37 57 2007 The new Subversion repository size on Robert s machine 4 0K bacula tst dav 12K bacula tst locks 40K bacula tst hooks 20 16K bacula tst conf 190M bacula tst db revs 17M bacula tst db revprops 4 0K bacula tst db transactions 206M bacula tst db 206M bacula tst Main Subversion Web Page h
47. ame way as the standard library functions the SMARTALLOC versions expand the buffers they allocate to 119 include information that identifies where each buffer was allocated and to chain all allocated buffers together When a buffer is released it is removed from the allocated buffer chain A call on sm_dump is able by scanning the chain of allocated buffers to find all orphaned buffers Buffers allocated while sm_static 1 is in effect are specially flagged so that despite appear ing on the allocated buffer chain sm_dump will not deem them orphans When a buffer is allocated by malloc or expanded with realloc all bytes of newly allocated storage are set to the hexadecimal value 0x55 alter nating one and zero bits Note that for realloc this applies only to the bytes added at the end of buffer the original contents of the buffer are not modified Initializing allocated storage to a distinctive nonzero pattern is intended to catch code that erroneously assumes newly allocated buffers are cleared to zero in fact their contents are random The calloc function defined as returning a buffer cleared to zero continues to zero its buffers under SMARTALLOC Buffers obtained with the SMARTALLOC functions contain a special sen tinel byte at the end of the user data area This byte is set to a special key value based upon the buffer s memory address When the buffer is released the key is tested and if it has been overwritten an
48. amount of overhead to a program except for programs which use realloc extensively see below SMARTALLOC focuses on proper stor age management rather than internal consistency of the heap as checked by the malloc_debug facility available on some systems SMARTALLOC does not conflict with malloc_debug and both may be used together if you wish SMARTALLOC makes no assumptions regarding the internal structure of the heap and thus should be compatible with any C language implementa tion of the standard memory allocation functions Installing SMARTALLOC SMARTALLOC is provided as a Zipped archive smartall zip see the down load instructions below To install SMARTALLOC in your program simply add the statement 116 to every C program file which calls any of the memory allocation functions malloc calloc free etc SMARTALLOC must be used for all memory allocation with a program so include file for your entire program if you have such a thing Next define the symbol SMARTALLOC in the compilation before the inclusion of smartall h I usually do this by having my Makefile add the DSMARTALLOC option to the C compiler for non production builds You can define the symbol manually if you prefer by adding the statement define SMARTALLOC At the point where your program is all done and ready to relinquish control to the operating system add the call sm_dump datadump where datadump specifies whether the contents of orp
49. and copies the buffer except in the trivial case where the size of the buffer is not being changed By forcing the buffer to move on every call and destroying the contents of the old buffer when it is released SMARTALLOC 120 traps programs which keep pointers into a buffer across a call on realloc which may move it This strategy may prove very costly to programs which make extensive use of realloc If this proves to be a problem such programs may wish to use actuallymalloc actuallyrealloc and actuallyfree for such frequently adjusted buffers trading error detec tion for performance Although not specified in the System V Interface Definition many C library implementations of realloc permit an old buffer argument of NULL causing realloc to allocate a new buffer The SMARTALLOC version permits this When SMARTALLOC is Disabled When SMARTALLOC is disabled by compiling a program with the symbol SMARTALLOC not defined calls on the functions otherwise redefined by SMARTALLOC go directly to the system functions In addition compile time definitions translate calls on the actually O functions into the corresponding library calls actuallymalloc 100 for example compiles into malloc 100 The two special SMARTALLOC functions sm_dump and sm_static are defined to generate no code hence the null statement Finally if SMARTALLOC is not defined compilation of the file smartall c generates no code o
50. anent storage medium Any number of storage daemons may be run on a given machine each associated with an individual storage device connected to it and BAC ULA operations may employ storage daemons on any number of hosts connected by a network local or remote The ability to employ re mote storage daemons with appropriate security measures permits automatic off site backup possibly to publicly available backup repos itories SD Development Outline In order to provide a high performance backup and restore solution that scales to very large capacity devices and networks the storage daemon must be able to extract as much performance from the stor age device and network with which it interacts In order to accom plish this storage daemons will eventually have to sacrifice simplicity and painless portability in favor of techniques which improve perfor mance My goal in designing the storage daemon protocol and de veloping the initial prototype storage daemon is to provide for these additions in the future while implementing an initial storage daemon which is very simple and portable to almost any POSIX like environ ment This original storage daemon and its evolved descendants can serve as a portable solution for non demanding backup requirements such as single servers of modest size individual machines or small local networks while serving as the starting point for development of higher performance configurable derivatives whic
51. ate svn diff gt change patch If you plan on doing significant development work over a period of time after having your first patch reviewed and approved you will be eligible for having developer SVN access so that you can commit your changes directly to the SVN repository To do so you will need a userid on Source Forge Copyrights To avoid future problems concerning changing licensing or copyrights all code contributions more than a hand full of lines must be in the Public Domain or have the copyright transferred to the Free Software Foundation Europe e V with a Fiduciary License Agreement FLA as in the current code Note prior to November 2004 the code was copyrighted by Kern Sibbald and John Walker After November 2004 the code was copyrighted by Kern Sibbald then on the 15th of November 2006 the copyright was transferred to the Free Software Foundation Europe e V Your name should be clearly indicated as the author of the code and you must be extremely careful not to violate any copyrights or use other people s code without acknowledging it The purpose of this requirement is to avoid future copyright patent or intellectual property problems Please read the LICENSE agreement in the main source code directory When you sign the Fiduciary License Agreement FLA and send it in you are argeeing to the terms of that LICENSE file To understand the possible source of future problems please examine the difficulties Mozilla
52. atedly call bnet_open Any error message will generally be sent to the JCR 112 bnet_send To send a packet one uses the subroutine int bnet_send BSOCK sock This routine is equivalent to a write except that it handles the low level details The data to be sent is expected to be in sock gt msg and be sock gt msglen bytes To send a packet bnet_send first writes four bytes in network byte order than indicate the size of the following data packet It returns Returns 0 on failure Returns 1 on success In the case of a failure an error message will be sent to the JCR contained within the bsock packet bnet_fsend This form uses int bnet_fsend BSOCK sock char format and it allows you to send a formatted messages somewhat like fprintf The return status is the same as bnet_send Additional Error information Fro additional error information you can call is_bnet_error BSOCK bsock which will return 0 if there is no error or non zero if there is an error on the last transmission The is_bnet_stop BSOCK bsock func tion will return 0 if there no errors and you can continue sending It will return non zero if there are errors or the line is closed no more transmissions should be sent bnet_recv To read a packet one uses the subroutine int bnet recv BSOCK sock This routine is similar to a read except that it handles the low level details bnet_read first reads packet length that follows as four
53. atime Last access Last access Last access time 66 time since time since years epoch epoch 12 st_mtime Last modify Last modify Last access time 66 time since time since years epoch epoch 13 st_ctime Inode change File create File create time 66 time since time since years epoch epoch Old Depreciated Tape Format The format of the Block Header version 1 26 and earlier is uint32_t CheckSum uint32_t BlockSize uint32_t BlockNumber Block check sum Block byte size including the header Block number 78 char ID 4 BBO1 Identification and block level The format of the Record Header version 1 26 or earlier is uint32_t VolSessionId Unique ID for this session uint32_t VolSessionTime Start time date of session int32_t FileIndex File index supplied by File daemon int32_t Stream Stream number supplied by File daemon uint32_t DataSize size of following data record in bytes Current Bacula Tape Format 6 June 2001 Version BBO1 is the old deprecated format A Bacula tape is composed of tape Blocks Each block has a Block header followed by the block data Block Data consists of Records Records consist of Record Headers followed by Record Data Block Header 16 bytes version BBO1 Record Header 20 bytes version BBO1 Record Header 20 bytes version BBO1 Block Header the first item in
54. ation of Records for a Save 51 Job Messages 31 JoblId 64 Label Session 71 Length Filenames and Maximum Filename 49 Libraries Living with 118 Library Initialization and Cleanup 91 License 12 GNU ree Documentation 1 Link 77 Living with Libraries 118 Low Level Network Protocol 35 Management Bacula Memory 108 Manipulating Socket State 94 Memory Dynamically Allocated 108 Pooled and Non pooled 109 Statically Allocated 108 Memory Messages 32 Message Classes 30 Messages Debug 30 Error 30 Job 31 32 Memory 32 Minimal Code in Console Program 88 Blocking MySQL Installing and Configuring 50 MySQL Table Definition 60 Name 105 Negotiating a TLS Connection 94 Negotiation TLS Authentication 95 Notes Bacula Porting 84 Old Depreciated Tape Format 78 Options 106 Other Tests 101 Outline SD Development 44 Overall Format 67 Overall Storage Format 72 Overlays and Underhandedness 122 Parameters Setting the Configuration 97 Platform Requirements 33 Platform Support 33 Pooled and Non pooled Memory 109 Porting Steps to Take for 85 Porting Requirements 84 PostgreSQL Installing and Configuring 50 Program Minimal Code in Console 88 Test and Demonstration 122 Protocol Daemon 35 General Daemon 35 Low Level Network 35 TCP IP Network 112 Protocol Used Between the Direc tor and the File Daemon 37 Protocol Used Between the Direc tor and the
55. bMediaId INTEGER UNSIGNED NOT NULL AUTO_INCREMENT JobId INTEGER UNSIGNED NOT NULL REFERENCES Job MediaId INTEGER UNSIGNED NOT NULL REFERENCES Media FirstIndex INTEGER UNSIGNED NOT NULL DEFAULT O LastIndex INTEGER UNSIGNED NOT NULL DEFAULT 0 StartFile INTEGER UNSIGNED NOT NULL DEFAULT 0 EndFile INTEGER UNSIGNED NOT NULL DEFAULT O StartBlock INTEGER UNSIGNED NOT NULL DEFAULT O EndBlock INTEGER UNSIGNED NOT NULL DEFAULT O VolIndex INTEGER UNSIGNED NOT NULL DEFAULT O PRIMARY KEY JobMediald INDEX JobId Mediald CREATE TABLE Media MediaId INTEGER UNSIGNED NOT NULL AUTO_INCREMENT VolumeName TINYBLOB NOT NULL Slot INTEGER NOT NULL DEFAULT O PoolId INTEGER UNSIGNED NOT NULL REFERENCES Pool MediaType TINYBLOB NOT NULL FirstWritten DATETIME NOT NULL LastWritten DATETIME NOT NULL LabelDate DATETIME NOT NULL VolJobs INTEGER UNSIGNED NOT NULL DEFAULT 0 VolFiles INTEGER UNSIGNED NOT NULL DEFAULT O VolBlocks INTEGER UNSIGNED NOT NULL DEFAULT O VolMounts INTEGER UNSIGNED NOT NULL DEFAULT O VolBytes BIGINT UNSIGNED NOT NULL DEFAULT O VolErrors INTEGER UNSIGNED NOT NULL DEFAULT 0 VolWrites INTEGER UNSIGNED NOT NULL DEFAULT 0 VolCapacityBytes BIGINT UNSIGNED NOT NULL VolStatus ENUM Full Archive Append Recycle Purged Read Only Disabled Error Busy Used Cleaning NOT NULL 61 Recycle TINYINT NOT NULL DEFAULT 0 VolRetention BIGINT UNSIGNED NOT NULL
56. backup data for multiple clients some users may want to maintain multiple databases one for each machine to be backed up This reduces the risk of confusion of accidental restoring a file to the wrong machine as well as reducing the amount of data in a single database thus increasing efficiency and reducing the impact of a lost or damaged database Sequence of Creation of Records for a Save Job Start with StartDate ClientName Filename Path Attributes Medi aName MediaCoordinates PartNumber NumParts In the steps below Create new means to create a new record whether or not it is unique Create unique means each record in the database should be unique Thus one must first search to see if the record exists and only if not should a new one be created otherwise the existing RecordId should be used Create new Job record with StartDate save JobId Create unique Media record save Mediald Create unique Client record save ClientId Create unique Filename record save Filenameld Create unique Path record save PathId Create unique Attribute record save Attributeld store Clientld Filenameld PathId and Attributes 7 Create new File record store Jobld Attributeld MediaCoordi nates etc 8 Repeat steps 4 through 8 for each file 9 Create a JobMedia record save Mediald 10 Update Job record filling in EndDate and other Job statistics Database Tables 51 Filename Column Name Data Type
57. ber beginning from one that increments with each file or directory sent where lt Stream Id gt will be 1 for the Attributes record and 2 for uncompressed File data 3 is reserved for the MD5 signature for the file where lt Info gt transmit information about the Stream to the Storage Daemon It is a character string field where each character has a meaning The only character currently defined is 0 zero which is simply a place holder a no op In the future there may be codes indicating compressed data encrypted data etc Immediately following the header packet the Storage daemon will ex pect any number of data packets The series of data packets is termi nated by a zero length packet which indicates to the Storage daemon that the next packet will be another header packet As previously mentioned a negative length packet is a request for the Storage dae mon to temporarily enter command mode and send a reply to the File daemon Thus an actual conversation might contain the following exchanges FD lt 1 1 0 gt header packet FD lt data packet containing file attributes gt FD Null packet FD lt 1 2 0 gt FD lt multiple data packets containing the file data gt FD Packet length 1 SD 3000 OK FD lt 2 1 0 gt FD lt data packet containing file attributes gt FD Null packet FD lt 2 2 0 gt FD lt multiple data packets containing the file data gt FD Null packet FD Null packet FD append e
58. buffers by adding code to release them but by doing so you re adding unnecessary complexity and code size to your program just to silence the nattering of a SMARTALLOC so an escape hatch is provided to eliminate the need to release these buffers Normally all storage allocated with the functions malloc calloc and realloc is monitored by SMARTALLOC If you make the function call 117 sm_static 1 you declare that subsequent storage allocated by malloc calloc and realloc should not be considered orphaned if found to be allocated when sm_dump is called I use a call on sm_static 1 before I allocate things like program configuration tables so I don t have to add code to release them at end of program time After allocating unmonitored data this way be sure to add a call to sm_static 0 to resume normal monitoring of buffer allocations Buffers allocated while sm_static 1 is in effect are not checked for having been orphaned but all the other safeguards provided by SMARTALLOC remain in effect You may release such buffers if you like but you don t have to Living with Libraries Some library functions for which source code is unavailable may gratuitously allocate and return buffers that contain their results or require you to pass them buffers which they subsequently release If you have source code for the library by far the best approach is to simply install SMARTALLOC in it particularly since th
59. ckout can have surprising results so you should take a careful look at the documen tation Robert has kindly provided the following documentation on the new svn repository and how to use it Here is the list of branches Branch 1 32a Branch 1 32e Branch 1 34 2 Branch 1 34 5 Branch 1 36 Branch 1 36 1 Branch 1 36 2 15 Branch 1 3 Branch 2 0 import vendor The list of tags is 8 Release 1 1 Release 1 19 Release 1 19a Release 1 19b Release 1 20 Release 1 21 Release 1 22 Release 1 23 Release 1 23a Release 1 24 Release 1 25 Release 1 25a Release 1 26 Release 1 27 Release 1 27a Release 1 27b Release 1 27c Release 1 28 Release 1 29 Release 1 30 Release 1 31 Release 1 31a Release 1 32 Release 1 32a Release 1 32b Release 1 32c Release 1 32d Release 1 32e Release 1 32f Release 1 32f 2 Release 1 32f 3 Release 1 32f 4 Release 1 32f 5 Release 1 34 0 Release 1 34 1 Release 1 34 3 Release 1 34 4 Release 1 34 5 Release 1 34 6 Release 1 35 1 Release 1 35 2 Release 1 35 3 Release 1 35 6 Release 1 35 7 Release 1 35 8 Release 1 36 0 Release 1 36 1 Release 1 36 2 Release 1 36 3 Release 1 38 0 Release 1 38 1 Release 1 38 10 Release 1 38 11 Release 1 38 2 Release 1 38 3 Release 1 38 4 Release 1 38 5 Release 1 38 6 Release 1 38 7 Release 1 38 8 Release 1 38 9 Release 1 8 1 Release 1 8 2 Release 1 8 3 Release 1 8 4 Release 1 8 5 Release 1 8 6 Release 2 0 0 Release 2 0 1 Release 2 0 2 Here is a list of commands to get you started The recommende
60. concerning files that were backed up and their locations on the storage media We have investigated the possibility of using the following SQL engines for Bacula Beagle mSQL GNU SQL PostgreSQL SQLite Oracle and MySQL Each presents certain problems with either licensing or maturity At present we have chosen for development purposes to use MySQL PostgreSQL and SQLite MySQL was chosen because it is fast proven to be reliable widely used and actively being developed MySQL is released under the GNU GPL license PostgreSQL was chosen because it is a full featured very mature database and because Dan Langille did the Bacula driver for it PostgreSQL is distributed under the BSD license SQLite was chosen because it is small efficient and can be directly embedded in Bacula thus requiring much less effort from the system administrator or person building Bacula In our testing SQLite has performed very well and for the functions that we use it has never encountered any errors except that it does not appear to handle databases larger than 2GBytes That said we would not recommend it for serious production use The Bacula SQL code has been written in a manner that will allow it to be easily modified to support any of the current SQL database sys tems on the market for example mSQL iODBC unixODBC Solid OpenLink ODBC EasySoft ODBC InterBase Oracle8 Oracle7 and DB2 If you do not specify either with mysql or with postgresql
61. constant string appears many times In modules that contain many calls on allocation functions this results in substantial wasted storage for the strings that identify the file name If your compiler permits optimization of multiple occurrences of constant strings enabling this mode will eliminate the overhead for these strings Of course it s up to you to make sure choosing this compiler mode won t wreak havoc on some other part of your program Test and Demonstration Program A test and demonstration program smtest c is supplied with SMARTAL LOC You can build this program with the Makefile included Please refer to the comments in smtest c and the Makefile for information on this program If you re attempting to use SMARTALLOC on a new machine or with a new compiler or operating system it s a wise first step to check it out with smtest first 122 Invitation to the Hack SMARTALLOC is not intended to be a panacea for storage management problems nor is it universally applicable or effective it s another weapon in the arsenal of the defensive professional programmer attempting to create reliable products It represents the current state of evolution of expedient debug code which has been used in several commercial software products which have collectively sold more than third of a million copies in the retail market and can be expected to continue to develop through time as it is applied to ever more demanding projects The v
62. cument free in the sense of freedom to assure everyone the effective freedom to copy and redistribute it with or without modifying it either commercially or noncommercially Secondarily this License pre serves for the author and publisher a way to get credit for their work while not being considered responsible for modifications made by others This License is a kind of copyleft which means that derivative works of the document must themselves be free in the same sense It complements the GNU General Public License which is a copyleft license designed for free software We have designed this License in order to use it for manuals for free software because free software needs free documentation a free program should come with manuals providing the same freedoms that the software does But this License is not limited to software manuals it can be used for any textual work regardless of subject matter or whether it is published as a printed book We recommend this License principally for works whose purpose is instruction or reference 1 APPLICABILITY AND DEFINITIONS This License applies to any manual or other work in any medium that contains a notice placed by the copyright holder saying it can be distributed under the terms of this License Such a notice grants a world wide royalty free license unlimited in duration to use that work under the conditions stated herein The Document below refers to any such man
63. d book is Version Control with Subversion by Ben Collins Sussmann Brian W Fitzpatrick and Michael Pilato O Reilly The book is Open Source so it is also available on line at http svnbook red bean com Get a list of commands svn help Get a help with a command svn help command Checkout the HEAD revision of all modules from the project into the directory bacula new 16 svn co https bacula svn sourceforge net svnroot bacula trunk bacula new Checkout the HEAD revision of the bacula module into the bacula subdirectory svn checkout https bacula svn sourceforge net svnroot bacula trunk bacula See which files have changed in the working copy svn status See which files are out of date svn status u Add a new file file c svn add file c Create a new directory svn mkdir newdir Delete an obsolete file svn delete file c Rename a file svn move file c newfile c Move a file to a new location svn move file c newdir file c Copy a file retaining the original history in the new file svn copy file c newfile c Update the working copy with the outstanding changes 17 svn update Compare working copy with the repository svn diff file c Commit the changes in the local working copy svn commit Specify which files are ignored in the current directory svn propedit svn ignore Mark a file to be executable svn propset svn executable prog sh Unmark a file as executable svn propdel svn executable prog sh List a file s p
64. d places explicit trust upon the two way Cram MD5 implementation Cram MD5 is subject to known plaintext attacks and is should be considered considerably less secure than PKI certificate based authen tication Appropriate autoconf macros have been added to detect and use OpenSSL Two additional preprocessor defines have been added HAVE_TLS and HAVE_OPENSSL All changes not specific to OpenSSL rely on HAVE_TLS OpenSSL specific code is constrained to src lib tls c to facilitate the support of alternative TLS implemen tations 0 1 2 New Configuration Directives Additional configuration directives have been added to both the Con sole and Director resources These new directives are defined as fol lows TLS Enable yes no Enable TLS support TLS Require yes no Require TLS connections TLS Certificate path Path to PEM encoded TLS certificate Used as either a client or server certificate 90 TLS Key path Path to PEM encoded TLS private key Must correspond with the TLS certificate TLS Verify Peer yes no Verify peer certificate Instructs server to request and verify the client s x509 certificate Any client certificate signed by a known CA will be accepted unless the TLS Allowed CN configuration directive is used Not valid in a client context TLS Allowed CN string list Common name attribute of allowed peer certificates If directive is specified all client certificates will be verifi
65. d FileIndex field of the Record Header and does not appear in the data part of the record Session Label The Session Label is written at the beginning and end of each session as well as the last record on the physical medium It has the following binary format 71 char Id 32 Bacula Immortal uint32_t VerNum Label version number uint32_t Jobld Job id uint32_t VolumeIndex sequence no of vol Prior to VerNum 11 float64_t write_date Date this label written VerNum 11 and greater btime_t write_btime time date record written The following is zero VerNum 11 and greater float64_t write_time Time this label written char PoolName 128 Pool name char PoolType 128 Pool type char JobName 128 base Job name char ClientName 128 Added in VerNum 10 char Job 128 Unique Job name char FileSetName 128 FileSet name uint32_t JobType uint32_t JobLevel In addition the EOS label contains The remainder are part of EOS label only uint32_t JobFiles uint64_t JobBytes uint32_t start_block uint32_t end_block uint32_t start_file uint32_t end_file uint32_t JobErrors In addition for VerNum greater than 10 the EOS label contains in addition to the above uint32_t JobStatus Job termination code Note the LabelType Volume Label Volume PreLabel Session Start Label is stored in th
66. d current version of the Block Header BB02 is uint32_t CheckSum Block check sum uint32_t BlockSize Block byte size including the header uint32_t BlockNumber Block number char ID 4 BBO2 Identification and block level uint32_t VolSessionld Applies to all records uint32_t VolSessionTime contained in this block As with the previous version the BB02 Block header is a fixed length and fixed format and is followed by Record Headers and Record Data The CheckSum field is a 32 bit CRC checksum of the block data and the block header but not including the CheckSum field The Block Header is always immediately followed by a Record Header If the tape is damaged a Bacula utility will be able to recover as much information as possible from the tape by recovering blocks which are valid The Block header is written using the Bacula serialization routines and thus is guaranteed to be in machine independent format Version 2 Record Header Version 2 Record Header is written to the medium when using Ver sion BB02 Block Headers The memory representation of the record is identical to the old BBO1 Record Header but on the storage medium 70 the first two fields namely VolSessionId and VolSessionTime are not written The Block Header is filled with these values when the First user record is written i e non label record so that when the block is written it will have the current and unique VolSessionId a
67. data transfers between various hosts Using the routines described below bnet_open bnet_write bnet_recv and bnet_close guarantees that the number of bytes you write into the socket will be received as a single record on the other end regardless of how many low level write and read calls are needed All data transferred are considered to be binary data bnet and Threads These bnet routines work fine in a threaded environment However they assume that there is only one reader or writer on the socket at any time It is highly recommended that only a single thread access any BSOCK packet The exception to this rule is when the socket is first opened and it is waiting for a job to start The wait in the Storage daemon is done in one thread and then passed to another thread for subsequent handling If you envision having two threads using the same BSOCK think twice then you must implement some locking mechanism However it probably would not be appropriate to put locks inside the bnet subroutines for efficiency reasons bnet_open To establish a connection to a server use the subroutine BSOCK bnet_open void jcr char host char service int port int fatal bnet_open if successful returns the Bacula sock descriptor pointer to be used in subsequent bnet_send and bnet_read requests If not successful bnet_open returns a NULL If fatal is set on return it means that a fatal error occurred and that you should not repe
68. ddress for the data block Volume Label Id 32 bytes SSSA SS Se VerNum uint32_t ES E aS aS label_date float64_t label_btime btime_t VerNum 11 RE a label_time float64_t write_btime btime_t VerNum 11 PS write_date float64_t 0 float64_t VerNum 11 write_time float64_t 0 float64_t VerNum 11 ae VolNane 128 bytes y ES PrewWolNane 128 bytes o y E PoolNane 128 bytes SS PoolType 128 bytes er MediaType 128 bytes y aes meae A y O A O AE PeopVersion 82 bytes y ae A _ GP byees e T Id 32 byte Bacula identifier Bacula 1 0 immortal n old version also recognized Id 32 byte Bacula identifier Bacula 0 9 mortal n LabelType Saved in the FileIndex of the Header record PRE_LABEL 1 Volume label on unwritten tape VOL_LABEL 2 Volume label after tape written EOM_LABEL 3 Label at EOM not currently implemented SOS_LABEL 4 Start of Session label format given below EOS_LABEL 5 End of Session label format given below VerNum 11 label_date Julian day tape labeled label_time Julian time tape labeled write_date Julian date tape first used data written write_time Julian time tape first used data written VolName Physical Volume name PrevVolName The VolName of the previous tape if this tape is a continuation of the previous one PoolName Pool Name PoolType Pool Type MediaType Media Type HostName Name of host that is first wr
69. e the Filelndex the Stream and the size of the data record which follows The Record Header is always immediately followed by a Data Record if the size given in the Header is greater than zero Note for Block headers of level BB02 version 1 27 and later the Record header as written to tape does not contain the Volume Session Id and the Volume Session Time as these two fields are stored in the BB02 Block header The in memory record header does have those fields for convenience Data Record A data record consists of a binary stream of bytes and is always preceded by a Record Header The details of the mean ing of the binary stream of bytes are unknown to the Storage daemon but the Client programs File daemon defines and thus knows the details of each record type Volume Label A label placed by the Storage daemon at the begin ning of each storage volume It contains general information about the volume It is written in Record format The Stor age daemon manages Volume Labels and if the client wants he may also read them Begin Session Label The Begin Session Label is a special record placed by the Storage daemon on the storage medium as the first record of an append session job with a File daemon This record is useful for finding the beginning of a particular session Job since no records with the same VolSessionId and VolSessionTime will precede this record This record is not normally visible outside of the Storage daemon The
70. e VolSessionId and VolSessionTime 54 form a unique identification of the Job This will be the case even if multiple Directors are using the same Storage daemon The Job Type or simply Type can have one of the following values Value Meaning Backup Job V Verify Job R Restore Job C Console program not in database D Admin Job A Archive Job not implemented The JobStatus field specifies how the job terminated and can be one of the following 5 E O Meaning Created but not yet running Running Blocked Terminated normally Terminated in Error Non fatal error Fatal error Verify Differences Canceled by the user Waiting on the File daemon Waiting on the Storage daemon Waiting for a new Volume to be mounted Waiting for a Mount Waiting for Storage resource Waiting for Job resource Waiting for Client resource Wating for Maximum jobs Waiting for Start Time Waiting for higher priority job to finish ojelele 2 2g 0 plo lo mauro FileSet Column Name Data Type Remark FileSet tinyblob FileSet name FileSetId integer Primary Key MD5 tinyblob MD5 checksum of FileSet 55 CreateTime datetime Time and date Fileset created The FileSet table contains one entry for each FileSet that is used The MD5 signature is kept to ensure that if the
71. e is 88 Directives TLS Configuration 90 Director Services Daemon 41 Directory Structure 104 Disabled 11 When SMARTALLOC is 121 Download md5 zip Zipped archive 107 Dynamically Allocated Memory 108 End Session Label 66 Error Messages 30 Exit Status 107 Extended Attributes 77 Fails If a Test 102 File Services Daemon 42 File Attributes 77 FileIndex 65 69 77 Filename 77 Filenames and Maximum File name Length 49 Files 106 Format Old Depreciated Tape 78 Overall 67 Overall Storage 72 Storage Daemon File Output 67 Storage Media Output 64 Volume Label 71 Function alloc 121 General 64 96 108 112 General 33 35 49 88 General Daemon Protocol 35 GNU ree Documentation License 1 GUI Interface is Difficult 88 Hack Invitation to the 123 Hand Running the Tests by 103 Header Block 68 Record 68 Version 2 Record 70 Version BB02 Block 70 Higher Level Conventions 115 If a Test Fails 102 Implementing a Bacula GUI Inter face 88 Information Additional Error 113 Command and Control 38 Data 38 Initialization and Cleanup Library 91 Installing and MySQL 50 Installing and Configuring Post greSQL 50 Installing and Configuring SQLite 90 Installing SMARTALLOC 116 Interface Implementing a Bacula GUI 88 Internal Bacula Catalog 50 Introduction SD Design 44 TLS 90 Invitation to the Hack 123 Configuring Job Sequence of Cre
72. e programs that run on the system to be backed up and provide the interface between the Host File system and Bacula in particular the Director and the Storage services When time comes for a backup the Director gets in touch with the File daemon on the client machine and hands it a set of marching orders which if written in English might be something like the following OK File daemon it s time for your daily incremental backup I want you to get in touch with the Storage daemon on host archive mysite com and perform the following save operations with the designated options You ll note that I ve attached include and exclude lists and patterns you should apply when backing up the file system As this is an incremental backup you should save only files modified since the time you started your last backup which as you may recall was 2000 11 19 06 43 38 Please let me know when you re done and how it went Thank you So having been handed everything it needs to decide what to dump and where to store it the File daemon doesn t need to have any further contact with the Director until the backup is complete providing there are no errors If there are errors the error messages will be delivered immediately to the Director While the backup is proceeding the File daemon will send the file coordinates and data for each file being backed up to the Storage daemon which will in turn pass the file coordinates to the Director to
73. e record FileIndex field and does not appear in the data part of the record Also the Stream field of the Record Header contains the Jobld This permits quick filtering without actu ally reading all the session data in many cases Overall Storage Format Current Bacula Tape Format 6 June 2001 Version BBO2 added 28 September 2002 Version BBO1 is the old deprecated format A Bacula tape is composed of tape Blocks Each block 72 has a Block header followed by the block data Block Data consists of Records Records consist of Record Headers followed by Record Data Block Header the first item in each block The format is shown below Partial Data block occurs if the data from a previous block spills over to this block the normal case except for the first block on a tape However this partial data block is always preceded by a record header Record Header identifies the Volume Session the Stream and the following Record Data size See below for format Record data arbitrary binary data Block Header Format BBO2 CheckSum uint32_t pa BlockSize wint32t e Blockiamber Gina A eee a A VolSessionTd uint32 t BBO2 Serves to identify the block as a Bacula block and also servers as a block format identifier should we ever need to change the format BlockSize is the size in bytes of the block When reading back a block if the BlockSize does not agree with the actual size read Bacula discards the block
74. e replaced by the Bacula safe equivalents char bstrncpy char dest char source int dest_size char bstrncat char dest char source int dest_size int bsnprintf char buf int32_t buf_len const char fmt int bvsnprintf char str int32_t size const char format va_list ap See src lib bsys c for more details on these routines Don t use the lld or the q printf format editing types to edit 64 bit integers they are not portable Instead use s with edit_uint64 For example char buf 100 uint64_t num something char ed1 50 bsnprintf buf sizeof buf Num 4 s n edit_uint64 num ed1 The edit buffer ed1 must be at least 27 bytes long to avoid overflow See src lib edit c for more details If you look at the code don t start screaming that I use lld I actually use subtle trick taught to me by John Walker The lld that appears in the editing routine is actually define to a what is needed on your OS usually lid or q and is defined in autoconf configure in for each OS C string concatenation causes the appropriate string to be concatenated to the Also please don t use the STL or Templates or any complicated C code 29 Message Classes Currently there are five classes of messages Debug Error Job Mem ory and Queued Debug Messages Debug messages are designed to be turned on at a specified debug level and are always sent to STDOUT There are designed t
75. each block The format is shown below Partial Data block occurs if the data from a previous block spills over to this block the normal case except for the first block on a tape However this partial data block is always preceded by a record header Record Header identifies the Volume Session the Stream and the following Record Data size See below for format Record data arbitrary binary data Block Header Format BBO1 deprecated CheckSum uint32_t ae So ee ee ae See ee BlockSize uint32_t Ha EE E e BlockNumber uint32_t 79 BBO1 Serves to identify the block as a Bacula block and also servers as a block format identifier should we ever need to change the format BlockSize is the size in bytes of the block When reading back a block if the BlockSize does not agree with the actual size read Bacula discards the block CheckSum a checksum for the Block BlockNumber is the sequential block number on the tape VolSessionId a unique sequential number that is assigned by the Storage Daemon to a particular Job This number is sequential since the start of execution of the daemon VolSessionTime the time date that the current execution of the Storage Daemon started It assures that the combination of VolSessionId and VolSessionTime is unique for all jobs written to the tape even if there was a machine crash between two writes Record Header Format BBO1 deprecated VolSessionId a unique sequ
76. eader followed by records Each record consists of a record header followed by the record data The first record on a tape will always be the Volume Label Record No Record Header will be split across Bacula blocks However Record Data may be split across any number of Bacula blocks Obviously this will not be the case for the Volume Label which will always be smaller than the Bacula Block size To simplify reading tapes the Start of Session SOS and End of Ses sion EOS records are never split across blocks If this is about to happen Bacula will write a short block before writing the session record actually the SOS record should always be the first record in a block excepting perhaps the Volume label Due to hardware limitations the last block written to the tape may not be fully written If your drive permits backspace record Bacula will backup over the last record written on the tape re read it and verify that it was correctly written When a new tape is mounted Bacula will write the full contents of the partially written block to the new tape ensuring that there is no loss of data When reading a tape Bacula will discard any block that is not totally written thus ensuring that there is no duplication of data In addition since Bacula blocks are sequentially numbered within a Job it is easy to ensure that no block is missing or duplicated Serialization All Block Headers Record Headers and Label Records are written us
77. eases of Bacula we will generally solicit Feature Request input for the next version and by way of this email we suggest that you send discuss and send in your Feature Requests for the next release Please verify that the Feature Request is not in the current list attached to this email Once users have had several weeks to submit Feature Requests the keeper of the projects list will organize them and request users to vote on them This will allow fixing prioritizing the Feature Requests Having a priority is one thing but getting it implement is another thing we are hoping that the Bacula community will take more responsibility for assuring the implementation of accepted Feature Requests Feature Request format Empty Feature Request form Item n One line summary Date Date submitted Origin Name and email of originator Status What More detailed explanation Why Why it is important Notes Additional notes or features omit if not used End Feature Request form Example Completed Feature Request form Item 1 Implement a Migration job type that will move the job data from one device to another Origin Sponsored by Riege Sofware International GmbH Contact Daniel Holtkamp lt holtkamp at riege dot com gt Date 28 October 2005 Status Partially coded in 1 37 much more to do Assigned to Kern What The ability to copy
78. eceived from the Director for a Restore 43 Storage Daemon Design e e 44 SD Design Introduction 0084 44 SD Development Outline o 44 SD Connections and Sessions e 45 SD Data Structures dut mie ga es Ee 47 Catalog Services oa iee a e e d a e e o 49 General g s crete thie A a Pa 49 Sequence of Creation of Records for a Save Job 51 Database Tables o 51 Storage Media Output Format 64 General A A A lt a a ee 64 Definitions Ann r e o Bae dio 2 es ak 64 Storage Daemon File Output Format 67 Overall Formate fo De ype ye aa ye a Sa Doe 67 Serialization es cross eee Le we eS ew ee es 67 Block Hedder is ci Rae A eo ee 68 Record Header e 68 Version BB02 Block Header 70 Version 2 Record Header o 70 Volume Label Format o 71 Session Label 0 0021 e ee ee a eee 71 Overall Storage Format a 00 72 Unix File Attributes 77 Old Depreciated Tape Format 78 Bacula Porting Notes 2 2 0 eee ee ee 84 Porting Requirements 0 0505 0a e 84 Steps to Take for Porting 85 Implementing a Bacula GUI Interface 88 General it A Re ee te ae a 88 Eby SIG 40 4 ea E Sy ne ASP Aes aan 90 TES a Biene
79. ed against this list This directive may be specified more than once Not valid in a client context TLS CA Certificate File path Path to PEM encoded TLS CA certificate s Multiple certificates are permitted in the file One of TLS CA Certificate File or TLS CA Certificate Dir are re quired in a server context if TLS Verify Peer is also specified and are always required in a client context TLS CA Certificate Dir path Path to TLS CA certificate direc tory In the current implementation certificates must be stored PEM encoded with OpenSSL compatible hashes One of TLS CA Certificate File or TLS CA Certificate Dir are required in a server context if TLS Verify Peer is also specified and are always required in a client context TLS DH File path Path to PEM encoded Diffie Hellman pa rameter file If this directive is specified DH ephemeral keying will be enabled allowing for forward secrecy of communications This directive is only valid within a server context To generate the parameter file you may use openssl openssl dhparam out dhi024 pem 5 1024 0 1 3 TLS API Implementation To facilitate the use of additional TLS libraries all OpenSSL specific code has been implemented within src lib tls c In turn a generic TLS API is exported Library Initialization and Cleanup int init_tls void Performs TLS library initialization including seeding of the PRNG PRNG seeding has not yet been impleme
80. edited only by proprietary word processors SGML or XML for which the DTD and or processing tools are not generally available and the machine generated HTML PostScript or PDF produced by some word processors for output purposes only The Title Page means for a printed book the title page itself plus such following pages as are needed to hold legibly the material this License requires to appear in the title page For works in formats which do not have any title page as such Title Page means the text near the most prominent appearance of the work s title preceding the beginning of the body of the text A section Entitled XYZ means a named subunit of the Document whose title either is precisely XYZ or contains XYZ in parentheses following text that translates XYZ in another language Here XYZ stands for a specific section name mentioned below such as Acknowledgements Dedi cations Endorsements or History To Preserve the Title of such a section when you modify the Document means that it remains a section Entitled XYZ according to this definition The Document may include Warranty Disclaimers next to the notice which states that this License applies to the Document These Warranty Dis claimers are considered to be included by reference in this License but only as regards disclaiming warranties any other implication that these War ranty Disclaimers may have is void a
81. emory Allocation With Orphaned Buffer Detection Few things are as embarrassing as a program that leaks yet few errors are so easy to commit or as difficult to track down in a large complicated program as failure to release allocated memory SMARTALLOC replaces the standard C library memory allocation functions with versions which keep track of buffer allocations and releases and report all orphaned buffers at the end of program execution By including this package in your program during development and testing you can identify code that loses buffers right when it s added and most easily fixed rather than as part of a crisis debugging push when the problem is identified much later in the testing cycle or even worse when the code is in the hands of a customer When program testing is complete simply recompiling with different flags removes SMARTALLOC from your program permitting it to run without speed or storage penalties In addition to detecting orphaned buffers SMARTALLOC also helps to find other common problems in management of dynamic storage including stor ing before the start or beyond the end of an allocated buffer referencing data through a pointer to a previously released buffer attempting to re lease a buffer twice or releasing storage not obtained from the allocator and assuming the initial contents of storage allocated by functions that do not guarantee a known value SMARTALLOC s checking does not usually add a large
82. en restore it and finally compares the restored data with the original data 101 all root tests All non tape tests requiring root permission These are a relatively small number of tests that require running as root The amount of data backed up can be quite large For example one test backs up usr another backs up etc One or more of these tests reports an error Il fix it one day all non root tape tests All tape test not requiring root There are currently three tests all run without being root and backup to a tape The first two tests use one volume and the third test requires an autochanger and uses two volumes If you don t have an autochanger then this script will probably produce an error all tape and file tests All tape and file tests not requiring root This includes just about everything and I don t run it very often If a Test Fails If you one or more tests fail the line output will be similar to If you want to determine why the test failed you will need to modify the script so that it prints Do so by finding the file in regress tests that corresponds to the name printed For example the script for the above error message is in regress tests concurrent jobs test At the very top of each script you will find lines that look like TestName backup bacula test JobName backup scripts functions set_debug 0 scripts cleanup scripts copy confs change_jobname Clienti JobName start_test I
83. ential number that is assigned by the Storage Daemon to a particular Job This number is sequential since the start of execution of the daemon VolSessionTime the time date that the current execution of the Storage Daemon started It assures that the combination of VolSessionId and VolSessionTime is unique for all jobs written to the tape even if there was a machine crash between two writes FileIndex a sequential file number within a job The Storage daemon enforces this index to be greater than zero and sequential Note however that the File daemon may send multiple Streams for the same Filelndex The Storage Daemon uses negative FileIndices to identify Session Start and End labels as well as the End of Volume labels Stream defined by the File daemon and is intended to be used to identify separate parts of the data saved for each file attributes file data 80 The Storage Daemon has no idea of what a Stream is or what it contains DataSize the size in bytes of the binary data record that follows the Session Record header The Storage Daemon has no idea of the actual contents of the binary data record For standard Unix files the data record typically contains the file attributes or the file data For a sparse file the first 64 bits of the data contains the storage address for the data block Volume Label Id 32 byte Bacula identifier Bacula 1 0 immortal n old version also recognized Id 32 byte Bacul
84. ersion of SMARTALLOC here has been tested on a Sun SPARCSta tion Silicon Graphics Indigo2 and on MS DOS using both Borland and Microsoft C Moving from compiler to compiler requires the usual small changes to resolve disputes about prototyping of functions whether the type returned by buffer allocation is char or void and so forth but following those changes it works in a variety of environments I hope you ll find SMARTALLOC as useful for your projects as I ve found it in mine Download smartall zip Zipped archive SMARTALLOC is provided as smartall zip a Zipped archive containing source code documentation and a Makefile to build the software under Unix Copying SMARTALLOC is in the public domain Permission to use copy modify and distribute this software and its documen tation for any purpose and without fee is hereby granted without any conditions or restrictions This software is pro vided as is without express or implied warranty by John Walker October 30th 1998 123 english babel T1 fontenc hyperref GNU Free Documentation License Version 1 2 November 2002 Copyright 2000 2001 2002 Free Software Foundation Inc 51 Franklin St Fifth Floor Boston MA 02110 1301 USA Everyone is permitted to copy and distribute verbatim copies of this license document but changing it is not allowed Preamble The purpose of this License is to make a manual textbook or other func tional and useful do
85. essed file data sparse file data The Storage Daemon has no idea of what a Stream is or what it contains except that the Stream is required to be a positive integer Negative Stream numbers are used internally by the Storage daemon to indicate that the record is a continuation of the previous record the previous record would not entirely fit in the block For Start Session and End Session Labels where the Filelndex is negative the Storage daemon uses the Stream field to contain the JobId The current stream definitions are define STREAM_UNIX_ATTRIBUTES 1 Generic Unix attributes define STREAM_FILE_DATA 2 Standard uncompressed data define STREAM_MD5_SIGNATURE 3 MD5 Signature for the file define STREAM_GZIP_DATA 4 GZip compressed file data Extended Unix attributes with Win32 Extended data Deprecated define STREAM_UNIX_ATTRIBUTES_EX 5 Extended Unix attr for Win32 EX define STREAM_SPARSE_DATA 6 Sparse data stream define STREAM_SPARSE_GZIP_DATA T define STREAM_PROGRAM_NAMES 8 program names for program data define STREAM_PROGRAM_DATA 9 Data needing program define STREAM_SHA1_SIGNATURE 10 SHA1 signature for the file define STREAM_WIN32_DATA 11 Win32 BackupRead data define STREAM_WIN32_GZIP_DATA 12 Gzipped Win32 BackupRead data define STREAM_MACOS_FORK_DATA 13 Mac resource fork define STREAM_HFSPLUS_ATTRIBUTES 14 Mac OS extra attributes
86. etails will be filled in along with a comprehensive list of status codes along with which requests can produce them in an update to this document SD Append Requests append open session lt JobId gt lt Password gt A data ap pend session is opened with the Job ID given by Job d with client password if required given by Password If the session is suc cessfully opened a status of 3000 OK is returned with a ticket number reply used to identify subsequent messages in the ses sion If too many sessions are open or a conflicting session for example a read in progress when simultaneous read and append sessions are not permitted a status of 3502 Volume busy is returned If no volume is mounted or the volume mounted can not be appended to a status of 3503 Volume not mounted is returned append data lt ticket number gt If the append data is ac cepted a status of 3000 OK data address lt IPaddress gt 45 port lt port gt is returned where the IPaddress and port specify the IP address and port number of the data channel Error status codes are 3504 Invalid ticket number and 3505 Session aborted the latter of which indicates the entire append session has failed due to a daemon or media error Once the File daemon has established the connection to the data channel opened by the Storage daemon it will transfer a header packet followed by any number of data packets The header packet is of the form
87. g M_ABORT Causes the daemon to immediately abort This should be used only in extreme cases It attempts to produce a traceback 30 M ERROR TERM Causes the daemon to immediately terminate This should be used only in extreme cases It does not produce a traceback M_FATAL Causes the daemon to terminate the current job but the daemon keeps running M_ERROR Reports the error The daemon and the job continue running M_WARNING Reports an warning message The daemon and the job continue running M_INFO Reports an informational message There are other error message classes but they are in a state of being redesigned or deprecated so please do not use them Some actual examples are Emsgl M_ABORT 0 Cannot create message thread s n str error status Emsg3 M_WARNING 0 Connect to File daemon s at s d failed Retrying An client gt hdr name client gt address client gt port Emsg3 M_FATAL 0 bdird lt filed bad response from Filed to s command d s n cmd n strerror errno Job Messages Job messages are messages that pertain to a particular job such as a file that could not be saved or the number of files and bytes that were saved They Are coded as Jmsg jcr M _FATAL 0 Text of message A Jmsg with M_ FATAL will fail the job The Jmsg takes varargs so can have any number of arguments for substituted in a printf like format Output from the Jmsg will go to the Job
88. h of the data that follow in the subsequent write If the four byte integer is zero or negative it in dicates a special request a sort of network signaling capability In this case no data packet will follow The low level BSOCK routines expect that only a single thread is accessing the socket at a time It is advised that multiple threads do not read write the same socket If you must do this you must provide some sort of locking mechanism It would not be appropriate for efficiency reasons to make every call to the BSOCK routines lock and unlock the packet General Daemon Protocol In general all the daemons follow the following global rules There may be exceptions depending on the specific case Normally one dae mon will be sending commands to another daemon specifically the Director to the Storage daemon and the Director to the File daemon Commands are always ASCII commands that are upper lower case dependent as well as space sensitive All binary data is converted into ASCII either with printf state ments or using base64 encoding 35 All responses to commands sent are always prefixed with a re turn numeric code where codes in the 1000 s are reserved for the Director the 2000 s are reserved for the File daemon and the 3000 s are reserved for the Storage daemon Any response that is not prefixed with a numeric code is a com mand or subcommand if you like coming from the other end For example while
89. h use techniques such as POSIX threads shared memory asynchronous I O buffering 44 to high speed intermediate media and support for tape changers and jukeboxes SD Connections and Sessions A client connects to a storage server by initiating a conventional TCP connection The storage server accepts the connection unless its max imum number of connections has been reached or the specified host is not granted access to the storage server Once a connection has been opened the client may make any number of Query requests and or initiate if permitted one or more Append sessions which transmit data to be stored by the storage daemon and or Read sessions which retrieve data from the storage daemon Most requests and replies sent across the connection are simple ASCII strings with status replies prefixed by a four digit status code for easier parsing Binary data appear in blocks stored and retrieved from the storage Any request may result in a single line status reply of 3201 Notification pending which indicates the client must send a Query notification request to retrieve one or more notifications posted to it Once the notifications have been returned the client may then resubmit the request which resulted in the 3201 status The following descriptions omit common error codes yet to be defined which can occur from most or many requests due to events like media errors restarting of the storage daemon etc These d
90. han new size no buffer size change will occur However if a buffer size change is needed the original contents of the buffer will be preserved but the buffer address may change Many of the low level Bacula subroutines expect to be passed a pool memory buffer and use this call to ensure the buffer they use is sufficiently large 110 Releasing All Pooled Memory In order to avoid orphaned buffer error messages when terminating the program use void close_memory_pool to free all unused memory retained in the Bacula memory pool Note any memory not returned to the pool via free_pool memory will not be released by this call Pooled Memory Statistics For debugging purposes and performance tuning the following call will print the current memory pool statistics void print_memory_pool_stats an example output is Pool Maxsize Maxused Inuse 0 256 0 0 1 256 1 0 2 256 1 0 111 TCP IP Network Protocol General This document describes the TCP IP protocol used by Bacula to com municate between the various daemons and services The definitive defi nition of the protocol can be found in src lib bsock h src lib bnet c and src lib bnet_server c Bacula s network protocol is basically a packet oriented protocol built on a standard TCP IP streams At the lowest level all packet transfers are done with read and write requests on system sockets Pipes are not used as they are considered unreliable for large serial
91. haned buffers are to be dumped in addition printing to their size and place of allocation The data are dumped only if datadump is nonzero so most programs will normally use sm_dump 0 If a mysterious orphaned buffer appears that can t be identified from the information this prints about it replace the statement with sm_dump 1 Usually the dump of the buffer s data will furnish the additional clues you need to excavate and extirpate the elusive error that left the buffer allocated Finally add the files smartall h and smartall c from this release to your source directory make dependencies and linker input You needn t make inclusion of smartall c in your link optional if compiled with SMARTALLOC not defined it generates no code so you may always include it knowing it will waste no storage in production builds Now when you run your program if it leaves any buffers around when it s done each will be reported by sm_dump on stderr as follows Orphaned buffer 120 bytes allocated at line 50 of gutshot c Squelching a SMARTALLOC Usually when you first install SMARTALLOC in an existing program you ll find it nattering about lots of orphaned buffers Some of these turn out to be legitimate errors but some are storage allocated during program ini tialisation that while dynamically allocated is logically static storage not intended to be released Of course you can get rid of the complaints about these
92. hecks whether the commonName attribute matches any strings supplied via the verify_list parameter Returns true if there is a match false otherwise Manipulating TLS Connections TLS_CONNECTION new_tls_connection TLS_CONTEXT ctx int fd Allocates and initializes a new TLS_CONNECTION structure with context ctx and file descriptor fd void free_tls_connection TLS_CONNECTION tls 92 Deallocates memory associated with the tls structure bool tls_bsock_connect BSOCK bsock Negotiates a a TLS client connection via bsock Returns true if suc cessful false otherwise Will fail if there is a TLS protocol error or an invalid certificate is presented bool tls_bsock_accept BSOCK bsock Accepts a TLS client connection via bsock Returns true if successful false otherwise Will fail if there is a TLS protocol error or an invalid certificate is presented bool tls_bsock_shutdown BSOCK bsock Issues a blocking TLS shutdown request to the peer via bsock This function may not wait for the peer s reply int tls_bsock_writen BSOCK bsock char ptr int32_t nbytes Writes nbytes from ptr via the TLS_CONNECTION associated with bsock Due to OpenSSL s handling of EINTR bsock is set non blocking at the start of the function and restored to its original blocking state before the function returns Less than nbytes may be written if an error occurs The actual number of bytes written will be returned int tls_bsock_
93. hem into Blob fields with the filename indexed on the first 30 characters and the path name indexed on the 255 max characters takes 5 mins 18 seconds and creates a 5 24 MB database Rerunning the job with the database already created takes about 2 mins 50 seconds Running the same as the last one Path and Filename Blob but Filename indexed on the first 30 characters and the Path on the first 50 52 characters linear search done there after takes 5 mins on the average and creates a 3 4 MB database Rerunning with the data already in the DB takes 3 mins 35 seconds Finally saving only the full path name rather than splitting the path and the file and indexing it on the first 50 characters takes 6 mins 43 seconds and creates a 7 35 MB database File Column Name Data Type Remark Fileld integer Primary Key FileIndex integer The sequential file number in the Job JoblId integer Link to Job Record PathId integer Link to Path Record Filenameld integer Link to Filename Record MarklId integer Used to mark files during Verify Jobs LStat tinyblob File attributes in base64 encoding MD5 tinyblob MD5 signature in base64 encoding The File table shown above contains one entry for each file backed up by Bacula Thus a file that is backed up multiple times as is normal will have multiple entries in the File table This will probably be the table with the most number of records Consequently i
94. ile and end block are the file and block with the last data from the session on the volume and volume session id is the volume session ID for blocks from the session stored on that volume SD Read Requests Read open session lt JobId gt lt Volume id gt lt start file gt lt start block gt lt end file gt where Volume id is the volume label start file and start block are 46 the file and block containing the first data from that session on the volume end file and end block are the file and block with the last data from the session on the volume and volume session id is the volume session ID for blocks from the session stored on that volume If the session is successfully opened a status of 3100 OK Ticket number is returned with a reply used to identify subsequent messages in the session If too many sessions are open or a conflicting session for example an append in progress when simultaneous read and append sessions are not permitted a status of 3502 Volume busy is returned If no volume is mounted or the vol ume mounted cannot be appended to a status of 3503 Volume not mounted is returned If no block with the given volume session ID and the correct client ID number appears in the given first file and block for the volume a status of 3505 Session not found is returned Read data lt Ticket gt gt lt Block gt The specified Block of data from open read session with the specified Ticket n
95. ile daemon Win32 files to make bacula fd be a service Unix file find library for File daemon GNOME version of console program General Bacula library Storage daemon Tcl tk console program not yet working test programs normally only in Kern s tree Various tool programs Native Win32 File daemon Visual Studio project file compatibility interface library links to src filed links to src findlib links to src lib beginning of native console program wxWidget console Win32 specific parts wxWidgets console main source program Regression scripts temporary directory to hold Bacula installed binaries temporary directory to hold Bacula source scripts and conf files test scripts temporary directory for temp files temporary working directory for Bacula daemons documentation directory Developer s guide Bacula s home page source html document directory French translation German translation Technical development notes Bacula rescue CDROM Linux rescue CDROM Linux rescue CDROM code Solaris rescue incomplete FreeBSD rescue incomplete Bacula GUI projects Bacula web php management code 24 bimagemgr Web application for burning CDROMs Header Files Please carefully follow the scheme defined below as it permits in gen eral only two header file includes per C file and thus vastly simplifies programming With a large complex
96. ing Bacula s serialization routines These routines guarantee that 67 the data is written to the output volume in a machine independent format Block Header The format of the Block Header version 1 27 and later is uint32_t CheckSum Block check sum uint32_t BlockSize Block byte size including the header uint32_t BlockNumber Block number char ID 4 BBO2 Identification and block level uint32_t VolSessionld Session Id for Job uint32_t VolSessionTime Session Time for Job The Block header is a fixed length and fixed format and is followed by Record Headers and Record Data The CheckSum field is a 32 bit checksum of the block data and the block header but not including the CheckSum field The Block Header is always immediately followed by a Record Header If the tape is damaged a Bacula utility will be able to recover as much information as possible from the tape by recov ering blocks which are valid The Block header is written using the Bacula serialization routines and thus is guaranteed to be in machine independent format See below for version 2 of the block header Record Header Each binary data record is preceded by a Record Header The Record Header is fixed length and fixed format whereas the binary data record is of variable length The Record Header is written using the Bacula serialization routines and thus is guaranteed to be in machine inde pendent format The format
97. involves only one or two command line commands and is fully automated Ideally there are one or more persons who will package each Bac ula release 33 Ideally there are one or more developers who can respond to and fix platform specific bugs Ideal requirements for a test machine The principal developer will have non root ssh access to the test machine at all times The pricipal developer will have a root password The test machine will provide approximately 200 MB of disk space for continual use The test machine will have approximately 500 MB of free disk space for temporary use The test machine will run the most common version of the OS The test machine will have an autochanger of DDS 4 technology or later having two or more tapes The test machine will have MySQL and or PostgreSQL database access for account bacula available The test machine will have sftp access The test machine will provide an smtp server Minimum requirements for a test machine The principal developer will have non root ssh access to the test machine when requested approximately once a month The pricipal developer not have root access The test machine will provide approximately 80 MB of disk space for continual use The test machine will have approximately 300 MB of free disk space for temporary use The test machine will run the the OS The test machine will have a tape drive of DDS 4 technology or later that ca
98. ion Entitled Endorsements Such a section may not be included in the Modified Version Do not retitle any existing section to be Entitled Endorsements or to conflict in title with any Invariant Section O Preserve any Warranty Disclaimers If the Modified Version includes new front matter sections or appendices that qualify as Secondary Sections and contain no material copied from the Document you may at your option designate some or all of these sections as invariant To do this add their titles to the list of Invariant Sections in the Modified Version s license notice These titles must be distinct from any other section titles You may add a section Entitled Endorsements provided it contains noth ing but endorsements of your Modified Version by various parties for ex ample statements of peer review or that the text has been approved by an organization as the authoritative definition of a standard You may add a passage of up to five words as a Front Cover Text and a passage of up to 25 words as a Back Cover Text to the end of the list of Cover Texts in the Modified Version Only one passage of Front Cover Text and one of Back Cover Text may be added by or through arrangements made by any one entity If the Document already includes a cover text for the same cover previously added by you or by arrangement made by the same entity you are acting on behalf of you may not add another but you may replace the
99. is was having finding previous contributors at http www mozilla org MPL missing html The other important issue is to avoid copyright patent or intellectual property violations as are currently May 2003 being claimed by SCO against IBM Although the copyright will be held by the Free Software Foundation Europe e V each developer is expected to indicate that he wrote and or modified a particular module or file and any other sources The copyright assignment may seem a bit unusual but in reality it is not Most large projects require this If you have any doubts about this please don t hesitate to ask The objective is to assure the long term servival of the Bacula project Items not needing a copyright assignment are most small changes enhance ments or bug fixes of 5 10 lines of code which amount to less than 20 Copyright Assignment Fiduciary License Agreement Since this is not a commercial enterprise and we prefer to believe in ev eryone s good faith previously developers could assign the copyright by explicitly acknowledging that they do so in their first submission This was sufficient if the developer is independent or an employee of a not for profit organization or a university However in an effort to ensure that the Bac ula code is really clean beginning in August 2006 all previous and future developers with SVN access will be asked to submit a copyright assignment or Fiduciary License Agreement
100. is kind of ill structured dynamic storage management is the source of so many storage leaks Without source code however there s no option but to provide a way to bypass SMARTALLOC for the buffers the library allocates and or releases with the standard system functions For each function zzz redefined by SMARTALLOC a corresponding rou tine named actuallyzxx1 is furnished which provides direct access to the underlying system function as follows Standard function Direct access function malloc size actuallymalloc size calloc nelem elsize actuallycalloc nelem elsize realloc ptr size actuallyrealloc ptr size free ptr actuallyfree ptr For example suppose there exists a system library function named getimage which reads a raster image file and returns the address of a buffer containing it Since the library routine allocates the image directly with malloc you can t use SMARTALLOC s free as that call ex 118 pects information placed in the buffer by SMARTALLOC s special version of malloc and hence would report an error To release the buffer you should call actuallyfree as in this code fragment struct image ibuf getimage ratpack img display_on_screen ibuf actuallyfree ibuf Conversely suppose we are to call a library function putimage which writes an image buffer into a file and then releases the buffer with free Since the system free is being called we can t
101. ite3 there is nothing more to do you can simply run the tests as described in the next section If you are using MySQL or PostgreSQL you will need to establish an account with your database engine for the user name regress and you will need to manually create a database named regress that can be used by user name regress which means you will have to give the user regress sufficient permissions to use the database named regress There is no password on the regress account You have probably already done this procedure for the user name and database named bacula If not the manual describes roughly how to do it and the scripts in bacula regress build src cats named create ysqlqatabase createyostgresqlyatabase grantmysql privileges andgrant ostgresql rivi Generally to do the above you will need to run under root to be able to create databases and modify permissions within MySQL and PostgreSQL 99 Running the Disk Only Regression Every time the Bacula source code has changed you will need to recompile it under the regression directory You do so with make setup The above will then copy the source code within the regression tree in directory regress build configure it and build it There should be no errors If there are please correct them before continuing From this point on as long as you don t change the Bacula source code you should not need to repeat any of the above steps If you pull down a new version of the
102. iting the tape LabelProg Name of the program that labeled the tape ProgVersion Version of the label program ProgDate Date Label program built Session Label Id 32 bytes Rasa Ss SSS A eee ee Sa Sa Se ees VerNum uint32_t o ee a ee JobId uint32_t 79 0 float64_t VerNum 11 aes PoolNane 128 bytes O E Polis gt UE a E tae G28 bytes AS ClientNane 128 bytes E en a a Job 128 bytes oo FileSetName 128 bytes A A sum Se Joblevel O ERE ae eae dea JobFiles uint32_t ae eee Saree ee ES JobBytes uint32_t Er a te e start_block uint32_t Garra iaa cie end_block uint32_t E start_file uint32_t estes anna a a a a end_file uint32_t gt nncan JobErrors uint32_t 22222 aaa po OS JobStatus uint32_t VerNum 11 gt fields deprecated Id 32 byte Bacula Identifier Bacula 1 0 immortal n LabelType in FileIndex field of Header EOM_LABEL 3 Label at EOM SOS_LABEL 4 Start of Session label EOS_LABEL 5 End of Session label VerNum 11 JobId Jobld write_btime Bacula time date this tape record written write_date Julian date tape this record written deprecated write_time Julian time tape this record written deprecated PoolName Pool Name PoolType Pool Type 76 MediaType Media Type ClientName Name of File daemon or Client writing this session Not used for EOM_LABEL Unix File Attributes
103. ix like systems The Win32 code was written according to the MSDN documentation but has not been tested These functions are prototyped as follows int bnet_set_nonblocking BSOCK bsock Enables non blocking I O on the socket associated with bsock Returns a copy of the socket flags prior to modification int bnet_set_blocking BSOCK bsock Enables blocking I O on the socket associated with bsock Returns a copy of the socket flags prior to modification void bnet_restore_blocking BSOCK bsock int flags Restores blocking or non blocking IO setting on the socket associated with bsock The flags argument must be the return value of either bnet_set_blocking or bnet_restore_blocking 94 0 1 5 Authentication Negotiation Backwards compatibility with the existing SSL negotiation hooks implemented in src lib cram md5 c have been maintained The cram_md5_get_auth function has been modified to accept an integer pointer argument tls remote_need The TLS requirement advertised by the remote host is returned via this pointer After exchanging cram md5 authentication and TLS requirements both the client and server independently decide whether to continue if cram_md5_get_auth dir password amp tls_remote_need cram_md5_auth dir password tls_local_need snip Verify that the remote host is willing to meet our TLS requirements if tls_remote_need lt tls_local_need amp amp tls_local_need BNET_TLS_
104. l continue As a consequence I am going to modify how I normally do development and instead of making a list of all the features that I will implement in the next version I will personally sign up for one maybe two projects at a time and when they are complete I will release a new version 10 The difference is that I will have more time to review the new code that is being contributed and will be able to devote more time to a smaller number of projects 1 38 had too many new features for me to handle correctly T expect that future release schedules will be much the same and the number of new features will also be much the same providing that the contributions continue to come and they show no signs of let up Feature Requests In addition I would like to formalize the feature requests a bit Instead of me maintaining an informal list of everything I run into kern stodo I would like to maintain a formal list of projects This means that all new feature requests including those recently discussed on the email lists must be formally submitted and approved Formal submission of feature requests will take two forms 1 non mandatory but highly recommended is to discuss proposed new features on the mailing list 2 Formal submission of an Feature Request in a special format ll give an example of this below but you can also find it on the web site under Support gt Feature Requests Since it take
105. lease see the Internal Bacula Database chapter of this manual for more details Database Table Design All discussions that follow pertain to the MySQL database The de tails for the PostgreSQL and SQLite databases are essentially identical except for that all fields in the SQLite database are stored as ASCII text and some of the database creation statements are a bit different The details of the internal Bacula catalog are not discussed here Because the Catalog database may contain very large amounts of data for large sites we have made a modest attempt to normalize the data tables to reduce redundant information While reducing the size of the database significantly it does unfortunately add some complications to the structures 50 In simple terms the Catalog database must contain a record of all Jobs run by Bacula and for each Job it must maintain a list of all files saved with their File Attributes permissions create date and the location and Media on which the file is stored This is seemingly a simple task but it represents a huge amount interlinked data Note the list of files and their attributes is not maintained when using the internal Bacula database The data stored in the File records which allows the user or administrator to obtain a list of all files backed up during a job is by far the largest volume of information put into the Catalog database Although the Catalog database has been designed to handle
106. level directory that we call bacula can be named anything you want Note all the standard regression scripts run as non root and can be run on the same machine as a production Bacula system Kern runs it this way To create the directory structure for the current trunk code and to checkout the necessary files do the following note we assume you are working in your home directory in a non root account cd mkdir bacula cd bacula svn checkout https bacula svn sourceforge net svnroot bacula trunk bacula svn checkout https bacula svn sourceforge net svnroot bacula trunk regress This will get the base files that you need The above should be needed only once Thereafter to update to the latest code you do cd bacula bacula svn update cd regress svn update If you want to test with SQLite and it is not installed on your system you will need to download the latest depkgs release from Source Forge and unpack it into bacula depkgs then simply 96 cd bacula depkgs make There are two different aspects of regression testing that this document will discuss 1 Running the Regression Script 2 Writing a Regression test Running the Regression Script There are a number of different tests that may be run such as the standard set that uses disk Volumes and runs under any userid a small set of tests that write to tape another set of tests where you must be root to run them Normally I run all my tests as non root and ve
107. lose to within 2GB any given file in a backup For restoring a full Job these records are not very important but if you want to retrieve a single file that was written near the end of a 100GB backup the JobMedia records can speed it up by orders of magnitude by permitting forward spacing files and blocks rather than reading the whole 100GB backup 56 Media Column Name Data Type Remark Mediald integer Primary Key VolumeName tinyblob Volume name Slot integer Autochanger Slot number or zero Poolld integer Link to Pool Record Media Type tinyblob The Media Type supplied by the user First Written datetime Time date when first written Last Written datetime Time date when last written LabelDate datetime Time date when tape labeled VolJobs integer Number of jobs written to this media VolFiles integer Number of files written to this media VolBlocks integer Number of blocks written to this me dia VolMounts integer Number of time media mounted VolBytes bigint Number of bytes saved in Job VolErrors integer Number of errors during Job VolWrites integer Number of writes to media MaxVolBytes bigint Maximum bytes to put on this media VolCapacityBytes bigint Capacity estimate for this volume VolStatus enum Status of media Full Archive Ap pend Recycle Read Only Disabled Error Busy Recycle tinyint Whether or not Bacula can recycle the Volumes
108. lt Storage daemon address gt port lt port number gt name lt DeviceName gt mediatype lt MediaType gt FD 2000 OK storage DR include DR lt directory1 gt DR lt directory2 gt DR Null packet FD 2000 OK include DR exclude DR lt directory1 gt DR lt directory2 gt DR Null packet FD 2000 OK exclude DR full FD 2000 OK full DR save FD 2000 OK save FD Attribute record for each file as sent to the Storage daemon described above FD Null packet FD lt append close responses from Storage daemon gt e g 3000 OK Volumes lt number of volumes gt 37 3001 Volume lt volume id gt lt start file gt lt start block gt lt end file gt lt end block gt lt volume session id gt 3002 Volume data lt date time of last write gt lt Number bytes written gt lt number errors gt additional Volume Volume data pairs for volumes 2 n FD Null packet FD close socket The Save Protocol Between the File Daemon and the Storage Daemon Once the Director has send a save command to the File daemon the File daemon will contact the Storage daemon to begin the save In what follows FD refers to information set via the network from the File daemon to the Storage daemon and SD refers to information set from the Storage daemon to the File daemon Command and Control Information Command and control information is exchanged in human readable ASCII commands FD listens SD makes con
109. media record exists for each of the NumVols contained in the Pool The PoolType is a Bacula defined keyword The MediaType is defined by the administrator and corre sponds to the MediaType specified in the Director s Storage definition record The CurrentVol is the sequence number of the Media record for the current volume Client Column Name Data Type Remark ClientId integer Primary Key Name TinyBlob File Services Name UName TinyBlob uname a from Client not yet used AutoPrune tinyint yes no for autopruning FileRetention bigint 64 bit seconds to retain Files JobRetention bigint 64 bit seconds to retain Job The Client table contains one entry for each machine backed up by Bacula in this database Normally the Name is a fully qualified domain 58 name Unsaved Files Column Name Data Type Remark Unsaved Id integer Primary Key Jobld integer JobId corresponding to this record PathId integer Id of path Filenameld integer Id of filename The UnsavedFiles table contains one entry for each file that was not saved Note This record is not yet implemented Counter Column Name Data Type Remark Counter tinyblob Counter name Min Value integer Start Min value for counter MaxValue integer Max value for counter Current Value integer Current counter value WrapCounter tinyblob Name of another counter The Counter
110. mpiler must provide support for 64 bit signed and un signed integers You will need a recent copy of the autoconf tools loaded on your system version 2 13 or later The autoconf tools are used to build the configuration program but are not part of the Bacula source distribution There are certain third party packages that Bacula needs Except for MySQL they can all be found in the depkgs and depkgs1 releases To build the Win32 binaries we use Microsoft VC standard 2003 Please see the instructions in bacula source src win32 README win32 for more details If you want to use VC Express please see README vc8 Our build is done under the most recent version of Cygwin but Cygwin is not used in the Bacula binaries that are produced Unfortunately we do not have the resources to help you build your own version of the Win32 FD so you are pretty much on your own You can ask the bacula devel list for help but please don t expect much Bacula requires a good implementation of pthreads to work The source code has been written with portability in mind and is mostly POSIX compatible Thus porting to any POSIX com patible operating system should be relatively easy 84 Steps to Take for Porting The first step is to ensure that you have version 2 13 or later of the autoconf tools loaded You can skip this step but making changes to the configuration program will be difficult or impossi ble The run a configure comma
111. n each job has its own record and block structures so there is a device control record or DCR that is the primary way of interfacing to the physical device The DCR contains all the job specific data as well as a pointer to the Device resource DEVRES structure and the physical DEVICE structure Now if a job is writing to two devices it could be writing two separate streams to the same device it must have two DCRs Today the code only permits one This won t be hard to change but it is new code Today three jobs threads two physical devices each job writes to only one device Jobi gt DCR1 gt DEVICE1 Job2 gt DCR2 gt DEVICE1 Job3 gt DCR3 gt DEVICE2 To be implemented three jobs three physical devices but jobl is writ ing simultaneously to three devices Jobi gt DCR1 gt DEVICE1 gt DCR4 gt DEVICE2 gt DCR5 gt DEVICES Job2 gt DCR2 gt DEVICE1 Job3 gt DCR3 gt DEVICE2 Job job control record DCR Job contorl data for a specific device DEVICE Device only control data 48 Catalog Services General This chapter is intended to be a technical discussion of the Catalog services and as such is not targeted at end users but rather at devel opers and system administrators that want or need to know more of the working details of Bacula The Bacula Catalog services consist of the programs that provide the SQL database engine for storage and retrieval of all information
112. n Gn Trt ae Se soak Gk BS 90 0 1 1 Introduction to TLS 90 TLS Introducir oe hae Re eR Be es 90 0 1 2 New Configuration Directives 90 New Configuration Directives 0 0004 90 0 1 3 TLS API Implementation 91 TLS API Implementation 91 0 1 4 Bnet API Changes 93 Bnet API Changes 2 22 00 0004 93 0 1 5 Authentication Negotiation 95 Authentication Negotiation 95 Bacula Regression Testing o 96 General vi o r a iog al e 2 nd y ale Dw A 96 Running the Regression Script ooo 97 Running a Single Test 103 Writing a Regression Test 103 a tt Da e as Ir A E C Ada ia 105 Command Line Message Digest Utility 105 Download md5 zip Zipped archive 107 Bacula Memory Management 2 004 108 General a A EA a A Bd hye hy eee 108 TCP IP Network Protocol ue e A a 112 General soa Sd a a A ee a ia Soe 112 bnetiand Threads cana a de 112 bileb Open la ia E a Gre Sed 112 bnet send ato aa a ee a aS 113 piet fenduro en a a e A DA a 113 Additional Error information 113 PRETEC egna ara a a Me BE AAA A da 113 Piet sit e etry ka aat fees a a ee eee Ker ee at 114 BrietsStrerror Ln e a eee Sa A ee ee ee 114 bnetuclose gt a dl eee eee Bas 114 Becoming a Server diia Gok ake p
113. n Win32 command line executable program md5 exe compiled using Microsoft Visual C 5 0 and in source code form along with a Makefile to build the program under Unix See Also sum 1 Exit Status md5 returns status 0 if processing was completed without errors 1 if the c option was specified and the given signature does not match that of the input and 2 if processing could not be performed at all due for example to a nonexistent input file Copying This software is in the public domain Permission to use copy modify and distribute this software and its documen tation for any purpose and without fee is hereby granted without any conditions or restrictions This software is pro vided as is without express or implied warranty Acknowledgements The MD5 algorithm was developed by Ron Rivest The public domain C language implementation used in this program was written by Colin Plumb in 1993 by John Walker January 6th MIM 107 Bacula Memory Management General This document describes the memory management routines that are used in Bacula and is meant to be a technical discussion for developers rather than part of the user manual Since Bacula may be called upon to handle filenames of varying and more or less arbitrary length special attention needs to be used in the code to ensure that memory buffers are sufficiently large There are four possibilities for memory usage within Bacula Each will be described in
114. n be scheduled for access The test machine will not have MySQL and or PostgreSQL database access The test machine will have no sftp access The test machine will provide no email access Bare bones test machine requirements The test machine is available only to a designated test person your own machine The designated test person runs the regession tests on demand The test machine has a tape drive available 34 Daemon Protocol General This document describes the protocols used between the various dae mons As Bacula has developed it has become quite out of date The general idea still holds true but the details of the fields for each command and indeed the commands themselves have changed con siderably It is intended to be a technical discussion of the general daemon proto cols and as such is not targeted at end users but rather at developers and system administrators that want or need to know more of the working details of Bacula Low Level Network Protocol At the lowest level the network protocol is handled by BSOCK pack ets which contain a lot of information about the status of the network connection who is at the other end etc Each basic Bacula network read or write actually consists of two low level network read writes The first write always sends four bytes of data in machine indepen dent byte order If data is to follow the first four bytes are a positive non zero integer indicating the lengt
115. nature u dinput_text infile outfile Description A message digest is a compact digital signature for an arbitrarily long stream of binary data An ideal message digest algorithm would never generate the same signature for two different sets of input but achieving such theoretical perfection would require a message digest as long as the input file Prac tical message digest algorithms compromise in favour of a digital signature of modest size created with an algorithm designed to make preparation of input text with a given signature computationally infeasible Message di gest algorithms have much in common with techniques used in encryption but to a different end verification that data have not been altered since the signature was published Many older programs requiring digital signatures employed 16 or 32 bit cycli cal redundancy codes CRC originally developed to verify correct transmis sion in data communication protocols but these short codes while adequate to detect the kind of transmission errors for which they were intended are insufficiently secure for applications such as electronic commerce and verifi cation of security related software distributions The most commonly used present day message digest algorithm is the 128 bit MD5 algorithm developed by Ron Rivest of the MIT Laboratory for Computer Science and RSA Data Security Inc The algo rithm with a reference implementation was published as Internet
116. nd VolSes sionTime On reading each record from the Block the VolSessionId and VolSessionTime is filled in the Record Header from the Block Header Volume Label Format Tape volume labels are created by the Storage daemon in response to a label command given to the Console program or alternatively by the btape program created Each volume is labeled with the following information using the Bacula serialization routines which guarantee machine byte order independence For Bacula versions 1 27 and later the Volume Label Format is char Id 32 Bacula 1 0 Immortal n uint32_t VerNum Label version number VerNum 11 and greater Bacula 1 27 and later btime_t label_btime Time date tape labeled btime_t write_btime Time date tape first written The following are O in VerNum 11 and greater float64_t write_date Date this label written float64_t write_time Time this label written char VolName 128 Volume name char PrevVolName 128 Previous Volume Name char PoolName 128 Pool name char PoolType 128 Pool type char MediaType 128 Type of this media char HostName 128 Host name of writing computer char LabelProg 32 Label program name char ProgVersion 32 Program version char ProgDate 32 Program build date time Note the LabelType Volume Label Volume PreLabel Session Start Label is stored in the recor
117. nd has no effect on the meaning of this License 2 VERBATIM COPYING You may copy and distribute the Document in any medium either commer cially or noncommercially provided that this License the copyright notices and the license notice saying this License applies to the Document are re produced in all copies and that you add no other conditions whatsoever to those of this License You may not use technical measures to obstruct or control the reading or further copying of the copies you make or distribute However you may accept compensation in exchange for copies If you dis tribute a large enough number of copies you must also follow the conditions in section 3 You may also lend copies under the same conditions stated above and you may publicly display copies 3 COPYING IN QUANTITY If you publish printed copies or copies in media that commonly have printed covers of the Document numbering more than 100 and the Document s license notice requires Cover Texts you must enclose the copies in covers that carry clearly and legibly all these Cover Texts Front Cover Texts on the front cover and Back Cover Texts on the back cover Both covers must also clearly and legibly identify you as the publisher of these copies The front cover must present the full title with all words of the title equally prominent and visible You may add other material on the covers in addition Copying with changes limited to the covers as long as
118. nd in the main source directory and examine the output It should look something like the fol lowing Configuration on Mon Oct 28 11 42 27 CET 2002 Host 1686 pc linux gnu redhat 7 3 Bacula version Source code location 1 27 26 October 2002 sbin Install binaries Install config files etc bacula C Compiler gcc C Compiler c Compiler flags g 02 Linker flags Libraries 1pthread Statically Linked Tools no Database found no Database type Internal Database lib Job Output Email root localhost Traceback Email root localhost SMTP Host Address localhost Director Port 9101 File daemon Port 9102 Storage daemon Port 9103 Working directory etc bacula working SQL binaries Directory Large file support yes readline support yes cweb support yes home kern bacula depkgs cweb TCP Wrappers support no ZLIB support yes enable smartalloc yes enable gnome no gmp support yes The details depend on your system The first thing to check is that it properly identified your host on the Host line The first part added in version 1 27 is the GNU four part identification of your system The part after the is your system and the system version Generally if your system is not yet supported you must correct these If the configure does not function properly you must deter mine the cause and fix it Generally it will be because some 85 required system routine is not available
119. nd session lt ticket number gt SD 3000 OK end FD append close session lt ticket number gt SD 3000 OK Volumes lt number of volumes gt SD 3001 Volume lt volumeid gt lt start file gt lt start block gt 39 lt end file gt lt end block gt lt volume session id gt SD 3002 Volume data lt date time of last write gt lt Number bytes written gt lt number errors gt SD additional Volume Volume data pairs for volumes 2 n FD close socket The information returned to the File daemon by the Storage daemon in response to the append close session is transmit in turn to the Director 40 Director Services Daemon This chapter is intended to be a technical discussion of the Director services and as such is not targeted at end users but rather at devel opers and system administrators that want or need to know more of the working details of Bacula The Bacula Director services consist of the program that supervises all the backup and restore operations To be written 41 File Services Daemon Please note this section is somewhat out of date as the code has evolved significantly The basic idea has not changed though This chapter is intended to be a technical discussion of the File dae mon services and as such is not targeted at end users but rather at developers and system administrators that want or need to know more of the working details of Bacula The Bacula File Services consist of th
120. nection FD append open session lt JobId gt lt password gt SD 3000 OK ticket lt number gt FD append data lt ticket number gt SD 3000 OK data address lt IPaddress gt port lt port gt Data Information The Data information consists of the file attributes and data to the Storage daemon For the most part the data information is sent one way from the File daemon to the Storage daemon This allows the File daemon to transfer information as fast as possible without a lot of handshaking and network overhead However from time to time the File daemon needs to do a sort of checkpoint of the situation to ensure that everything is going well with the Storage daemon To do so the File daemon sends a packet with a negative length indicating that he wishes the Storage daemon to respond by sending a packet of information to the File daemon The File daemon then waits to receive a packet from the Storage daemon before continuing 38 All data sent are in binary format except for the header packet which is in ASCII There are two packet types used data transfer mode a header packet the contents of which are known to the Storage daemon and a data packet the contents of which are never examined by the Storage daemon The first data packet to the Storage daemon will be an ASCII header packet consisting of the following data lt File Index gt lt Stream Id gt lt Info gt where lt File Index gt is a se quential num
121. nput stdin UA_sock if UA_sock bnet_sig UA_sock BNET_TERMINATE send EOF bnet_close UA_sock exit 0 Then the read_and_process_input routine looks like the following get input to send to the Director bnet_fsend UA_sock s input stat bnet_recv UA_sock process output from the Director For a GUI program things will be a bit more complicated Basically in the very inner loop you will need to check and see if any output is available on the UA sock For an example please take a look at the GNOME GUI interface code in lt bacula source amp gt src gnome console console c 89 0 1 TLS Written by Landon Fuller 0 1 1 Introduction to TLS This patch includes all the back end code necessary to add complete TLS data encryption support to Bacula In addition support for TLS in Console Director communications has been added as a proof of concept Adding support for the remaining daemons will be straight forward Supported features of this patchset include Client Server TLS Requirement Negotiation TLSv1 Connections with Server and Client Certificate Validation Forward Secrecy Support via Diffie Hellman Ephemeral Keying This document will refer to both server and client contexts These terms refer to the accepting and initiating peer respectively Diffie Hellman anonymous ciphers are not supported by this patchset The use of DH anonymous ciphers increases the code complexity an
122. nt or need to know more of the working details of Bacula Definitions Block A block represents the primitive unit of information that the Storage daemon reads and writes to a physical device Normally for a tape device it will be the same as a tape block The Storage daemon always reads and writes blocks A block consists of block header information followed by records Clients of the Storage daemon the File daemon normally never see blocks However some of the Storage tools bls bscan bextract may be use block header information In older Bacula tape versions a block could contain records see record definition below from multi ple jobs However all blocks currently written by Bacula are block level BB02 and a given block contains records for only a single job Different jobs simply have their own private blocks that are intermingled with the other blocks from other jobs on the Volume previously the records were intermingled within the blocks Having only records from a single job in any give block permitted moving the VolumeSessionId and VolumeSessionTime see below from each record heading to the Block header This has two advantages 1 a block can be quickly rejected based on the contents of the header without reading all the records 2 because there is on the average more than one record per block less data is written to the Volume for each job Record A record consists of a Record Header which is managed by the Sto
123. nted for win32 int cleanup_tls void Performs TLS library cleanup 91 Manipulating TLS Contexts TLS_CONTEXT new_tls_context const char ca_certfile const char ca_certdir const char certfile const char keyfile const char dhfile bool verify_peer Allocates and initalizes a new opaque TLS_CONTEXT structure The TLS_ CONTEXT structure maintains default TLS settings from which TLS_CONNECTION structures are instantiated In the future the TLS_CONTEXT structure may be used to maintain the TLS session cache ca_certfile and ca_certdir arguments are used to initialize the CA verification stores The certfile and keyfile arguments are used to initialize the local certificate and private key If dhfile is non NULL it is used to initialize Diffie Hellman ephemeral keying If verify_peer is true client certificate validation is enabled void free_tls_context TLS_CONTEXT ctx Deallocated a previously allocated TLS CONTEXT structure Performing Post Connection Verification bool tls_postconnect_verify_host TLS_CONNECTION tls const char host Performs post connection verification of the peer supplied x509 cer tificate Checks whether the subjectAltName and commonName at tributes match the supplied host string Returns true if there is a match false otherwise bool tls_postconnect_verify_cn TLS_CONNECTION tls alist verify_list Performs post connection verification of the peer supplied x509 certifi cate C
124. ntifies itself and presents its password If the password check is OK the Storage daemon accepts the Director The Director then passes the Storage daemon the Jobld to be run as well as the File daemon authorization append read all or read for a specific session The Storage daemon will then pass back to the Director a enabling key for this JobId that must be presented by the File daemon when opening the job Until this process is complete the Storage daemon is not available for use by File daemons SD listens 36 DR makes connection DR Hello lt Director name gt calling lt password gt SD 3000 OK Hello DR JobId nnn Allow append read Session SessionId Session not implemented yet SD 3000 OK Job Authorization lt password gt DR use device lt device name gt media_type lt media type gt pool_name lt pool name gt pool_type lt pool_type gt SD 3000 OK use device For the Director to be authorized the lt Director name gt and the lt password gt must match the values in one of the Storage daemon s Director resources there may be several Directors that can access a single Storage daemon The Protocol Used Between the Director and the File Daemon A typical conversation might look like the following FD listens DR makes connection DR Hello lt Director name gt calling lt password gt FD 2000 OK Hello DR JobId nnn Authorization lt password gt FD 2000 OK Job DR storage address
125. o only be used in the development debug process They are coded as DmsgN level message argl where the N is a number indicating how many arguments are to be substituted into the message i e it is a count of the number arguments you have in your message generally the number of percent signs level is the debug level at which you wish the message to be printed message is the debug message to be printed and argl are the arguments to be substituted Since not all compilers support defines with varargs you must explicitly specify how many arguments you have When the debug message is printed it will automatically be prefixed by the name of the daemon which is running the filename where the Dmsg is and the line number within the file Some actual examples are Dmsg2 20 MD5len d MD5 s n strlen buf buf Dmsg1 9 Created client s record n client hdr name Error Messages Error messages are messages that are related to the daemon as a whole rather than a particular job For example an out of memory condition my generate an error message They should be very rarely needed In general you should be using Job and Job Queued messages Jmsg and Qmsg They are coded as EmsgN error code level message argl As with debug messages you must explicitly code the of arguments to be substituted in the message error code indicates the severity or class of error and it may be one of the followin
126. ode and make reading it difficult unless you are in love with colons Do Use Whenever Possible Locking and unlocking within a single subroutine A single point of exit from all subroutines A goto is per fectly OK to use to get out early but only to a label named bailout andpossiblyanok ut Seecurrentcodeexamples Malloc and free within a single subroutine Comments and global explanations on what your code or algorithm does 26 Indenting Standards I cannot stand code indented 8 columns at a time This makes the code unreadable Even 4 at a time uses a lot of space so I have adopted indenting 3 spaces at every level Note indention is the visual appearance of the source on the page while tabbing is replacing a series of up to 8 spaces from a tab character The closest set of parameters for the Linux indent program that will produce reasonably indented code are nbad bap bbo nbc br brs c36 cd36 ncdb ce ci3 cli0 cp36 d0 di1 ndj nfc1 nfca hnl i3 ip0 185 lp npcs nprs npsl saf sai saw nsob nss nbc ncs nbfda You can put the above in your indent pro file and then just invoke indent on your file However be warned This does not produce perfect indenting and it will mess up C class statements pretty badly Braces are required in all if statements missing in some very old code To avoid generating too many lines the first brace appears on the first line e g of an if
127. of the Record Header version 1 27 or later is int32_t FileIndex File index supplied by File daemon int32_t Stream Stream number supplied by File daemon uint32_t DataSize size of following data record in bytes This record is followed by the binary Stream data of DataSize bytes followed by another Record Header record and the binary stream data For the definitive definition of this record see record h in the src stored directory Additional notes on the above 68 The VolSessionId is a unique sequential number that is assigned by the Storage Daemon to a particular Job This number is sequential since the start of execution of the daemon The VolSessionTime is the time date that the current execution of the Storage Daemon started It assures that the combination of VolSessionId and VolSessionTime is unique for every jobs written to the tape even if there was a machine crash between two writes The FileIndex is a sequential file number within a job The Storage daemon requires this index to be greater than zero and sequential Note however that the File daemon may send multiple Streams for the same FileIndex In addition the Storage daemon uses negative FileIndices to hold the Begin Session Label the End Session Label and the End of Volume Label The Stream is defined by the File daemon and is used to identify separate parts of the data saved for each file Unix attributes Win32 attributes file data compr
128. om the Director for a 43 Bacula Building the Test 99 Bacula Memory Management 108 Bacula Porting Notes 84 10 Bacula Regression Testing 96 Becoming a Server 114 Begin Session Label 66 Block 64 Block Header 65 68 Blocking State Socket Manipulating 94 Bnet and Threads 112 Bnet API Changes 93 Bnet_close 114 Bnet_fsend 113 Bnet_open 112 Bnet_recv 113 Bnet_send 113 Bnet sig 114 Bnet_strerror 114 Bugs 107 Building the Test Bacula 99 Catalog Internal Bacula 50 Catalog Services 49 Classes Message 30 Command and Control Informa tion 38 Command Line Message Digest Utility 105 Commands Received from the Di rector for a Backup 43 Commands Received from the Di rector for a Restore 43 Connections Manipulating TLS 92 Contexts Manipulating TLS 92 Conventions Higher Level 115 Copying 107 123 Daemon Director Services 41 File Services 42 Protocol Used Between the Director and the File 37 Protocol Used Between the Director and the Storage 36 Save Protocol Between the File Daemon and the Storage 38 Daemon Protocol 35 Data Information 38 Data Record 66 Database Table Design 50 Database Tables 51 DataSize 69 Debug Messages 30 Definition MySQL Table 60 Definitions 64 Description 105 Design Database Table 50 Storage Daemon 44 Details SMARTALLOC 119 Detection Smart Memory Allocation With Orphaned Buffer 116 Difficult GUI Interfac
129. ome a server i e wait for a connection from a client use the routine bnet_thread server The calling sequence is a bit complicated please refer to the code in bnet_server c and the code at the beginning of each daemon as examples of how to call it 114 Higher Level Conventions Within Bacula we have established the convention that any time a single record is passed it is sent with bnet_send and read with bnet_recv Thus the normal exchange between the server S and the client C are wait for connection C attempt connection accept connection C bnet_send send request bnet_recv wait for request act on request bnet_send send ack C bnet_recv wait for ack vn nn un un Thus a single command is sent acted upon by the server and then acknowl edged In certain cases such as the transfer of the data for a file all the information or data cannot be sent in a single packet In this case the convention is that the client will send a command to the server who knows that more than one packet will be returned In this case the server will enter a loop while n bnet_recv bsock gt 0 act on request if n lt 0 error The client will perform the following bnet_send bsock bnet_send bsock bnet_sig bsock BNET_EOD Thus the client will send multiple packets and signal to the server when all the packets have been sent by sending a zero length record 115 Suatalloc Smart M
130. put in the catalog During a Verify of the catalog the situation is different since the File daemon will have an exchange with the Director for each file and will not contact the Storage daemon A Restore operation will be very similar to the Backup except that during the Restore the Storage daemon will not send storage coordi nates to the Director since the Director presumably already has them On the other hand any error messages from either the Storage daemon or File daemon will normally be sent directly to the Directory this of course depends on how the Message resource is defined 42 Commands Received from the Director for a Backup To be written Commands Received from the Director for a Restore To be written 43 Storage Daemon Design This chapter is intended to be a technical discussion of the Storage daemon services and as such is not targeted at end users but rather at developers and system administrators that want or need to know more of the working details of Bacula This document is somewhat out of date SD Design Introduction The Bacula Storage daemon provides storage resources to a Bacula in stallation An individual Storage daemon is associated with a physical permanent storage device for example a tape drive CD writer tape changer or jukebox etc and may employ auxiliary storage resources such as space on a hard disk file system to increase performance and or optimize use of the perm
131. py of the repository and it will check for conflicts If there are any it will tell you and you will need to resolve them The problems of resolving conflicts are a bit more than this document can cover but you can examine the files it claims have conflicts and look for lt lt lt lt or look in the rej files that it creates If you have problems just ask on the developer s list Note doing the above svn update is not absolutely necessary There are times when you may be working on code and you want to commit it but you explicitly do not want to move up to the latest version of the code in the SVN If that is the case you can simply skip the svn update and do the commit shown below If the commit fails because of a conflict it will tell you and you must resolve the conflict before it will permit you to do the commit Once your local copy of the repository has been updated you can now commit your changes svn commit m Some comment about what you changed or if you really only want to commit a single file you can do 19 svn commit m comment scripts file I edited Note if you have done a build in your directory or you have added other new files the commit will update only the files that are actually in the repository For example none of the object files are stored in the repository so when you do a commit those object files will simply be ignored If you want to add new files or remove files from the m
132. r data at all effectively removing it from the program even if named in the link instructions Thus except for unusual circumstances a program that works with SMAR TALLOC defined for testing should require no changes when built without it for production release The alloc Function Many programs I ve worked on use very few direct calls to malloc us ing the identically declared alloc function instead Alloc detects out of memory conditions and aborts removing the need for error checking on every call of malloc and the temptation to skip checking for out of memory As a convenience SMARTALLOC supplies a compatible version of alloc in the file alloc c with its definition in the file alloc h This version of alloc is sensitive to the definition of SMARTALLOC and cooperates with SMARTALLOC s orphaned buffer detection In addition when SMARTAL LOC is defined and alloc detects an out of memory condition it takes advantage of the SMARTALLOC diagnostic information to identify the file and line number of the call on alloc that failed 121 Overlays and Underhandedness String constants in the C language are considered to be static arrays of characters accessed through a pointer constant The arrays are potentially writable even though their pointer is a constant SMARTALLOC uses the compile time definition smartall wml to obtain the name of the file in which a call on buffer allocation was performed Rather than re
133. r testing but are created from the above directories and are removed with make distclean bin This is the install directory for Bacula to be used testing build Where the Bacula source build tree is tmp Most temp files go here S working Bacula working directory weird files Weird files used in two of the tests Adding a New Test If you want to write a new regression test it is best to start with one of the existing test scripts and modify it to do the new test When adding a new test be extremely careful about adding anything to any of the daemons configuration files The reason is that it may change the prompts that are sent to the console For example adding a Pool means that the current scripts which assume that Bacula automatically selects a Pool will now be presented with a new prompt so the test will fail If you need to enhance the configuration files consider making your own versions 104 Bacula MD5 Algorithm Command Line Message Digest Utility This page describes md5 a command line utility usable on either Unix or MS DOS Windows which generates and verifies message digests digital signatures using the MD5 algorithm This program can be useful when developing shell scripts or Perl programs for software installation file com parison and detection of file corruption and tampering Name md5 generate check MD5 message digest Synopsis md5 csig
134. rage daemon and Record Data which is the data received from the Client A record is the primitive unit of information sent to and from the Storage daemon by the Client File daemon programs The details are described below JobId A number assigned by the Director daemon for a particular job This number will be unique for that particular Director 64 Catalog The daemons use this number to keep track of in dividual jobs Within the Storage daemon the JobId may not be unique if several Directors are accessing the Storage daemon simultaneously Session A Session is a concept used in the Storage daemon corre sponds one to one to a Job with the exception that each session is uniquely identified within the Storage daemon by a unique Ses sionId SessionTime pair see below VolSessionId A unique number assigned by the Storage daemon to a particular session Job it is having with a File daemon This number by itself is not unique to the given Volume but with the VolSessionTime it is unique VolSessionTime A unique number assigned by the Storage daemon to a particular Storage daemon execution It is actually the Unix time_t value of when the Storage daemon began execution cast to a 32 bit unsigned integer The combination of the VolSes sionId and the VolSessionTime for a given Storage daemon is guaranteed to be unique for each Job or session FileIndex A sequential number beginning at one assigned by the File daemon to the files wi
135. readn BSOCK bsock char ptr int32_t nbytes Reads nbytes from the TLS CONNECTION associated with bsock and stores the result in ptr Due to OpenSSL s handling of EINTR bsock is set non blocking at the start of the function and restored to its original blocking state before the function returns Less than nbytes may be read if an error occurs The actual number of bytes read will be returned 0 1 4 Bnet API Changes A minimal number of changes were required in the Bnet socket API The BSOCK structure was expanded to include an associated TLS_CONNECTION structure as well as a flag to designate the cur rent blocking state of the socket The blocking state flag is required for win32 where it does not appear possible to discern the current blocking state of a socket 93 Negotiating a TLS Connection bnet_tls_server and bnet_tls_client were both implemented using the new TLS API as follows int bnet_tls_client TLS_CONTEXT ctx BSOCK bsock Negotiates a TLS session via bsock using the settings from ctz Returns 1 if successful 0 otherwise int bnet_tls_server TLS_CONTEXT ctx BSOCK bsock alist verify_list Accepts a TLS client session via bsock using the settings from ctx If verify_list is non NULL it is passed to tls_postconnect_verify_cn for client certificate verification Manipulating Socket Blocking State Three functions were added for manipulating the blocking state of a socket on both Win32 and Un
136. report jbr If the Jmsg is followed with a number such as Jmsgl the number indicates the number of arguments to be substituted varargs is not standard for defines and what is more important is that the file and line number will be prefixed to the message This permits a sort of debug from user s output 31 Queued Job Messages Queued Job messages are similar to Jmsg s except that the message is Queued rather than immediately dispatched This is necessary within the network subroutines and in the message editing routines This is to prevent recursive loops and to ensure that messages can be delivered even in the event of a network error Memory Messages Memory messages are messages that are edited into a memory buffer Generally they are used in low level routines such as the low level device file dev c in the Storage daemon or in the low level Catalog routines These routines do not generally have access to the Job Control Record and so they return error messages reformatted in a memory buffer Mmsg is the way to do this 32 Platform Support General This chapter describes the requirements for having a supported plat form Operating System In general Bacula is quite portable It supports 32 and 64 bit architectures as well as bigendian and littleen dian machines For full support the platform Operating System must implement POSIX Unix system calls However for File daemon support only a small compatibility
137. roperties svn proplist file c Create a branch for a new version svn copy https bacula svn sourceforge net svnroot bacula trunk https bacula svn sourceforge net svnroot bacula branches Branch 2 Tag a version for a new release svn copy https bacula svn sourceforge net svnroot bacula branches Branch https bacula svn sourceforge net svnroot bacula branches Release 2 Let s say you are working in the directory scripts You would then do cd scripts edit some files 18 when you are happy with your changes you can do the following cd bacula to your top level directory svn diff my changes patch When the command is done you can look in the file my changes patch and you will see all the changes you have made to your copy of the repository Make sure that you understand all the changes that it reports before proceeding If you modified files that you do do not want to commit to the main repository you can simply delete them from your local directory and they will be restored from the repository with the svn update that is shown below Normally you should not find changes to files that you do not want to commit and if you find yourself in that position a lot you are probably doing something wrong Let s assume that now you want to commit your changes to the main SVN repository First do cd bacula svn update When you do this it will pull any changes made by other developers into your local co
138. rotocol is packet based and thus pieces of information sent can be ASCII or binary The packet interface permits knowing where the end of a list is The packet interface permits special signals to be passed rather than data The Director has a number of commands that are non interactive They all begin with a period and provide things such as the list of all Jobs list of all Clients list of all Pools list of all Stor age Thus the GUI interface can get to virtually all informa tion that the Director has in a deterministic way See lt bacula source gt src dird ua_dotcmds c for more details on this Most console commands allow all the arguments to be specified on the command line e g run job NightlyBackup level Full One of the first things to overcome is to be able to establish a conver sation with the Director Although you can write all your own code it is probably easier to use the Bacula subroutines The following code is used by the Console program to begin a conversation 88 static BSOCK UA_sock NULL static JCR jcr read your config getting address and pasword UA_sock bnet_connect NULL 5 15 Director daemon dir gt address NULL dir gt DIRport 0 if UA_sock NULL 4 terminate_console 0 return 1 jcr dir_bsock UA_sock if authenticate_director amp jcr dir fprintf stderr ERR s UA_sock gt msg terminate_console 0 return 1 read_and_process_i
139. rovided constructed for ALL systems Link when the FT code is FT_LNK or FT_LNKSAVED the item in question is a Unix link and this field contains the fully qualified link name When the FT code is not FT_LNK or FT_LNKSAVED this field is null Extended Attributes The exact format of this field is operating system dependent It contains additional or extended attributes of a system dependent nature Currently this field is used only on WIN32 systems where it contains a ASCII base64 representation TT of the WIN32_FILE_ATTRIBUTE_DATA structure as defined by Windows The fields in the base64 representation of this structure are like the File Attributes separated by spaces The File attributes consist of the following Field No Stat Name Unix Win98 NT MacOS 1 st_dev Device number Drive number vRefNum of filesystem 2 stino Inode number Always 0 fileID dirID 3 st_mode File mode File mode 777 dirs apps 666 docs 444 locked docs 4 st_nlink Number of Number of Always 1 links to the file link only on NTFS 5 st_uid Owner ID Always 0 Always 0 6 st_gid Group ID Always 0 Always 0 7 st_rdev Device ID for Drive No Always 0 special files 8 st_size File size in File size in Data fork file size in bytes bytes bytes 9 st_blksize Preferred block Always 0 Preferred block size size 10 st_blocks Number of Always 0 Number of blocks al blocks allo located cated 11 st_
140. run for a long time As a consequence from the very beginning Bacula uses SmartAlloc to ensure that there are no memory leaks To make detection of memory leaks effective all Bacula code that dynamically allocates memory MUST have a way to release it In general when the memory is no longer needed it should be immediately released but in some cases the memory will be held during the entire time that Bacula is executing In that case there MUST be a routine that can be called at termination time that releases the memory In this way we will be able to detect memory leaks Be sure to immediately correct any and all memory leaks that are printed at the termination of the daemons 22 Special Files Kern uses files named 1 2 9 with any extension as scratch files Thus any files with these names are subject to being rudely deleted at any time When Implementing Incomplete Code Please identify all incomplete code with a comment that contains kFIXME where there are three asterisks before and after the word FIXME in capitals and no intervening spaces This is important as it al lows new programmers to easily recognize where things are partially implemented Bacula Source File Structure The distribution generally comes as a tar file of the form bac ula x y z tar gz where x y and z are the version release and update numbers respectively Once you detar this file you will have a directory structure as follows
141. ry rarely run the root tests The tests vary in length and running the full tests including disk based testing tape based testing autochanger based testing and multiple drive autochanger based testing can take 3 or 4 hours Setting the Configuration Parameters There is nothing you need to change in the source directory To begin cd bacula regress The very first time you are going to run the regression scripts you will need to create custom xxx conf file for your system You can either edit prototype conf directly or copy it to a new file and edit it To see a real example of a configuration file look at kern conf The variables you need to modify are Where to get the source to be tested BACULA_SOURCE HOME bacula bacula Where to send email 11111 Change me EMAIL your name your domain com SMTP_HOST localhost Full default path where to find sqlite no quotes SQLITE3_DIR HOME bacula depkgs sqlite3 SQLITE_DIR HOME bacula depkgs sqlite TAPE_DRIVE dev nst0 if you don t have an autochanger set AUTOCHANGER to dev null 97 AUTOCHANGER dev sg0 For two drive tests set to dev null if you do not have it TAPE_DRIVE1 dev null This must be the path to the autochanger including its name AUTOCHANGER_PATH usr sbin mtx Set your database here WHICHDB with sqlite SQLITE_DIR WHICHDB with sqlite3 SQLITE3_DIR WHICHDB with mysql WHICHDB with postgresql
142. s csignature Computes the signature of the specified infile or the string supplied by the d option and compares it against the specified signature If the two signatures match the exit status will be zero otherwise the exit status will be 1 No signature is written to outfile or standard output only the exit status is set The sig nature to be checked must be specified as 32 hexadecimal digits dinput_text A signature is computed for the given input_text which must be quoted if it contains white space characters instead of input from infile or standard input If input is specified with the d option no infile should be specified u Print how to call information Files If no infile or d option is specified or infile is a single md5 reads from standard input if no outfile is given or outfile is a single output is sent to standard output Input and output are processed strictly serially consequently md5 may be used in pipelines 106 Bugs The mechanism used to set standard input to binary mode may be specific to Microsoft C if you rebuild the DOS Windows version of the program from source using another compiler be sure to verify binary files work properly when read via redirection or a pipe This program has not been tested on a machine on which int and or long are longer than 32 bits Download md5 zip Zipped archive The program is provided as md5 zip a Zipped archive containing an ready to ru
143. s and required Cover Texts given in the Document s license notice Include an unaltered copy of this License Preserve the section Entitled History Preserve its Title and add to it an item stating at least the title year new authors and publisher of the Modified Version as given on the Title Page If there is no section Entitled History in the Document create one stating the title year authors and publisher of the Document as given on its Title Page then add an item describing the Modified Version as stated in the previous sentence Preserve the network location if any given in the Document for public access to a Transparent copy of the Document and likewise the network locations given in the Document for previous versions it was based on These may be placed in the History section You may omit a network location for a work that was published at least four years before the Document itself or if the original publisher of the version it refers to gives permission For any section Entitled Acknowledgements or Dedications Preserve the Title of the section and preserve in the section all the substance and tone of each of the contributor acknowledge ments and or dedications given therein Preserve all the Invariant Sections of the Document unaltered in their text and in their titles Section numbers or the equivalent are not considered part of the section titles Delete any sect
144. s a bit of time to properly fill out a Feature Request form you probably should check on the email list first Once the Feature Request is received by the keeper of the projects list it will be sent to me and I will either accept it send it back asking for clarification send it to the email list asking for opinions or reject it If it is accepted it will go in the projects file a simple ASCII file main tained in the main Bacula source directory Implementation of Feature Requests Any qualified developer can sign up for a project The project must have an entry in the projects file and the developer s name will appear in the Status field How Feature Requests are accepted Acceptance of Feature Requests depends on several things 1 feedback from users If it is negative the Feature Request will probably not be accepted 2 the difficulty of the project A project that is so difficult that I cannot imagine finding someone to implement probably won t be accepted 3 whether or not the Feature Request fits within the current stategy of Bacula for example an Feature Request that requests changing the tape to tar format would not be accepted 11 How Feature Requests are prioritized Once an Feature Request is accepted it needs to be implemented If you can find a developer for it or one signs up for implementing it then the Feature Request becomes top priority at least for that developer Between rel
145. serve space in a buffer to save this information SMARTALLOC simply stores the pointer to the compiled in text of the file name This works fine as long as the program does not overlay its data among modules If data are overlayed the area of memory which contained the file name at the time it was saved in the buffer may contain something else entirely when sm_dump gets around to using the pointer to edit the file name which allocated the buffer If you want to use SMARTALLOC in a program with overlayed data you ll have to modify smartall c to either copy the file name to a fixed length field added to the abufhead structure or else allocate storage with malloc copy the file name there and set the abfname pointer to that buffer then remember to release the buffer in sm_free Either of these approaches are wasteful of storage and time and should be considered only if there is no alternative Since most initial debugging is done in non overlayed environ ments the restrictions on SMARTALLOC with data overlaying may never prove a problem Note that conventional overlaying of code by far the most common form of overlaying poses no problems for SMARTALLOC you need only be concerned if you re using exotic tools for data overlaying on MS DOS or other address space challenged systems Since a C language constant string can actually be written into most C compilers generate a unique copy of each string used in a module even if the same
146. source code simply run make setup again Once Bacula is built you can run the basic disk only non root regression test by entering make test This will run the base set of tests using disk Volumes If you are testing on a non Linux machine several of the of the tests may not be run In any case as we add new tests the number will vary It will take about 1 hour and you don t need to be root to run these tests I run under my regular userid The result should be something similar to Test results auto label test OK 12 31 33 backup bacula test OK 12 32 32 bextract test OK 12 33 27 bscan test OK 12 34 47 bsr opt test OK 12 35 46 compressed test OK 12 36 52 compressed encrypt test OK 12 38 18 concurrent jobs test OK 12 39 49 data encrypt test OK 12 41 11 encrypt bug test OK 12 42 00 fifo test OK 12 43 46 backup bacula fifo OK 12 44 54 differential test OK 12 45 36 four concurrent jobs test OK 12 47 39 four jobs test OK 12 49 22 incremental test OK 12 50 38 query test OK 12 51 37 recycle test OK 12 53 52 restore2 by file test OK 12 54 53 restore by file test OK 12 55 40 restore disk seek test OK 12 56 29 six vol test OK 12 57 44 span vol test OK 12 58 52 100 sparse compressed test OK 13 00 00 sparse test OK 13 01 04 two jobs test OK 13 02 39 two vol test OK 13 03 49 verify vol
147. stems there is no definition for socklen_t so it is made in this file If your system has struc ture alignment requirements check the definition of BALIGN in this file Currently all Bacula allocated memory is aligned on a double boundary If you are having problems with Bacula s type definitions you might look at src bc_types h where all the types such as uint32_t uint64_t etc that Bacula uses are defined 87 Implementing a Bacula GUI Interface General This document is intended mostly for developers who wish to develop a new GUI interface to Bacula Minimal Code in Console Program Until now I have kept all the Catalog code in the Directory with the exception of dbcheck and bscan This is because at some point I would like to add user level security and access If we have code spread everywhere such as in a GUI this will be more difficult The other advantage is that any code you add to the Director is automatically available to both the tty console program and the GNOME program The major disadvantage is it increases the size of the code however compared to Networker the Bacula Director is really tiny GUI Interface is Difficult Interfacing to an interactive program such as Bacula can be very dif ficult because the interfacing program must interpret all the prompts that may come This can be next to impossible There are are a number of ways that Bacula is designed to facilitate this The Bacula network p
148. t in a print statement e g if ua gt verbose amp amp del num_del 0 bsendmsg ua _ Pruned d s on Volume s from catalog n del num_del del num_del 1 Job Jobs mr gt VolumeName Leave a certain amount of debug code Dmsg in code you submit so that future problems can be identified This is particularly true for complicated code likely to break However try to keep the debug code to a minimum to avoid bloating the program and above all to keep the code readable Please keep the same style in all new code you develop If you include code previously written you have the option of leaving it with the old indenting or re indenting it If the old code is indented with 8 spaces then please re indent it to Bacula standards If you are using vim simply set your tabstop to 8 and your shiftwidth to 3 28 Tabbing Tabbing inserting the tab character in place of spaces is as normal on all Unix systems a tab is converted space up to the next column multiple of 8 My editor converts strings of spaces to tabs automat ically this results in significant compression of the files Thus you can remove tabs by replacing them with spaces if you wish Please don t confuse tabbing use of tab characters with indenting visual alignment of the code Don ts Please don t use strcpy O strcat strncpy strncat sprintf snprint O They are system dependent and un safe These should b
149. t is essential to keep the size of this record to an absolute minimum At the same time this table must contain all the information or pointers to the information about the file and where it is backed up Since a file may be backed up many times without having changed the path and filename are stored in separate tables This table contains by far the largest amount of information in the Catalog database both from the stand point of number of records and the stand point of total database size As a consequence the user must take care to periodically reduce the number of File records using the retention command in the Console program Job Column Name Data Type Remark Jobld integer Primary Key Job tinyblob Unique Job Name Name tinyblob Job Name PurgedFiles tinyint Used by Bacula for purging retention periods 53 Type binary 1 Job Type Backup Copy Clone Archive Migration Level binary 1 Job Level ClientId integer Client index JobStatus binary 1 Job Termination Status SchedTime datetime Time date when Job scheduled StartTime datetime Time date when Job started EndTime datetime Time date when Job ended JobTDate bigint Start day in Unix format but 64 bits used for Retention period VolSessionId integer Unique Volume Session ID VolSessionTime integer Unique Volume Session Time JobFiles integer Number of files saved in Job
150. t you release the Modified Version under precisely this License with the Modified Version filling the role of the Document thus licensing distribution and modification of the Modified Version to whoever possesses a copy of it In addition you must do these things in the Modified Version A Use in the Title Page and on the covers if any a title distinct from that of the Document and from those of previous versions which should if there were any be listed in the History section of the Document You may use the same title as a previous version if the original publisher of that version gives permission B List on the Title Page as authors one or more persons or entities responsible for authorship of the modifications in the Modified Version together with at least five of the principal authors of the Document all of its principal authors if it has fewer than five unless they release you from this requirement State on the Title page the name of the publisher of the Modified Version as the publisher Preserve all the copyright notices of the Document Add an appropriate copyright notice for your modifications adja cent to the other copyright notices Include immediately after the copyright notices a license notice giving the public permission to use the Modified Version under the terms of this License in the form shown in the Addendum below Preserve in that license notice the full lists of Invariant Section
151. table contains one entry for each permanent counter defined by the user Version Column Name Data Type Remark VersionId integer Primary Key The Version table defines the Bacula database version number Bac ula checks this number before reading the database to ensure that it is compatible with the Bacula binary file BaseFiles Column Name Data Type Remark Baseld integer Primary Key BaseJobld integer Jobld of Base Job JoblId integer Reference to Job Fileld integer Reference to File FileIndex integer File Index number 59 The BaseFiles table contains all the File references for a particular JobId that point to a Base file i e they were previously saved and hence were not saved in the current Jobld but in BaseJobId under Fileld FileIndex is the index of the file and is used for optimization of Restore jobs to prevent the need to read the Fileld record when creating the in memory tree This record is not yet implemented MySQL Table Definition The commands used to create the MySQL tables are as follows USE bacula CREATE TABLE Filename FilenameId INTEGER UNSIGNED NOT NULL AUTO_INCREMENT Name BLOB NOT NULL PRIMARY KEY Filenamela INDEX Name 30 CREATE TABLE Path PathId INTEGER UNSIGNED NOT NULL AUTO_INCREMENT Path BLOB NOT NULL PRIMARY KEY Pathld INDEX Path 50 J CREATE TABLE File FileId INTEGER UNSIGNED NOT NULL AUTO_INCREMENT
152. test OK 13 04 56 weird files2 test OK 13 05 47 weird files test OK 13 06 33 migration job test OK 13 08 15 migration jobspan test OK 13 09 33 migration volume test OK 13 10 48 migration time test OK 13 12 59 hardlink test OK 13 13 50 two pool test OK 13 18 17 fast two pool test OK 13 24 02 two volume test OK 13 25 06 incremental 2disk OK 13 25 57 2drive incremental 2disk OK 13 26 53 scratch pool test OK 13 28 01 Total time 0 57 55 or 3475 secs and the working tape tests are run with make full_test Test results Bacula tape test OK Small File Size test OK restore by file tape test OK incremental tape test OK four concurrent jobs tape OK four jobs tape OK Each separate test is self contained in that it initializes to run Bacula from scratch i e newly created database It will also kill any Bacula session that is currently running In addition it uses ports 8101 8102 and 8103 so that it does not intefere with a production system Other Tests There are a number of other tests that can be run as well All the tests are a simply shell script keep in the regress directory For example the make test simply executes all non root tests The other tests are all_non root tests All non tape tests not requiring root This is the standard set of tests that in general backup some data th
153. the Director is corresponding with the Stor age daemon the Storage daemon can request Catalog services from the Director This convention permits each side to send commands to the other daemon while simultaneously responding to commands Any response that is of zero length depending on the context either terminates the data stream being sent or terminates com mand mode prior to closing the connection Any response that is of negative length is a special sign that normally requires a response For example during data trans fer from the File daemon to the Storage daemon normally the File daemon sends continuously without intervening reads How ever periodically the File daemon will send a packet of length 1 indicating that the current data stream is complete and that the Storage daemon should respond to the packet with an OK ABORT JOB PAUSE etc This permits the File daemon to ef ficiently send data while at the same time occasionally polling the Storage daemon for his status or any special requests Currently these negative lengths are specific to the daemon but shortly the range 0 to 999 will be standard daemon wide signals while 1000 to 1999 will be for Director user 2000 to 2999 for the File daemon and 3000 to 3999 for the Storage daemon The Protocol Used Between the Director and the Storage Daemon Before sending commands to the File daemon the Director opens a Message channel with the Storage daemon ide
154. they preserve the title of the Document and satisfy these conditions can be treated as verbatim copying in other respects If the required texts for either cover are too voluminous to fit legibly you should put the first ones listed as many as fit reasonably on the actual cover and continue the rest onto adjacent pages If you publish or distribute Opaque copies of the Document numbering more than 100 you must either include a machine readable Transparent copy along with each Opaque copy or state in or with each Opaque copy a computer network location from which the general network using public has access to download using public standard network protocols a complete Transparent copy of the Document free of added material If you use the latter option you must take reasonably prudent steps when you begin dis tribution of Opaque copies in quantity to ensure that this Transparent copy will remain thus accessible at the stated location until at least one year af ter the last time you distribute an Opaque copy directly or through your agents or retailers of that edition to the public It is requested but not required that you contact the authors of the Doc ument well before redistributing any large number of copies to give them a chance to provide you with an updated version of the Document 4 MODIFICATIONS You may copy and distribute a Modified Version of the Document under the conditions of sections 2 and 3 above provided tha
155. thin a job that are sent to the Storage dae mon for backup The Storage daemon ensures that this number is greater than zero and sequential Note the Storage daemon uses negative Filelndexes to flag Session Start and End Labels as well as End of Volume Labels Thus the combination of VolSessionId VolSessionTime and FileIndex uniquely identifies the records for a single file written to a Volume Stream While writing the information for any particular file to the Volume there can be any number of distinct pieces of information about that file e g the attributes the file data The Stream indicates what piece of data it is and it is an arbitrary number assigned by the File daemon to the parts Unix attributes Win32 attributes data compressed data of a file that are sent to the Storage daemon The Storage daemon has no knowledge of the details of a Stream it simply represents a numbered stream of bytes The data for a given stream may be passed to the Storage daemon in single record or in multiple records Block Header A block header consists of a block identification BB02 a block length in bytes typically 64 512 a check sum and sequential block number Each block starts with a Block Header and is followed by Records Current block headers also contain the VolSessionId and VolSessionTime for the records written to that block 65 Record Header A record header contains the Volume Session Id the Volume Session Tim
156. tion to access the database and create and delete tables TAPE_DRIVE is the full path to your tape drive The base set of regression tests do not use a tape so this is only important if you want to run the full tests Set this to dev null if you do not have a tape drive 98 TAPE_DRIVE1 is the full path to your second tape drive if have one The base set of regression tests do not use a tape so this is only important if you want to run the full two drive tests Set this to dev null if you do not have a second tape drive AUTOCHANGER is the name of your autochanger control device Set this to dev null if you do not have an autochanger AUTOCHANGER_PATH is the full path including the pro gram name for your autochanger program normally mtx Leave the default value if you do not have one TCPWRAPPERS defines whether or not you want the con figure to be performed with tcpwrappers enabled OPENSSL used to enable disable SSL support for Bacula com munications and data encryption Building the Test Bacula Once the above variables are set you can build the Makefile by enter ing config xxx conf Where xxx conf is the name of the conf file containing your system pa rameters This will build a Makefile from Makefile in and you should not need to do this again unless you want to change the database or other regression configuration parameter Setting up your SQL engine If you are using SQLite or SQL
157. ttp subversion tigris org Subversion Book http svnbook red bean com Subversion Clients For Windows users the TortoiseSVN package is awesome GUI UNIX client link http rapidsvn tigris org A nice KDE GUI client kdesvn Developing Bacula Typically the simplest way to develop Bacula is to open one xterm window pointing to the source directory you wish to update a second xterm window at the top source directory level and a third xterm win dow at the bacula directory lt top gt src bacula After making source changes in one of the directories in the top source directory xterm build the source and start the daemons by entering make and startit then in the enter console or gnome console to start the Console program Enter any commands for testing For example run kernsverify full Note the instructions here to use startit are different from using a production system where the administrator starts Bacula by enter ing bacula start This difference allows a development version of Bacula to be run on a computer at the same time that a production system is running The startit strip starts Bacula using a different set of configuration files and thus permits avoiding conflicts with any production system To make additional source changes exit from the Console program and in the top source directory stop the daemons by entering stopit then repeat the process 21 Debugging Probably the first thing to
158. ual or work Any member of the public is a licensee and is addressed as you You accept the license if you copy modify or distribute the work in a way requiring permission under copyright law A Modified Version of the Document means any work containing the Document or a portion of it either copied verbatim or with modifications and or translated into another language A Secondary Section is a named appendix or a front matter section of the Document that deals exclusively with the relationship of the publishers or authors of the Document to the Document s overall subject or to related matters and contains nothing that could fall directly within that overall subject Thus if the Document is in part a textbook of mathematics a Secondary Section may not explain any mathematics The relationship could be a matter of historical connection with the subject or with related matters or of legal commercial philosophical ethical or political position regarding them The Invariant Sections are certain Secondary Sections whose titles are designated as being those of Invariant Sections in the notice that says that the Document is released under this License If a section does not fit the above definition of Secondary then it is not allowed to be designated as Invariant The Document may contain zero Invariant Sections If the Document does not identify any Invariant Sections then there are none The Cover Texts
159. umber is re turned with a status of 3000 OK followed by a Length size line giving the length in bytes of the block data which immedi ately follows Blocks must be retrieved in ascending order but blocks may be skipped If a block number greater than the largest stored on the volume is requested a status of 3201 End of volume is returned If a block number greater than the largest in the file is requested a status of 3401 End of file is returned Read close session lt Ticket gt The read session with Ticket number is closed A read session may be closed at any time you needn t read all its blocks before closing it by John Walker January 30th MM SD Data Structures In the Storage daemon there is a Device resource i e from conf file that describes each physical device When the physical device is used it is controled by the DEVICE structure defined in dev h and typically refered to as dev in the C code Anyone writing or reading a physical device must ultimately get a lock on the DEVICE structure this controls the device However multiple Jobs defined by a JCR structure src jcr h can be writing a physical DEVICE at the same time of course they are sequenced by locking the DEVICE structure There are a lot of job dependent device variables that 47 may be different for each Job such as spooling one job may spool and another may not and when a job is spooling it must have an i o packet ope
160. ument is void and will automatically terminate your rights under this License However parties who have received copies or rights from you under this License will not have their licenses terminated so long as such parties remain in full compliance 10 FUTURE REVISIONS OF THIS LICENSE The Free Software Foundation may publish new revised versions of the GNU Free Documentation License from time to time Such new versions will be similar in spirit to the present version but may differ in detail to address new problems or concerns See http www gnu org copyleft Each version of the License is given a distinguishing version number If the Document specifies that a particular numbered version of this License or any later version applies to it you have the option of following the terms and conditions either of that specified version or of any later version that has been published not as a draft by the Free Software Foundation If the Document does not specify a version number of this License you may choose any version ever published not as a draft by the Free Software Foundation ADDENDUM How to use this License for your documents To use this License in a document you have written include a copy of the License in the document and put the following copyright and license notices just after the title page Copyright YEAR YOUR NAME Permission is granted to copy distribute and or modify this document under the terms of the GN
161. user changes anything inside the FileSet it will be detected and the new FileSet will be used This is particularly important when doing an incremental update If the user deletes a file or adds a file we need to ensure that a Full backup is done prior to the next incremental JobMedia Column Name Data Type Remark JobMediald integer Primary Key Jobld integer Link to Job Record Mediald integer Link to Media Record FirstIndex integer The index sequence number of the first file written for this Job to the Me dia LastIndex integer The index of the last file written for this Job to the Media StartFile integer The physical media tape file number of the first block written for this Job EndFile integer The physical media tape file number of the last block written for this Job StartBlock integer The number of the first block written for this Job EndBlock integer The number of the last block written for this Job VolIndex integer The Volume use sequence number within the Job The JobMedia table contains one entry at the following start of the job start of each new tape file start of each new tape end of the job Since by default a new tape file is written every 2GB in general you will have more than 2 JobMedia records per Job The number can be varied by changing the Maximum File Size specified in the Device resource This record allows Bacula to efficiently position c
162. won t go into the details of each of the above but the commands such as setgebug are foundinthescripts functions file Fordebugging theimportantlinetochangeisthe set set_debug 1 102 Then from the regress directory all regression scripts assume that you have regress as the current directory enter tests test name where test name should be the name of a test script for example tests backup bacula test Running a Single Test If you wish to run a single test you can simply cd regress tests lt name of test gt or if the source code has been updated you would do cd bacula bacula svn update cd regress svn update make setup tests backup to null Writing a Regression Test Any developer who implements a major new feature should write a regres sion test that exercises and validates the new feature Each regression test is a complete test by itself It terminates any running Bacula initializes the database starts Bacula then runs the test by using the console program Running the Tests by Hand You can run any individual test by hand by cd ing to the regress directory and entering tests lt test name gt 103 Directory Structure The directory structure of the regression tests is regress Makefile scripts to start tests scripts Scripts and conf files tests All test scripts are here 2222 All directories below this point are used fo

It comes in the night and sucks the essence from your

Contents

Download Pdf Manuals

Related Search

Related Contents