Home

A Specification Language for Program Auralization

1. 17 VDAP spec VDAP begin l function VDAP end 43 18 19 21 23 24 26 39 40 42 44 45 spec def gt spec def body gt spec command gt named command gt name tag list gt unnamed command set globals command gt global par list gt global par gt play command play list gt specdef id spec par list declarations begin id spec def body end id spec command spec def body spec command named command unnamed command name tag list unnamed command id name tag list id set globals command play command notify command dtrack command atrack command assign command loop command if command specdef use command VDAP call command turn command toggle command sync command set global par list global par list global par global par score const id device const id play play list pattern specifier play list 44 48 54 55 57 59 62 64 65 67 69 70 72 73 76 pattern specifier play pars tagged list tags score const id const id dotted id score tag device tag list device const id device tag notify command all selective label parameter pattern specifier amp amp play list play list id constant specdef use command VDAP call command pattern specifier play pars play list with tagged list tagged list tags tags score
2. id id gt id command where each subscripted zd above denotes a name and command denotes any event data or activity specification command Multiple commands can share a name Each id when used as the name of a command is treated as the name of a class The class so named consists of events data or activities specified in the commands named by id One command can be assigned multiple names This makes it easy to define classes that are not disjoint Consider the following example function_related notify rule function_call function_related notify rule function return data related dtrack a and b and c special data_related dtrack p and q The above three commands have been named to identify three classes Class function_related consists of events that correspond to function calls and return Another class named data_related consists of data items a b c p and q Yet another class named special consists of data items p and q The notion of a class can be used to model abstraction during program auralization For 5Classes defined in this section have no intentional relationship with the notion of classes in C and object oriented programming literature 27 example consider the auralization of tractor control software The programmer may like to group all the events into two classes One class consists of events that correspond to engine control Another class consists of events that correspond to the contro
3. 3 2 Occurrence space characterization Ideally we would like to be able to specify any auralization To do so we need to characterize the space of all possible occurrences that might arise during program execution Towards this end we select a three dimensional space using the orthogonal notions of position data and time Position refers to any identifiable point in a program For example in a C program beginning of a function call end of a function return start of a while loop start of a while loop body and start of a condition are all positions In general an identifiable point is any point in the program at which an executable syntactic entity begins or ends This implies that a position cannot be in the middle of an identifier or a constant In terms of a parse tree for a given program any node of the parse tree denotes a position For example the subscripted dot e denotes seven possible positions in the following assignment e Xe X 0 03 0 07 2 Data in a program refers to constants allowed in the language of the program being auralized and the values of program variables A data relationship is an expression consisting of constants variables and function calls Time refers to the execution time of the program It is measured in units dependent on the system responsible for the execution of the auralized program In a heterogeneous system time is measured in units agreed upon by all elements of the system As shown
4. sync par mmkeyword gt mmspec gt duration expression f then else command if condition then statement body if condition then statement body else statement body id actual par list id actual par list actual par actual par expression id list l id l actual parameter list turn on off device tag list on off toggle toggle source constant keyboard midi syncto sync to id program sync par list sync par list sync par sync par bufsize const noslow mmkeyword mmkeyword mmspec mm mmabs mmrel duration expression const duration expression duration factor duration expression duration factor 48 173 175 180 181 183 185 186 188 190 191 193 195 196 198 206 207 209 210 duration factor duration attribute declarations applicability apply list apply decl const declaration const list const val pair var declaration var decl list var type list id list type array declarator range list range expression gt gt o gt gt gt duration factor duration attribute duration expression f hlqlels applicability const declaration var declaration apply list apply list apply decl apply decl applyto tagged scope list const const list const val pair const list const val pair id c
5. LSL implementation consists of four software components i an LSL preprocessor ii an LSL editor iii an LSL sound library and iv an LSL graphic interface Below we outline the elements of the preprocessor the library and the graphic interface The editor has already been described above 37 LSL Library LSL specification j file MA LSLed Program to be ese auralized Electronic Computer keyboard keyboard Figure 6 LSLed in a programming environment 7 1 LSL preprocessor The task of the preprocessor is to instrument the program to be auralized with calls to sound library procedures The instrumentation is guided by the LSL specification In the absence of any specifications the preprocessor may be required to process LSL commands embedded within the program The preprocessor takes an LSL specification and a program P as input and produces the instrumented version Pr of P Py is then compiled and linked to the library routines using the traditional C compiler and linker This generates an object program with auralization commands compiled in The following steps provide a high level view of the sequence of actions taken by the preprocessor to transform P into Py 1 Parse each source file F that is input to the preprocessor This results in a parse tree T for source file F whose nodes are typed Using the type information it is possible to examine a node of T and determine what syntactic construct it
6. M Cohen Extending the notion of a window system to audio IEEE Computer 23 8 66 72 1990 15 R Sethi Programming Languages Concepts and Constructs Addison Wesley Publish ing Company Reading MA 1989 16 Symantec Think C User Manual chapter The Editor Symantec Corporation Cuper tino CA 1991 17 T Thompson Keynote a language and extensible graphic editor for music Computing Systems 3 2 331 358 Spring 1990 18 E S Yeung Pattern recognition by audio representation of multivariate analytical data Analytical Chemistry 52 7 1120 1123 June 1980 42 9 LSL Syntax Conventions The syntax of LSL is described below using a modified form of BNF 1 Nonterminals are in italics keywords in teletype font and lexical symbols in bold font Alternates of a nonterminal are separated by the symbol 1 lsl spec begin auralspec spec module list end auralspec 2 spec module list spec module list spec module spec module 5 spec module specmodule id program td list global interaction list declarations spec def list VDAP list begin id spec def body end id 6 program id list gt external ext id list 8 global interaction list gt global interactions global interaction list 10 global interactions interact id id list 11 interact id import export 13 spec def list gt spec def spec def list 15 VDAP list VDAP spec VDAP list
7. an identifier exported by another module by explicitly importing it using the import declaration All program variables used in an specdef or a specmodule body must be specified as externals Program identifiers global to a VDAP definition need not be declared However all such identifiers must be declared in the context wherein VDAP will be placed and compiled by the C compiler A VDAP specification must be a valid C function when using LSL C The formal and actual parameters must match in number and type between a specifi cation definition and its use All matching begins and ends must match in the identifiers that follow the correspond ing keyword Thus for example a begin gear which matches with an end change will be flagged as a warning because gear and change do not match LSL has default values for various parameters such as metronome channel and instru ment The expression in a relative timed event must evaluate to a positive integer or else a run time warning is issued A relative timed event is ignored if it occurs after program execution terminates A file or function specified in a scope tag must exist for the program to be auralized While monitoring an activity or data tracking will terminate upon program termina tion if the start event occurs after the terminating event An expression in a range list must evaluate to an integer and must not contain any variable names Subscript expressi
8. be affected by this command During program execution the auralization state can be toggled using the source specified in the command For example if the middle C on a MIDI keyboard is the toggle source tapping the middle C once after program execution begins turns the sound off Tapping it again turns it on Input from the toggle source is processed only when an auralized event occurs When such an event occurs an LSL library routine is invoked to check for a pending toggle request If a request is pending the auralization state is switched to OFF if it is ON or to ON if it is OFF A program may contain both turn and toggle commands A turn might change the auralization state to off only to be switched back to on by a toggle This is certainly one useful scenario Note that whereas turn commands are placed into the code prior to compilation and do not provide the user any control after compilation the toggle command permits dynamic changes to the auralization state The toggle default in LSL is the space bar on the computer keyboard Thus even when no toggle is specified in a program auralization state may be toggled using the space bar Regardless of the auralization state note values are generated and sent to the library routine responsible for playback It is this library routine that decides based on the current auralization state if the received notes are to be played or not In the metronome sync mode all notes emitted are buffered
9. const id device const id score tag const id mm mmspec mode specifier constant dotted id id keysig timesig device const id device tag list device const id device tag const id chan inst notify all selective label parameter event specifier sound specifier scope specifier all selective e label label list 45 78 80 82 84 91 93 94 96 98 100 101 103 105 107 108 label list event specifier connector event instance list instance sound specifier scope specifier tagged scope list tagged scope scope tag scope tagid list scope tagid dtrack command atrack command scope specifier label list id id event specifier connector event event and or rule id rule id instance list instance instance list assertion l condition rtime expression after event event specifier event first instance list amp amp instance instance string using play list in tagged scope list tagged scope list and tagged scope tagged scope scope tag scope tagid list filename func scope tagid list scope tagid scope tagid selector string dtrack dtrack id list start event spec term event spec sound specifier scope specifier atrack start event spec term event spec sound specifier 46 109 111 start event spec term event spec
10. in Figure 2 a three dimensional space is used for specifying occurrences in LSL Two kinds of occurrences are distinguished events and activities LSL allows an arbitrary combination of data relationships positions and time to specify an event or an activity associated with program execution 3 3 Sound space characterization The sound space is characterized by sound patterns comprised of notes durations play styles and instruments Notes of arbitrary durations can be combined to form sound patterns Each note can be associated with one of several play styles and with an arbitrary instrument For example a note can be played staccato on a piano with a specified volume Combining notes in various ways gives rise to a domain consisting of an infinity of sound patterns Digitized sound such as human voice is considered a sound pattern 3 4 Programming language independence The second requirement stated above is significant as we want LSL to be usable by program mers regardless of their preference for one or the other programming language Adherence to this requirement has produced a language which in the strict sense should be considered as a meta language One can therefore adapt LSL to specific programming languages as described later In all examples below we use a C 12 adaptation of LSL termed LSL C 4 Features and syntax of LSL The salient features of LSL are reviewed next Details of LSL syntax and semantics appear in the Ap
11. in a special playback buffer maintained by the library routine The buffered notes are removed from the buffer when their turn comes for playback This is determined by the current metronome setting When playback resumes due to a toggle or a turn changing the auralization state to on the notes are played back in accordance with 26 the metronome setting In program sync mode notes received by the library routine are discarded if playback is turned off 4 10 Event data and activity classes An event class consists of one or more events A notify command specifies one or more events which may occur at several positions inside a program and several times during program execution Events specified in one or more notify commands constitute an event class Similarly a data class is a collection of one or more variables A dtrack command specifies one or more variables to be tracked Variables specified in one or more dtrack commands constitute a data class An activity class is defined similarly with respect to activities specified in one or more atrack commands A class that consists of at least two elements of different types e g event and activity or event and data or data and activity is known as a mixed class It is possible for a user to define each of the above classes in an LSL specification This is done by naming one or more notify dtrack and atrack commands Any of these three commands can be named using the following syntax
12. is an association of sound patterns in S to occurrences in E Such a mapping is specified as a set of pairs e s where e E and s S The term el sl Occurrence space Sound pattern space Figure 1 A domain based view of program auralization The ASPEC in this example consists of four occurrence and sound pattern pairs as shown e 5 are elements of the specified mapping program auralization for a given program P refers to the set e1 51 2 s2 n Sn where each e s 1 lt i lt n is an association of an occurrence to a sound pattern A language L for program auralization is a notation to specify any such mapping for any pro gram A mapping specified using L is referred to as auralization specification abbreviated as ASPEC We assume that specifications are always written with reference to a given though arbitrary program in some programming language Figure 1 illustrates this view of program auralization Note that an ASPEC is a many to many mapping Let e s be an element of an ASPEC for program P During the execution of P if each occurrence e is identified by a sound pattern s we say that the pair e s has been realized An ASPEC for program P is considered realized if all its elements are realized for all executions of P We say that an implementation of L for programs in a given programming language PL is correct if each ASPEC written in L for any program P written in PL is realized
13. makefile For programs that need to be auralized calls to the C compiler may be replaced by calls to 1p Recall from Section 4 that applicability constraints may be used to restrict the scope of LSL specifications within a source file The above algorithm may perform multiple passes over the parse tree The number of passes over the program parse tree T is equal to the number of times notify and dtrack commands are encountered while interpreting an LSL specification Development of a more efficient single pass algorithm for auralization based on a given LSL specification is an open problem Further the preprocessor may not always succeed in performing correct auraliza tion due to pointers and aliases The problem can be resolved using labels or scope specifiers in the LSL specification to inform the preprocessor where to look for an event or data item 7 2 Graphic interface A graphic interface is an integral part of an LSL environment The interface enables one to specify auralization requirements without having to learn LSL When using this interface a user uses pull down and pop up menus that guide the development of specifications These specifications are transformed into LSL syntax by the interface and saved in a specification file just as a user would do if LSL was used directly Once the specification is complete the interface can be instructed to invoke the LSL preprocessor and the C compiler to instrument and auralize a program The inte
14. represents During the construction of T identify any LSL commands Replace each command by an appropriate call to a sound library procedure This call now becomes a part of T 2 Begin parsing the LSL specification if one has been supplied If there is no LSL specifi cation then go to step 3 Otherwise for each notify or dtrack command encountered say C do the following a Traverse T and identify syntactic constructs that match the event specification in C To each syntactic construct S so identified add an appropriate call to the sound library procedure that will generate the sound specified in C This step updates T 38 3 The updated parse tree say T contains new nodes that represent calls to sound library function calls Deparse T by traversing it and generating a source file F 4 Invoke the C compiler available in the environment with input F This will result in the object version of F 5 Delete F In a UNIX environment the above sequence of steps is initiated by the following com mand lp spec aspec l file icons c Assuming that icons c contains a function named main the above command will result in a file named a out containing the executable auralized version of icons c The auralization will be done using the LSL specifications contained in the file aspec l For programs spread over several files makefiles 10 and the make command can be used The 1p command can be embedded in the
15. scope specifier 113 115 117 118 120 122 125 126 128 129 131 133 134 136 137 139 ext id list dtrack id list dtrack id init value capture specifier mode specifier assign command selector element selector expression list loop command for loop step expression while loop statement body f command when event specifier scope specifier until event specifier ext id list l id l id dtrack id list and dtrack id dtrack id l id init value capture specifier scope specifier init l expression capture id mode continuous mode discrete mode sustain selector expression id id element selector expression list expression list expression expression for loop while loop for id expression to expression step expression statement body step expression while condition do statement body begin spec def body end spec command if then command 47 141 142 143 145 147 143 150 151 152 154 155 157 158 160 162 166 169 170 if then command gt if then else commane gt specdef use commane actual par list gt actual par spec par list gt VDAP call commane turn command gt on off gt toggle command gt toggle source gt sync command sync to id gt sync par list gt
16. system dependent ticks each tick being the smallest unit by which time could be incre mented Thus any expression using time can be used as a timed event As an example 23 suppose that the gear_change function must be invoked in a program in less than 60 seconds after the program execution begins It is desired to playback variable bad_program if this condition is not satisfied The following notify illustrates how to write this specification in LSL notify rule function_call gear_change and assertion time lt sectotick 60 using bad_program mode discrete In the above example sectotick is an LSL predefined function to convert seconds to ticks Notice that the expression time gt sectotick 60 is a valid way to specify an event as described earlier while discussing the syntax of notify It is often required to specify time relative to the occurrence of some event This can be done in LSL using relative timed events as shown below rtime lt expression gt after lt event specifier gt Consider the use of this mechanism in the following example for tracking an event dtrack when rtime sectotick 30 after rule function call missile launch until rule function return target_hit using missile in_motion The above dtrack can be read as Begin tracking 30 seconds after the function missile_launch has been called and terminate tracking when the function target_hit returns The tracking sound is defined by the LSL varia
17. Note Indicates one or more play styles inst Note pattern Specifies which instrument is to play mm Pattern Metronome setting This is applicable only to patterns Notes not part of a pattern are played for a duration de termined by global metronome setting A metronome setting specified for a pattern takes priority over any global setting only while this pattern is played ptime Note pattern Specifies the exact time in seconds to play the note or a pattern t MIDI is an acronym for Musical Instrument Digital Interface 15 Table 3 Sample note values using LSL duration attributes Note value Attribute combinations Quarter note q or hh Eight note hq Sixteenth note hhq or qq Thirty second note hhhq Sixtyfourth note hhhhq or ss Dotted half note h q Dotted quarter note qthq Dotted eigth note hqthhq value The duration attributes can be multiplied or added to get dotted quarter note and other fractions of note values For example hq read as half of quarter denotes an eight note hhq read as half of half of a quarter denotes a sixteenth note Table 3 lists sample note values and the corresponding attribute combinations Various rests could be obtained using the attribute combinations shown in Table 3 with the letter R For example R hq hhq denotes a dotted eight rest Duration can be specified for a chord by a single duration attribute For example C4E4G4 q denotes a c
18. Pre or user defined functions are used to manipulate values of type ksig Constants of type ksig are enclosed inside double quotes and can be assigned to variables of the same type A key signature could be predefined or user defined A predefined key signature consists of two parts a key name and a modifier 2Throughout this work data of type voice refers to digitized sound Thus for example both digitized voice and digitized guitar sound are characterized as voice data 11 begin auralspec specmodule myprog auralize This module contains specifications to auralize myprog procedure ap Applicability constraints if any come here i Declarations for variables global and external to this module ae specdef specdef 1 parameters Declarations of parameters local variables and functions begin specdef_1 end spec specdef_1 spec def spec def 2 parameters Declarations of parameters local variables and functions begin specdef_2 end specdef_2 specdef specdef n parameters Declarations of parameters local variables and functions begin specdef_n end specdef_n begin myprog_auralize Specifications for module myprog_auralize end myprog_auralize Other module specifications end auralspec Figure 3 Structure of an LSL specification containing one module 12 Examples of key names are Eb denoting E flat and C denoting C sharp Modifiers could be ma
19. Preliminary Report On Design Rationale Syntax and Semantics of LSL A Specification Language for Program Auralization David B Boardman Aditya P Mathur August 30 1993 Abstract The need for specification of sound patterns to be played during program execu tion arises in contexts where program auralization is useful We present a language named LSL Listen Specification Language designed for specifying program auraliza tion Specifications written in LSL and included in the program to be auralized are preprocessed by an LSL preprocessor The preprocessed program when compiled and executed generates MIDI or voice data sent through a MIDI interface to a synthesizer module or via audio channels to an audio processor which transforms the notes or voice into audible sound LSL has the generality to specify auralization of a variety of occurrences during program execution It derives its broad applicability from its few generic elements that when adapted to any procedural programming language such as C C or Ada enable the writing and use of LSL specifications for auralizing sequential parallel or object oriented programs in that language We view LSL as a useful tool for building general purpose multimedia applications and for research in program auralization This work was supported in part by an educational supplement from the National Science Foundation No CCR 9102311 and 9123502 CDA The authors are with Software Engineerin
20. also be controlled using lt start event gt and lt term event gt Start and termi nating events are specified respectively using the when and until clauses A few examples of dtrack use appear below 1 dtrack speed will track variable speed using an initial value of 0 and default sound parameters such as note pitch and volume 2 dtrack crash init false will track crash assuming an initial value of false 3 dtrack x capture x reset will track x after capturing its initial value at the assign ment labelled by the LSL label z_reset 4 dtrack mouse and color using color_ mouse_melody amp mouseval amp colorval will track variables mouse and color using a user defined function named color_mouse_melody with two parameters 5 dtrack speed when speed gt 65 until x lt 65 mode continuous will begin tracking speed whenever its value exceeds 65 and will stop tracking it immediately after its value becomes equal to or less than 65 Tracking will resume if the start event occurs again The discreet mode can be used to avoid resumption of tracking of speed 4 5 Activity monitoring An activity is a sequence of actions between two events An activity begins at the occurrence of an event and ends at occurrence of a later event As mentioned earlier start and termi nation of program execution are considered as events LSL allows specification of tracking arbitrary activities using the atrack command given below 22 Sound S
21. alues of 0 to 87 Other notes values may be obtained using the predefined func tion inttonote Other predefined functions are listed in Table 6 A rest is treated as a silent note with duration spec ified by a duration attribute Set of pairs of values denoting a time signature The first element in the pair specifies the beat structure i e the number of beats per measure The second el ement is the note value that corresponds to one beat The beat structure could be complex as explained in the text Set of k tuples of pitch values The set may be speci fied using abbreviations such as Eb minor to indicate the key of Eb minor or by enumerating all pitches re gardless of their specific position on a keyboard as in the example Set of note and or chord patterns consisting of zero or more notes or chords Set of digitized voice patterns A variable of this type can be set to point to a memory or disk file containing a digitized voice pattern Set of file names File extensions are interpreted c is for C program files v for digitized voice files Any digitized sound in a suitable format e g AIFF 3 14 Table 2 Attributes in LSL Code Applicability Description f Note Indicates a full note h Note Indicates a half note q Note Indicates quarter note e Note Indicates eight note s Note Indicates sixteenth note chan Note pattern Specifies the MIDI channel on which to play play
22. and played through the same speakers A user may organize the audio playback hardware differently 6 1 LSL editor In principle it is possible to use any of the well known text editors such as gnuemacs or vi to edit an LSL specification However in several cases such editing may be inconvenient specially so for one not familiar with the piano keyboard or music terminology For this reason we propose a special editor named LSLed for preparing LSL specifications LSLed has features similar to those found in syntax directed editors on personal com puters such as the Think C editor on the Macintosh 16 computer LSLed is most useful when editing sound related commands For example suppose that variable named drag icon is declared to be of type pattern We wish to assign a sound pattern to drag icon One way to do so would be to type in the pattern using the notation described earlier in Section 4 1 Thus one may type in the following text drag icon C2D2E2F2G2A2B2C3 We identify two problems with this approach First an individual may not be familiar with the notation e g one who asks the question What is C2 Second one may be familiar with the notation but would like to experiment with various sounds and just play the sound pattern on an electronic keyboard instead of converting it to the textual notation LSLed helps overcome these problems During editing LSLed could be in one of two modes text and aural When in text mo
23. as compared to editing the code which might require careful passes through various source files Examples such as the ones above led us to consider designing a language for specifying auralizations We note the pioneering work in the design of languages for music 13 17 The main purpose of these languages was to specify music In their present form these languages are not suited to the auralization tasks mentioned above 3 Basic definitions and LSL requirements Based on the perceived need for a specification language we set forth the following idealized requirements for LSL 1 Generality It should be possible to specify any auralization using LSL 2 Language independence It should be possible to use LSL with the commonly used programming languages such as C Ctt Ada Pascal and Fortran Below we define basic terms and introduce concepts that help us formalize the above goals Our formalization brings reality to the above requirements LSL satisfies the requirements with respect to this formalization 3 1 ASPECs and realizations To be able to design a language that can specify all possible auralizations we need a quan tification of two domains Let E be the domain of all those occurrences during the execution of any program that one may wish to auralize The nature of such occurrences is discussed below Let S be the domain of all possible sound patterns that may be associated with each element of E A mapping from E to S
24. ast three different ways a software developer could use LSL An expert in the use of LSL could write an LSL specification or add LSL commands to the program itself and preprocess the program as shown in Figures 5 a and b Alternately as in Figure 5 c a graphical interface could be used to ease the task of specifying auralization This interface accepts and translates user commands to LSL specifications Specifications so generated are internal to the interface and hidden from a novice user LSL Library i LSL Preprocessor _ Compile Link LSL specification gt Program gt Object code a LSL Library Y Program Graphic interface for specifying Compile Link auralization Object code b LSL Library i LSL Preprocessor p 3 gt Compile Link Object code Program with LSL commands c Figure 5 Use of LSL in a programming environment 39 The result of using any of the above methods is an object program instrumented with calls to procedures from the LSL sound library When the instrumented program is executed these calls generate sound data that is directed to a sound module via MIDI The sound module output is sent through an amplifier to a speaker Voice is sent via an audio processor and amplifier to the speakers Non voice and voice data are mixed at the amplifier
25. ated differently 6 A constant can be an integer or a string An integer is a sequence of digits A string is a sequence of characters enclosed within double quotes As a constant can be interpreted in a variety of ways in LSL we provide below a complete grammar for constants 1 constant gt integer string time sig integer gt digit string gt char sequence char sequence note sequence key sig file name function name 10 note sequence note id note sequence attribute sequence note sequence 36 attribute 37 duration duration tagged value list 14 note gt note generic note modifier 15 note generic gt cldle f glalbl r C D E F G A B R 31 note modifier gt flat sharp octave 32 flat sharp gt b 34 octave 0 8 35 attribute sequence attribute X simple duration duration expression 39 simple duration f h ql el s ptime integer 51 45 48 90 92 93 62 63 64 66 67 68 69 70 Interpretation of a string is context dependent to a variable of type pattern the string cmajor C5 denotes a sequence of notes consisting of the value of the variable cmajor followed by the note C5 The same string when used in the context file cmajor C5 denotes a file name cmajor C5 Notes enclosed in parentheses such as in G3 C4E4G4 C5 are treated as forming a blocked chord The string hello resu
26. ble missile_in_motion Thus using a combination of time and rtime one may specify a variety of timed events for auralization 4 7 Playback synchronization Synchronization mode controls the playback of notes during program execution There are two such modes program or metronome In the program mode playback is synchronized to the program In the metronome mode it is synchronized to a global metronome The syncto command is used for setting the synchronization mode The syntax of syncto is syncto lt sync to gt The lt sync to gt parameter can be program or mm for synchronization with respectively program execution or a global metronome Multiple syncto commands may be placed in an LSL specification to alter the synchronization mode In the metronome mode a buffer holds the notes generated by the executing program When this buffer is full and the program attempts to send a note for playback the playback routine does not return control to the program until the received note can be buffered This may slow down program execution To avoid this situation in metronome mode one may use 24 the noslow parameter such as in the command syncto mm q 120 noslow When the noslow parameter has been specified playback routine discards notes that are received when the buffer is full This could cause some events or data tracking to pass by unauralized The size of the playback buffer can be controlled by setting the bufsize parameter such as in syn
27. cto mm 120 bufsize 1000 which specifies a buffer size that will hold at least 1000 notes 4 8 Assignments loops and conditionals An assignment command has the general syntax shown below identifier lt subscript_list gt lt expression gt where identifier is the name of a variable Expression is any valid expression that evaluates to the type of the identifier on the left of the assignment lt subscript_list gt is a list of subscripts used for selecting array elements if the identifier denotes an array Loops can be formulated in an LSL specification using the for and while constructs Syntax of these two constructs is given below for lt for_index gt lt init_expression gt to lt final expression gt step lt step_expression gt lt spec_sequence gt while lt condition gt do lt spec_sequence gt The semantics of each of the above commands are similar to that of the for and while statements in Pascal All expressions in a for command must evaluate to integers A lt spec_sequence gt is a sequence of zero or more LSL specification commands Conditional commands are provided in LSL for selectively specifying an auralization The syntax of a conditional command appears below Its semantics are similar to that of the if statement in Pascal if lt condition gt then lt spec_sequence gt else lt spec_sequence gt 4 9 Controlling auralization state During execution an auralized program can be in one of two aural
28. cution or properties of data to sound A survey of the use of sound in understanding program behavior or analyzing data collected from experiments or generated by a program is reported by Francioni and Jack son 7 It is believed and has been demonstrated in a few cases that the use of sound can enhance program understanding Yeung 18 proposed the use of sound to recognize multi variate analytical data Francioni and Jackson 7 used program auralization to understand the runtime behavior of parallel programs Brown and Hershberger 4 auralized some of their animations generated using the Zeus animation system In his doctoral dissertation Edwards 6 built and evaluated a word processor with an audio interface for use by visually handicapped users Gaver 8 9 proposed the use of auditory icons for use as a part of Apple s interface on the Macintosh machines It has been noted by researchers that in most situations program output is visual Pro gramming environments and applications developed so far have attempted to use visual media to a great extent 2 Though research in the use of audio in workstations 11 14 has been on the rise audio remains a distant second to visual media However with the availability of low cost audio devices such as synthesizer modules and sound digitizers and the provision of sound generators in workstations audio is within reach of most PC and workstation users We have designed a language that simplifies th
29. de all input to LSLed is from the computer keyboard When in aural mode the source of input could be MIDI via an electronic keyboard or the computer keyboard When the above assignment to variable drag_icon is typed LSLed expects input from the computer keyboard until all characters up to and including the assignment operator have been typed It then adds a double quote to indicate the starting of a string and prompts the user to play the pattern on the electronic keyboard At this point the editor screen would display among other characters e drag_icon The LISTEN system currently uses Proteus 3 World from E mu Systems as the sound module The Audiomedia II card from Digidesign is used for voice sampling and playback Macintosh is a trademark of Apple Computer Inc 36 As the user plays the notes on a keyboard the textual version of each note appears following the double quotes On completion the user hits the return key indicating that the pattern has terminated To change any previously entered pattern the user merely places the cursor at the beginning of the pattern immediately following the double quotes and repeats the above process Textual version may also be edited by switching LSLed to textual mode This may be useful for adding or altering note parameters such as for example duration or play style The above process can be used to enter other types of sound related constants too The advantage of this approach i
30. e second part is the definition of a C function named crit_dist_track which has embedded LSL commands The entire LSL specification module appears below begin auralspec VDAP begin crit_dist_track int distval rockval This is a C function distval is a pointer to dist_remain and rockval is a pointer to rock_remain if distval gt crit_distance amp amp rockval lt crit_rock LSL begin play grave note with inst bass mode continuous chan 4 end ue else LSL begin turn off chan 4 end WF VDAP end specmodule emergency sound applyto filename dist_compute c const grave_note G2 begin emergency_sound dtrack dist_remain and rock_remain using crit_dist_track amp dist remain amp rock_remain end emergency_sound end auralspec 33 While processing the above LSL module the LSL preprocessor adds calls to the C func tion crit_dist_track together with the specified parameters immediately after each program statement that could possibly alter the value of dist_remain or rock_remain The preproces sor also translates the LSL commands inside a VDAP to C code The function so obtained is placed in suitable file for compilation by the C compiler The preprocessor does not check for the correctness of the VDAP Example 7 A recursive function named factorial is to be auralized in such a way that each call to factorial generates a note with pitch proportional to the depth
31. e task of specifying which occurrences during program execution are to be auralized and how The language is named Listen Specification Language abbreviated as LSL Listen is the name of our project to investi gate possible uses of sound in programming environments and software based applications The need for LSL its syntax use and current implementation status are described in the remaining sections Section 2 outlines the need for specifications of program auralizations and LSL Language requirements and the underlying rationale are presented in Section 3 Essential features of LSL are presented in Section 4 Examples illustrating the use of LSL in various applications are given in Section 5 Section 6 explains how LSL can mesh with a programming environment An implementation strategy for LSL is outlined in Section 7 Current status of LSL implementation is reported in Section 8 A summary and preliminary conclusions regarding the utility of LSL appear in Section 8 We include human voice in this context 2 The need for LSL Our research is concerned with the investigation of various uses of sound in programming environments Some generic questions we ask and seek answers to are listed below 1 How useful is sound in debugging programs 2 How useful is sound in program understanding 3 Can program auralization be used to improve the quality of simulations of various kinds such as telephone networks mechanical systems and biol
32. f events as in Example 2 except that loop body begins and ends that contain any one of the two labels loop_1 and loop_2 will be selected for auralization 19 Table 5 Keywords and codes for LSL event specifiers in C Category Event specifier Codet Event specifier Code Program start start end end Expression variable var assignment_expression aex conditional_expression cex Iteration iteration statement ist iteration_body_begin ibb iteration_body_end ibe while statement enter wse while_statement_exit WSX do_while dow for_statement_enter fre for_statement_exit frx while_body_begin wbb while_body_end wbe for body begin fbb for_body end fbe do_while_body_begin dbb do_while_body_end dbe Jump jump_statement jmp continue_statement cst break_statement bst return_statement rst goto_statement gst Selection selection_statement sst if statement ist if then part itp if_else_part iep switch_statement sst switch_body_begin sbb switch_body_end sbe Functions function call fne function_entry fne function_return fnr Event specifiers and their abbreviated codes can be used interchange ably to specify a rule in a notify statement 20 Table 6 Predefined functions in LSL Function name Mapping Description intton int note Converts an integer to a note Integers in the inclusive range 0 to 87 get converted to notes AO to C8 ntoint note int Converts a note to an integer nab
33. g Research Center and Department of Computer Sciences Purdue University W Lafayette IN 47907 Aditya P Mathur can be contacted at 317 494 7822 or via email at apm cs purdue edu David Boardman can be contacted via email at boardman cs purdue edu Contents 1 Introduction 5 2 The need for LSL 6 3 Basic definitions and LSL requirements 7 3 1 ASPECs and realizations alo pa Wok ei A e RRA ASAE ES ri 3 2 Occurrence space characterization o 02000202 ae 8 3 3 Sound space characterization oa ooa a a ee oe a 9 3 4 Programming language independence a a e 9 4 Features and syntax of LSL 9 4 1 Constants variables and types E DRAE 11 4 2 Sound pattern specification a A AA AAA 17 4 3 Event notification 26 ane ay EA OA a 17 AA Data tracking noeros co era ae he pe eee ele ee eS eee ed 21 4 5 Activity MONDIAL E y 22 46 Timed events A BE 23 4 7 Playback synchronization a IS EA we oh Eee 24 4 8 Assignments loops and conditionals o o a o a e a 25 4 9 Controlling auralization state yo o o oaa ee 6 e 25 4 10 Event data and activity classes o a 21 4 11 Embedding LSL commands ooh do AA o A 28 5 Examples of LSL C use 29 6 LSL in a programming environment 35 Ole Sed eS a a A aa AR AAA AA AS a 36 7 An LSL implementation outline 37 BL LSD Preprocessor Ss lala e A E E AAA A 2 Bets 38 12 Graphie Wer ace ne ei ia A AA ala 39 TS The LSL sound WBE AY AA ho Be id Bt ee a ee 40 8 Summary 40 Acknowled
34. gements 41 Appendix Syntax of LSL 9 LSL Syntax Conventions 10 Lexical Conventions 11 Static Semantics 43 43 50 53 List of Figures 1 A domain based view of program auralization The ASPEC in this example consists of four occurrence and sound pattern pairs as shown e s are elements of the specified mapping 8 Occurrence space characterization in LSD 0 10 Structure of an LSL specification containing one module 12 Sample activity patterns specifiable in LSL El E2 and E3 denote events Start and End denote the start and end of program execution 23 Use of LSL in a programming environment s a so o e a e 35 LSLed in a programming environment 0000 eae 38 List of Tables NOT BW Nel Primtive ty pee Voli ES oe eS ee Ee is 14 Attributes im LSL ad Sao ee e Be et ed eo a a eS 15 Sample note values using LSL duration attributes 16 Default values of run time parameters 17 Keywords and codes for LSL event specifiers in C 20 Predefined functions in LSL ak on Soe a 21 Language Dependent Terminals in LSL Grammar 53 1 Introduction The idea of using sound to understand program behavior and to analyze data using sound has been reported by several researchers By program and data auralization we refer to respectively the activity of mapping aspects of program exe
35. hord consisting of three quarter notes Notes and chords for which the duration is not specified explicitly as in E4 are played for a duration determined by implementation dependent default durations See Table 4 for various defaults Type constructor Values of primitive types can be combined together into an array The following sequence declares an array of measures each measure being a pattern Elements of an array can be accessed by subscripting Thus tclef_staff k 1 refers to the k 1 th element of tclef staff which is of type pattern const scoresize 25 var tclef_staff array 1 scoresize of pattern 16 Table 4 Default values of run time parameters Item Default value Metronome q 120 Key signature C major Time signature 4 4 Channel 1 Instrument code 1 Note duration q Play mode discrete for notify discrete for dtrack continuous for atrack Pitch C4 4 2 Sound pattern specification The play command is used to specify what sounds are to be generated when some part of a program is executed The general syntax of play is play lt playlist gt where lt playlist gt is a list consisting of one or more notes and patterns specified using con stants variables and function calls Key and time signatures are some of the parameters that may be specified Elements of lt playlist gt can be separated by a comma or a parallel sign An example of play command a
36. included to play digitized voice during program execution We have assumed that voice will be digitized using a suitable digitizer e g the Audiome dia II 5 card from Digidesign and stored as a sample in a file It is this sample that becomes a constant and can be assigned to a variable of type voice Voice can be used in note patterns by specifying variables of type voice Variables must be declared before use The following declaration declares body_begin and body_end to be of type note loop_begin loop_end and measure to be of type pattern var body_begin body_end note loop_begin loop_end measure pattern Note and rest values Attributes aid in specifying various properties of notes and patterns Perhaps the most common attribute of a note or a chord sequence is its duration For example E4 q denotes a quarter note whose duration will be determined by the time signature and the metronome 13 Table 1 Primitive types in LSL Keyword Sample values Description int note tsig ksig pattern voice file 20 or 76 E4b R 3 8 or 3 2 2 4 Eb minor C D E F G AB G3E3C4 done voice v Set of integers Set of notes not all of these may be played back in a particular implementation A subset of the notes is labelled starting at AO and going up to C8 as found on an 88 key piano keyboard These 88 notes cor respond to integer v
37. ization states ON or OFF In the ON state any sound data resulting from the occurrence of an auralized event is sent to the sound processor In the OFF state any such sound data is suppressed LSL provides two commands to dynamically alter the auralization state These are the turn and the toggle commands These commands have no effect when placed inside an LSL specification They may affect the auralization state when placed inside the auralized program Each note belonging to a chord counts as one note 25 Using turn is one way to switch sounds on or off turn on switches the sound on and turn off switches it off The command may be placed anywhere inside the auralized program Upon the start of program execution the auralization state is ON The turn command takes effect immediately after it is executed Sound channels can be switched off selectively by specifying the channel number as in turn off chan 4 switches off any sound on channel 4 Another way to turn the sound on or off is with the toggle command The syntax of toggle is given below toggle id lt toggle source gt constant where lt toggle source gt could be the MIDI or computer keyboard indicated respectively by the keywords midi and keysig The constant is a string containing the toggle note from the MIDI keyboard and the toggle key from the computer keyboard When specified id denotes the name of a class defined below of events activities and data items to
38. jor minor same as harmonic minor lydian ionian same as major mixolydian dorian aeolian phrygian and locrian Thus for example C minor and E phrygian are valid key signatures A user defined key signature is any enumeration of notes For example C D Eb G A is a key signature of a pentatonic scale The set of time signatures constitutes the type tsig Constants of type tsig are enclosed within parentheses A time signature consists of two parts the beat structure and the note that takes one beat For example 4 4 is a simple time signature indicating 4 beats to a measure with a quarter note of one beat in duration A more complex time signature is 3 2 2 8 which indicates a beat structure of 3 2 2 with an eigth note taking one beat A beat structure such as 3 2 2 indicates that the first measure is of 3 beats in duration followed by two measures each of 2 beats duration followed by a measure of 3 beats and so on Time signatures can be assigned to variables of the same type and manipulated by functions Type file is the set of file names A filename is specified by enclosing the name within double quotes Thus your_name_please v can serve as a file name The use of file names is illustrated through LSL examples below Note that we use a string of characters enclosed within double quotes in a variety of contexts It is the context that unambiguously determines the type of a string A special type voice has been
39. l of parapher nalia attached to the tractor e g a seeding device By simply using the event specification mechanism of LSL there is no way to explicitly incorporate these classes into an LSL speci fication The mechanism of naming a command as described above however does provide a convenient means for defining classes Once defined classes of events can be accessed at an abstract level using their names For example during the execution of an auralized program it is possible to interact with the LSL run time system and turn off the auralization of all events within a class It is also possible to request LSLed described in Section 6 1 below to provide a comprehensive list of classes and their individual elements Thus the use of classes enables a user to interact with an auralized program in terms of high level occurrences e g events instead of dealing with syntax based definitions 4 11 Embedding LSL commands LSL commands can be embedded in C programs inside comments The LSL preprocessor recognizes an LSL command embedding if the first token beginning with a letter immediately following the comment begin delimiter is LSL Immediately following the delimiter a sequence of LSL commands can be placed enclosed within the begin and end delimiters For an example of such an embedding see Example 6 on page 32 The LSL commands so embedded are translated to C code by the LSL preprocessor LSL commands such as play and notify ge
40. lts in an invalid assignment command when it appears on the right side of an assignment to a variable of type pattern Ambiguity may arise while defining a note sequence such as in cbb To avoid this the notes may be separated by at least one space character such as in cb b duration expression op gt key sig gt pre defined gt mode gt user defined gt time sig beat structure gt filename gt function name gt tagged value list tagged value gt play attribute tag duration expression op simple duration simple duration duration expression le pre defined user defined note mode major minor lydian ionian mixolydian dorian aeolian phrygian locrian note sequence beat structure int beat structure int int char char tagged value list tagged value play attribute tag constant chan play inst mm mm mmspec 92 Thus for example when assigned Table 7 Language Dependent Terminals in LSL Grammar Terminal Meaning Example from C 1 condition Conditional expression 1 lt y amp amp p gt q which evaluates to true or false l id Identifier drag_icon l expression An expression that evaluates min val x 2 to a value of type matching the type of the left side of the assignment in which it appears l function A function invoked for track Any C function defi ing one or more variable
41. n a large program spread over several source files The task could be simplified if one could formally specify the auralization mentioned above Such a specification and the program to be auralized could be preprocessed automatically to generate a source program with code inserted for the desired auralization Any change in auralization will then require a change only to the specification and not to the source of the parallel program As another example consider the task of debugging a distributed system that controls various functions of an automobile It is desired to auralize the program so that each call to a control procedure e g to procedure gear_change is identified by a suitable sound As above one may edit the source to achieve the desired effect Another alternative is to formally specify the above auralization requirement and use preprocessing to add the necessary code A more difficult situation arises when the value of one or more variables is to be monitored Suppose that we want to monitor the value of a variable x in a program Whenever this variable exceeds a predetermined value a sound should be generated Again one may use an editor to add a few lines of code at all places in the program where x has been defined An easier alternative appears to be to formalize the above auralization and preprocess it During further development of the software if one decides to monitor other variables the specification can be easily modified
42. nd e or ez are also events The scope of a notify may be restricted using the lt scope specifier gt In LSL C the scope can be restricted to one or more functions or files For example if an assertion is to be checked only inside function sort one may suitably restrict the scope to that function Labels can be used in conjunction with scope restrictions to specify arbitrarily small regions in a program The sound specifier is a variable name constant or a function call that specifies the intended auralization of the selected events Sample notify commands appear below 1 notify all rule while loop body begin using body_begin notify all rule while loop _body_end using body_end 18 2 notify selective label special_loop rule while_loop_body_begin using body_begin notify selective label special loop rule while_loop_body_end using body_end 3 notify all instance count and search x using count_or_search in func y g search report 4 notify all assertion x lt y p gt q using assertion failed 5 notify all rule conditional_expression and assertion odd x using cond_sound in filename myfile c Example 1 above specifies two event types namely the beginning and end of a while loop body using two general purpose syntactic specifiers It also indicates that all positions in the program where such events could occur are to be auralized Thus a C program auralized using
43. note nabove next_note C major end end for_loop end auralspec The above LSL C specification uses an LSL function to insert all file names into an array The for loop then steps through each file name to specify the desired auralization Successive notes are obtained using a predefined function named nabove which takes a note as an argument and returns the next note a full step above on a given scale a Example 6 This example illustrates how dtrack can be used with VDAP Value Dependent Aural Pattern to track arbitrary functions of program variables Suppose that it is desired to track the dynamic relationship between two variables named rock_remain and dist_remain rock_remain represents the number of rockets remaining to be fired and dist_remain the remaining distance to be travelled Each time a rocket is fired the count of remaining rockets reduces by 1 The distance remaining to be travelled is updated by some process in 32 the program It is desired to emit a continuous audible sound only while the remaining distance dist_remain is more than the critical distance crit_dist and the number of remaining rockets rock_remain is less than the critical rocket count crit_rock An LSL specification to achieve the above auralization consists of two parts One part is an LSL specification module containing a dtrack command This command specifies the use of a VDAP function named crit_dist_track for tracking dist_remain Th
44. o a variety of procedural languages In examples below we assume that programs being auralized have been coded in C The syntax of notify appears below notify lt all selective gt lt label parameter gt lt event specifier gt lt sound specifier gt lt scope specifier gt lt all selective gt specifies which subset of events selected by a notify are to be auralized Possible event codes are all and selective If selective is used one or more labels must be specified to indicate which events are to be selected lt event specifier gt specifies one or more events to be notified aurally There are five ways to specify an event One may specify a general syntactic entity a special syntactic entity an assertion a relative timed event and any combination of the above four Relative timed events are discussed in Section 4 6 other methods are described below Table 5 lists all event codes in LSL C For example while statement enter is an event specifier the corresponding event occurs once each time a while statement is executed The start and termination of program execution serve as events The expression x lt y serves as a special syntactic entity The associated event occurs whenever the expression x lt y is executed An assertion such as x y gt p q also specifies an event which occurs whenever the assertion evaluates to false If e and ez are two events specified using any of the above approaches then e and ez a
45. of recursion The first call to factorial should generate C4 successive calls should generate one note higher in the C major scale The return sequence from factorial should play the notes back in the opposite order The following LSL specification meets the above requirements begin auralspec VDAP begin fact_enter LSL begin note_to_play nabove note_to_play C major play note_to_play end ES fact_exit LSL begin note_to_play nbelow note_to_play C major play note_to_play end VDAP end specmodule fact_auralize var note_to_play note begin fact_auralize 34 note_to_play B3 The next note is C4 notify all rule function_entry factorial using fact_enter notify all rule function return factorial using fact_exit end fact_auralize end auralspec The above specification contains two VDAP functions namely fact_enter and fact_ezit Each call to and return from factorial results in respectively the execution of fact_enter and fact_exit resulting in the desired playback 6 LSL in a programming environment LSL is designed to fulfill complex demands for program auralization In a programming environment one may use LSL to auralize parts of a program during the testing and de bugging phase or auralize an application to meet auralization requirements stated at the beginning of the development cycle As shown in Figure 5 there are at le
46. ogical systems 4 How can sound be used to make programming environments usable by in particu lar the visually handicapped with the same ease as by individuals without such a handicap As mentioned above researchers have attempted to obtain answers to similar questions We believe that significant progress remains to be made before sound is used widely in various programming related tasks While investigating answers to the above questions we encountered a need for a general purpose mechanism to specify the auralization of programs In the absence of such a mech anism auralization is done by editing the source code and adding calls to library procedures that generate sound For example suppose that a parallel program to be executed on a ma chine with 1000 processors is to be auralized The auralization should be such that whenever an even numbered processor sends data a sound with timbre characteristic T is generated and whenever an odd numbered processor sends data a sound with timbre 75 is generated When so auralized a listener will be able to distinguish between data sends by even and odd numbered processors Assuming that one has access to a library routine gen sound to generate sound with a specific timbre one can edit the program and add calls with appro priate parameters to gen_sound at all places where a data send occurs This could become an inconvenient task if communication statements are distributed over many procedures i
47. ols Addison Wesley Publishing Company Reading MA 1986 A L Ambler and M M Burnett Influence of visual technology on the evolution of language environments IEEE Computer 22 10 9 22 1989 Apple Computer Inc Inside Macintosh Volume VI Addison Wesley Publishing Com pany Reading MA 1992 M H Brown and J Hershberger Color and sound in algorithm animation Computer 25 12 52 63 December 1992 Digidesign Inc Sound designer II User s Guide 1992 A D N Edwards Soundtrack An auditory interface for blind users Human Computer Interaction 4 1 45 66 1989 J M Francioni and J A Jackson Breaking the silence Auralization of parallel program behavior Technical Report TR 92 5 1 Computer Science Department University of Southwestern Louisiana 1992 W W Gaver Using sound in computer interfaces Human Computer Interaction 2 167 177 1986 W W Gaver The sonicfinder An interface that uses auditory icons Human Computer Interaction 4 1 67 94 1989 Computer Systems Research Group UNIX User s Reference Manual URM USENIX Association 4 1986 R Kamel K Emami and R Eckert Px Supporting voice in workstations EEE Computer 23 8 73 80 1990 B W Kernighan and D M Ritchie The C Programming Language Prentice Hall Englewood Cliffs NJ 1988 41 13 P S Langston Little languages for music Computing Systems 3 2 193 282 Spring 1990 14 L F Ludwig N Pincever and
48. ons that evaluate to a value outside the specified range are not allowed If both the initial value and the capture location are specified for a variable to be tracked LSL will attempt to satisfy both requirements Thus the variable will be initialized at an appropriate point during program execution Its value will also be captured as specified The value captured will override any previous value of the variable 54 13 14 The syntax of LSL allows for the naming of any command However only names of notify dtrack and atrack correspond to classes Naming of other commands is per mitted to allow referencing of commands while editing or reading an LSL specification Use of toggle may give rise to ambiguities at run time For example if the space key on the computer keyboard has been specified as a toggle source and the executing program requests for input data it is not clear if the space character should be treated as a toggle request or input to the program The user may avoid such ambiguities by selecting a toggle source that will not be required as input to the program Alternately the user may rely on the run time window based monitor to input toggle requests 59
49. onstant var var decl list var type list var decl list var type list id list type id id list id int note pattern voice file ksig tsig array declarator array range list of type range list range range expression expression expression addop term 49 212 214 220 oes 224 227 229 231 232 238 term factor condition cterm cfactor addop mulop unop relop logop term term mulop factor factor expression unop factor id id actual par list id const condition relop cterm cterm cterm logop cfactor cfactor expression condition not cfactor ll S gt amp amp 10 Lexical Conventions Using regular expressions 1 we define the lexical elements of LSL Comments are enclosed inside and Comments may not appear within a token A comment within another comment is not allowed char denotes any ASCII character One or more spaces separates tokens Spaces may not appear within tokens 50 4 An id is a sequence of letters or digits with the first character being a letter The underscore _ can be used in an identifier Upper and lower case letters are treated as being different in an id id gt _ letter letter digit _ letter gt a zA Z digit gt 0 9 5 A keyword may not be used as an id Upper and lower case are tre
50. ove note x ksig note Returns the note above the input in the given scale nbelow note x ksig note Returns the note below the input in the given scale naboveh note note Returns the note one half step above the input nbelowh note note Returns the note one half step below the input circlen ksig ksig Returns the next key signature in the circle of fifths Valid only for predefined key signatures circlep ksig ksig Returns the previous key signature in the circle of fifths Valid only for predefined key signatures sectotick int int Converts seconds to system dependent ticks 4 4 Data tracking Event notification consists of specifying one or more events and reporting them aurally during program execution There are applications wherein changes to values of variables need to be monitored It is certainly possible to specify assignments to such variables as events and then report the execution of these assignments aurally Such reporting is however independent of the data being assigned To obtain data dependent auralization LSL provides the dtrack command The syntax of dtrack appears below dtrack lt track id list gt lt sound specifier gt lt mode specifier gt lt start event spec gt lt term event spec gt Using dtrack one or more variables can be tracked For the variable to be tracked an initial value can optionally be specified using the init keyword The type of the initial value m
51. pendix An LSL program is known as a specification Each specification is composed of one or more specification modules Each specification module is composed of zero or more specification definitions and one main specification A specification module a specification definition and a main specification are analogous to respectively a module a procedure Figure 2 Occurrence space characterization in LSL 10 and a module body in a Modula 2 15 program As an example of LSL specification structure consider the specification listed in Figure 3 It begins with begin auralspec and ends with end auralspec Each module begins with a header identified by the specmodule keyword followed by the module name such as spec module_1 spec_module_2 and so on in this example A module header is followed by applicability constraints which specify parts of the program to which the specifications are to be applied Then come declarations of variables used in this module followed by zero or more specification definitions such as spec def 1 spec def 2 and so on Global variables are shared between various modules Variables and specification definitions to be exported imported are listed in the export import declaration Variables declared in the program being auralized can also be used inside LSL specifications These are known as external variables 4 1 Constants variables and types LSL is a typed language It contains constants variables and type
52. ppears below play loop_background func_call no_parameters with mm q 120 inst piano The above play when executed will play the sound associated with the variable loop_background together with a sequence of sounds denoted by the variables func_call and no parameters Default key and time signatures will be used The metronome will be set to play 120 quarter notes per minute and the notes will be played using a piano sound 4 3 Event notification A useful characteristic of LSL is its ability to specify events to be auralized A programmer may formulate an event to be auralized in terms of the application However such a speci fication is translated in terms of program position data and time as described earlier For 3Syntactic entities are enclosed in lt and gt Optional entities are enclosed in and For a complete syntax of LSL see Appendix 17 example in an automobile simulator events such as gear change speed set resume cruise and oil check may be candidates for auralization Suppose that the occurrence of these events is indicated by calls to procedures that correspond to the simulation of an activity such as gear change It is these procedure calls that serve as event indicators to LSL Thus for example such a call to the gear_change procedure could be mapped to sound using an LSL specification Event specification is achieved by the notify command notify is a generic command and can be adapted t
53. re The following LSL C specification meets the above auralization requirements begin auralspec specmodule loop_auralize var l begin lend b_begin b_end note begin loop_auralize l_begin C4 h Lend C5 f b_begin E4 q b_end G4 q notify all rule while statement enter using l begin notify all rule while_statement_exit using l end notify all rule while_body_begin using b_begin notify all rule while body end using b_end end loop_auralize end auralspec Example 2 An automobile contains a distributed microcontroller network The software to control this network and other automobile functions is to be tested and debugged All calls to 29 functions gear_change oil_check and weak_battery are to be auralized by playing suitable sound patterns The following LSL C specification meets this auralization requirement begin auralspec specmodule call_auralize var gear_change pattern oil check pattern battery_weak_pattern pattern begin call_auralize gear_change_pattern F2G2F2G2F2G2C1 qq C1 f oil check pattern F6G6 h battery weak _pattern A202A2C2 notify all rule function_call gear_change using gear_change_ pattern notify all rule function call oil check using oil check pattern notify all rule function call battery_weak using battery_weak_pattern end call_auralize end auralspec Example 3 A parallel program con
54. rected editor for LSL named LSLed and a graphic interface LSLed is useful while writing LSL specifications directly using LSL The graphic interface is useful for a novice to LSL Whether one uses LSL directly or the graphic interface to specify an auralization the result is an LSL specification This specification serves as the basis for instrumenting the program to be auralized The instrumentation consists of calls to procedures from LSL sound library The implementation of various parts of the LSL environment is currently underway as part of the LISTEN project Once different components of this environment are available and integrated we will be sufficiently equipped to conduct experiments with the use of auralization in various program development activities The tools are also expected to help other researchers in the area of program auralization Acknowledgements Ronnie Martin spent many painful hours reviewing the first draft of this report The idea of using time in LSL is due to the guitarist humorist and computer scientist Professor Vernon Rego The most wonderful teachers Verna Abe and Helen Brown taught the sec ond author the rudiments of music notation and theory that shaped parts of LSL design 40 Will Montgomery and Neil Herzinger answered endless MIDI related questions Our thanks go to all these individuals References 1 2 A V Aho R Sethi and J D Ullman Compilers Principles Techniques and To
55. rface also provides an environment for executing an aural ized program Thus after having compiled the program it may be instructed to execute the program under programmer s control This feature may be used to debug the program and alter auralizations when necessary 39 7 3 The LSL sound library This is a library consisting of procedures that are responsible for sending appropriate data to MIDI or other sound generation devices during program execution The entire package to manage the processing of notes received during program execution is a part of the library In addition the library consists of several predefined functions callable from within an LSL specification and a C program 8 Summary Research efforts in program auralization appear to be on the increase We are not aware of any general purpose method to specify program auralizations and a supporting tool to auralize a program in accordance with the specifications We have presented the syntax and semantics of a language named LSL that provides a notation to specify a variety of program auralizations LSL is generic and needs to be adapted to the programming language of an environment in which programs are expected to be auralized A language specific implementation of LSL serves as a tool to auralize programs We have presented the design of LSL C a C adaptation of LSL Writing LSL specifications could be a tedious task To simplify the process we have proposed a syntax di
56. s nition l actual parameter list List of actual parameters int x int y 9 The grammar above contains some terminals prefixed by l Such terminals denote language specific constructs A complete list of such terminals appears in Table 7 These terminal symbols may be nonterminals or terminals in the grammar of the language L of the auralized program The LSL preprocessor attempts to parse over the strings corresponding to such symbols These strings are parsed by the compiler for L 11 Static Semantics The following constraints apply to LSL specifications These are not indicated by the syntax 1 All identifiers must be declared before use Identifiers that belong to the auralized program must appear as externals 2 Local attribute values such as metronome values channels etc which are specified explicitly as attributes take precedence over corresponding global values However they do not alter the global values Global values of such parameters may be set using the set command within an LSL specification or in the program 3 Identifiers declared within a specmodule M are global to M and may be used by all specdefs declared within M Identifiers declared within a specdef S are local to S 93 10 11 12 and may not be used by other specdefs or in any other specmodule Identifiers may be exported by an specmodule for use by any other module by explicitly mentioning it in an export declaration A module may use
57. s clear when one considers the ease of data entry The duration of notes played on the keyboard is determined using various LSLed parameter settings not described here LSLed also interprets commands in an LSL specification For example a play command in an LSL specification can be interpreted to hear how the patterns specified in the command will sound Interpretation of more complex commands such as notify and dtrack are also possible via the graphic interface as described in Section 7 As LSL is a typed language LSLed checks for any type mismatch For example if x is of type note and one types x 4 LSLed complains about the invalid type on the right side of the assignment Figure 6 shows how LSLed fits in a programming environment It takes as input an existing LSL specification file and creates an updated or a new file LSLed obtains its input from an electronic or a computer keyboard LSL library routines are used for the input and output of sound patterns The program to be auralized is also an input to LSLed This enables LSLed to check for any interaction errors between the LSL specification and the program For example if a notify command indicates that the events to be selected are within function con operation LSLed checks if such a function indeed exists in the program If not then a warning is issued In any case checks such as these are also made during LSL specification preprocessing 7 An LSL implementation outline An
58. s just as several other languages do An identifier name is a sequence of one or more characters consisting of upper or lower case letters digits and the underscore _ The first character in an identifier must be a letter or an underscore Upper and lower case letters are treated as being different Variables and constants can be assigned arbitrary names Values likely to arise during program auralization are grouped together into primitive types Table 1 lists the primitive types available in LSL Values of type note and pattern are enclosed in quotes to distinguish them from variable names A note is specified by indicating its pitch e g E4b indicates E flat above the middle C on a piano keyboard Attributes listed in Table 2 can be added to a note separated by a colon A pattern is a sequence of notes and voices played in the specified sequence A sequence of notes within a pattern can be enclosed in parentheses to indicate a blocked chord also referred to as a chord pattern A variable name can be used within a pattern by preceding it with a dot For example if the identifier cmajor denotes a chord pattern then p cmajor E5 denotes a pattern consisting of the value of cmajor followed by the note E5 Just as values could be printed or displayed in computer programs we say that values of type note and pattern could be played back during the execution of an auralized program The set of key signatures constitutes the type ksig
59. sists of a procedure MAIN which after initialization loads each of N processors with a copy of program SOLVE On each processor SOLVE aids in the solution of a partial differential equation using a discretization of a multidimensional input domain During execution SOLVE communicates with its neighboring processors by placing calls to procedures nread for read from a processor and nwrite for write to a processor It is desired to auralize this program so that calls to nread and nwrite are reported by suitable sound patterns Only calls that originate from even numbered processors are to be auralized The following LSL C specification meets the above requirements begin auralspec specmodule par_processing external even processor_num var read pattern write_pattern pattern begin par_processing read pattern G5 write pattern C4 30 notify all rule function_call nread and even processor_num using read pattern notify all rule function_call nwrite and even processor_num using write_pattern end par_ processing end auralspec Example 4 An editor accepts commands for text editing After having executed a command a voice message saying Done is to be generated Assume that after receiving an edit command that is to be voiced as above the editor invokes a function named process_command to process the input command The following LSL C specification meets the above requirement begin a
60. t translated into calls to library functions Other LSL commands such as assignments and dtrack commands get translated into more complex C code It is possible to identify specific constructs of a C program by labelling A label is placed inside a comment by using the keyword label as the first keyword starting with a letter immediately following the comment start delimiter Thus for example label here onemore provides two labels here and onemore for possible use by the LSL preprocessor The following example shows how to label the beginning and end of a loop while c getchar eof label special_loop This is an LSL label for the beginning of loop body nC 28 label special_loop This is an LSL label for the end of loop body 5 Examples of LSL C use We now present a few examples illustrating the use of LSL Each example consists of a problem statement and a solution using LSL C In each example we assume that unless specified otherwise default values are used for various sound related parameters such as MIDI channel timbre volume and metronome value Example 1 It is desired to auralize all loops in a C program Loops will be identified by the while do constructs On entry to a loop note C4 is to be played for half a measure duration Each time control reaches the start and end of a loop body notes E4 and G4 are to be played respectively On loop exit note C5 is to be played for a full measu
61. tart E End Program execution Sound Start El E2 End Program execution Sound gt Start El E2 E3 End Program execution Figure 4 Sample activity patterns specifiable in LSL E1 E2 and E3 denote events Start and End denote the start and end of program execution atrack when lt event specifier gt until lt event specifier gt lt sound specifier gt lt mode specifier gt lt event specifier gt lt sound specifier gt and lt mode specifier gt have the same meaning as in the dtrack command Tracking begins when the event specified immediately following when occurs start event and stops when the event specified following until occurs terminating event If the start event is omitted tracking begins at the start of program execution Tracking ends at program termination if the terminating event is omitted If both the start and terminating events are omitted then the entire program execution is tracked In continuous mode an activity begins whenever the starting event occurs and terminates at the terminating event In the discrete mode an activity occurs as above but does not resume Using the start and terminating events one may specify a variety of activity tracking patterns as shown in Figure 4 4 6 Timed events LSL provides a powerful mechanism to auralize timed events time is a special variable in LSL which denotes the time spent from the start of program execution Time is measured in
62. the above notify will generate the sound corresponding to the variables body_begin and body _end respectively whenever the beginning and end of a while loop body are executed Example 2 is the same as Example 1 except that the event selection is selective Thus any loop body labelled by special_loop will be auralized Any syntactic entity can be labelled in the program being auralized by placing an LSL label command in front of that entity as described in Section 4 11 Example 3 specifies the execution of the statements count and search 1 as the events When any of these two events occur count_or_search is played However these events are to be recognized only inside functions search and report Example 4 above specifies an event which occurs whenever the condition x lt y p gt q is not satisfied Note that this condition is based on variables in the program being auralized When this condition is not satisfied assertion_failed is to be played Example 5 shows how to specify the auralization of all conditional expressions that occur in file myfile c only when condition odd x is not satisfied The all and selective tags can restrict any event selection Multiple labels are used within one notify command as in the following notify selective label loop_1 loop_2 rule while_loop_body_begin using body_begin notify selective label special loop rule while loop_body_end using body_end The above notify commands specify the same type o
63. uralspec specmodule editor_auralize var done_voice voice myfilename file begin editor_auralize filename done container v done_voice filename notify all rule function return process command using done_voice end editor _auralize end auralspec Example 5 It is desired to auralize all labelled assignments in 10 files named file 1 c file 2 c and so on through file 10 c Assignments in the first of these files are to be notified by playing a C4 the next by playing a D4 then by E4 and so on using successive notes in the C major scale The following LSL C specification meets this requirement begin auralspec specmodule for_loop const num_of_files 10 31 var first_note next_note note next file int filenames array 1 num_of files of file specdef init_file names begin init_file names filenames 1 file 1 c filenames 2 file 2 c file 4 c filenames 3 file 3 c filenames 4 3 filenames 5 file 5 c filenames 6 file 6 c filenames 7 file 7 c filenames 8 file 8 c 9 file 9 c filenames 10 file 10 c end init_file names filenames begin for_loop next _file 1 next_note first_note init file names for next_file 1 to num_of_files do begin notify selective label this_assign rule assignment_expression using next_note in filename filenames next_file next_
64. ust match that of the variable to be tracked The initial value may also be captured immediately after the execution of an assignment labelled using an LSL label As in notify a lt sound specifier gt specifies the sound to be used while tracking the variables Here we introduce another method for specifying sounds which is particularly useful in conjunction with the dtrack command A sound pattern whose characteristics 21 depend on program generated data will be referred to as a Value Dependent Aural Pattern and abbreviated as VDAP The using clause in the lt sound specifier gt specifies the name of the function say f that emits a VDAP based on variables being tracked f is a language dependent function containing LSL commands for auralization Thus in LSL C f is a valid C function interspersed with LSL commands f is executed after each assignment to the variable being tracked Tracking may be carried out in continuous or discrete mode In continuous mode tracking begins at the start of program execution unless specified otherwise A note pattern is emitted continuously until there is a change in the value of the variable being monitored When the value changes a newly computed note pattern is emitted continuously In discrete mode a note pattern is emitted once whenever the tracked variable changes its value In discrete mode tracking begins the first time the tracked variable changes its value after program execution Tracking can

A Specification Language for Program Auralization

Contents

Download Pdf Manuals

Related Search

Related Contents