Enabling Collaborative Musical Activities through Wireless Sensor
Contents
1. following requirements which should be present in an ideal collaborative wireless interface 1 MIDI compatible The ideal system must be com patible with the multimedia protocols implemented in most academic commercial devices otherwise its adoption in real scenarios would be limited MIDI is a good option as it has been widely adopted within the musical industry 2 Transparent operation The communications system should be able to transmit the data through a wireless network as if the musical device was using its regular nonwireless interface 3 Collaborative activities Communications among all the nodes are possible allowing the intelligent asso ciation of two or more of them in order to carry out collaborative activities such as a light show for a live concert This requirement contrasts with the limitations associated with the master slave commu nications model used by multimedia protocols such as MIDI 4 Wireless communications The use of cables can be problematic in certain cases such as when assembling and disassembling musical devices on stage More over wireless devices are more user friendly since they allow for automatic configuration and greater freedom of movement for the musician 5 License free band The use of a license free band is almost mandatory since it reduces the expenses associated with the use of the technology 6 Coverage extension In some scenarios it would be necessary to ext2. 40 m indoor 80m outdoor 80m outdoor 120m outdoor Unidirectional Bidirectional Bidirectional x 32 Unlimited One channel Automatic Automatic available Low Optimized No Yes generic Yes optimized No No Yes FIGURE 14 Devices proposed to create the prototype xno information given by the manufacturer 12 rehearsal is possible and relatively easy The benefits of SVM musicians are clear as they serve to get an overview of the general composition with no human musicians thus saving time 7 Conclusions and Future Work This article has studied the possibility of using WSN as the technological basis for interconnecting various MIDI compatible musical devices WSN topologies make it possible to increase the number of communication links and enable musical collaborations to take place among different MIDI devices Such collaborations lend themselves to activities related with self identification and self configuration or even to exploring new ways of artistic expression In addition wireless communications facilitate the connection of MIDI devices with other multimedia protocols such as DMX512 making it possible to achieve a more complex level of integration between lights and music The proposed archi tecture offers the possibility of developing more advanced applications for smart musical devices A number of the many applications possible have been put forward score management stage rigging configuration computer aided
3. learning score transcription and the use of a virtual musician Moreover it has been observed that despite the benefits that can be obtained from collaborative activities in the music field none of the musical interfaces studied is valid for this purpose As a result we have developed a preliminary prototype based on the popular Arduino platform and which makes use of ZigBee communications Our prototype was aimed at testing some of the ideas described in this article but it was also valid for discovering the technical issues relating to their real life implementation In this way we performed multiple user power consumption coverage and latency tests It has been shown that it is necessary to optimize wireless communications in order to decrease latency probably by simplifying the protocols of the lower layers of the ZigBee stack The study of such optimizations can constitute an interesting topic for further research Further interesting future work consists in the imple mentation of a demonstration platform which can be used as a paradigm of the concepts described in this article Such a platform could be designed as a collaborative sensor based musical instrument with indoor location features that could be used in contemporary art and science museums Finally another interesting area for future research is that relating to the development of distributed algorithms that could be used in MCWN based collaborations among musical
4. scores This idea is represented in Figure9 where an interactive device like an iPad is placed on each musician s stand and on that of the conductor Such devices integrate the wi module so they would connect to the same MCWN Apart from the logical advantages of using electronic versions of the scores collaboration between the different devices may ease common tasks i Score annotation exchanged between musicians For example violins must follow the bow indications marked by the leader Thus the leader would write the bowing in the electronic score and would share it with the rest of the violinists ii Individual score annotations Each music stand can save annotations that although having a personal meaning for a particular musician may be shared with other colleges for their personal use For exam ple a flautist could add a more piano entry to a given passage because he or she considers that it is important to not to break the sound balance with the other sections These annotations could be shared with other musicians within the orchestra or even with the flute students in their charge iii General score annotations They are particularly useful to the conductor as he or she can mark rehearsal sections tempos and dynamics relating to the whole orchestra directly on to the score The collaborative network will take responsibility for announcing those annotations 10 Figure 10 Stage rigging The s
5. MIDI 26 a low latency 2 4 GHz wireless transmission system with a range of up to 20 meters With a set of alkaline batteries it lasts about 30 hours giving it excellent autonomy but it does not fulfill two of the requirements mentioned in Section 2 3 and 7 i It can only work as a transmitter or as a receiver but not at the same time ii It only admits up to 31 transmitter receiver pairs at the same time so would be inappropriate for use with large orchestras Furthermore the device is quite bulky making it hard to use with certain instruments Other examples of 2 4 GHz transmitters are the M Audio MidAir Wireless MIDI Interface 27 and the CME WIDI X8 Wireless MIDI System USB Interface 28 The M Audio MidAir Wireless MIDI Interface works in half duplex mode and has a range of up to 10 meters which is quite small compared to the range achieved with the CME WIDI X8 Wireless MIDI System USB Interface up to 80 meters with LOS line of sight The latter system is only powered by two AA batteries and allows full duplex communications with up to 64 MIDI channels There is another version of the CME transceiver the CME WIDI XU Wireless MIDI Interface 29 which has been designed to work with computers acting as a wireless USB MIDI adapter The main drawback of these three transceivers is that they are limited to acting as International Journal of Distributed Sensor Networks a mere wireless replacement of a one to one
6. W Piston Harmony W W Norton amp Company 5th edition 1987 26 MIDIJet PRO Wireless MIDI Manual Revision 2 http www organworks com Content Downloads MIDI_Products_ Specs MIDI jet 20Pro 20Documentation 002 pdf International Journal of Distributed Sensor Networks 27 MidAir Wireless USB MIDI System User Manual http www m audio com images global manuals 061114_MIDAIR SA UG_ENO1 pdf SA UG_ENO1 pdf 28 CME WIDI X8 MIDI System User Manual http www cme pro com en getfile php file_id 141 29 CME WIDI XU Wireless System Interface User Manual http www cme pro com en getfile php file_id 165 30 M Audio Mid Air 25 Midi Wireles 25 Key Keyboard User Guide http www m audio com images global manu als 060614_MidAir UG_ENO1 pdf 31 Kenton MidiStreaam MIDI System Operating Manual http www kentonuk com kmanualspdf midistreamman pdf 32 IEEE 802 15 4 2006 IEEE Standard for Local and Metropoli tan Area Networks Specifications for Low Rate Wireless Personal Area Networks 2003 33 ZigBee Standards Organization ZigBee 2007 Specification Q4 2007 http www zigbee org Standards ZigBeeSmartEn ergy Specification aspx 34 S Hsu and J D Tygar Wireless sensor networks a building block for mass creativity and learning in Proceedings of the ACM Creativity amp Cognition Understanding the Creative Conversation Workshop October 2009 35 S Hsu Manipulating digital archives using di
7. connection so there would be no possibility of collaborative work among multiple devices requirement 3 in Section 2 Moreover such devices are not able to extend the network coverage using mesh techniques requirement 6 M Audio also sells the M Audio Mid Air 25 Midi Wireless 25 Key Keyboard 30 a small music keyboard MIDI Controller with an embedded wireless transmitter This product offers basic MIDI editing features although its high current consumption 6 AA batteries that last an average of 2 hours is a significant limitation Another commercial system worth mentioning is the MIDIStream Wireless MIDI System 31 a specialized UHF ultra high frequency communications module for transmitting from a musical instrument to another device reaching a maximum distance of 80 meters outdoors and about 30 meters indoors The transmitter is relatively small the size of a pack of cigarettes and is powered by a 9V battery Unfortunately this module acts as a replacement of one to one MIDI communication and thus collaborative intercommunications among multiple devices are not possible so it therefore fails to fulfill requirement 3 of Section 2 Moreover this device does not offer mesh network capabilities so it is not possible to extend coverage through this mechanism requirement 6 A comparison of the most relevant features of the inter faces previously described is given in Figure 14 which also shows the main characteristics of o
8. g one keyboard to multiple synthesizers or multiple to one e g multiple instruments and one sequencer Host music application Collaborative support layer CSL Network layer NWK Medium access control layer MAC Physical layer PHY F GuRreE 4 Collaborative wireless musical network communications schemes This poses no problem for most WSN transceivers because star and mesh topologies are in general available by default For instance the IEEE 802 15 4 standard 32 supports the star topology whilst ZigBee 33 supports the mesh topology Our MCWN network is composed of two types of nodes i wi conductor which coordinates the network and establishes communications with most nodes within the network ii wi instrument player which joins the network and establishes communications with other nodes within the network The terminology we use is inherited from the field of WSN where a central node coordinator is able to communicate with all the other nodes that form the network end devices It is useful to distinguish between both types of node in order to improve wireless communications performance Also note that at this point the terms master and slave make no sense since one node can freely initiate communications with any other node Figure 4 also shows the complete network stack after adding a new network layer called collaborative support layer CSL at the to
9. signal strength indicator value obtained during the reception of 100 messages Each measure was averaged five times for each experiment to counteract the signal fluctuations caused by indoor fading Both modules transmitted with a power of 3 dBm and boost mode enabled 41 The results obtained are shown in Tables 2 and 3 Note that the total attenuation is the sum of the losses due to free space propagation and the existing obstacles 42 ii Latency tests Latency in music systems is defined as the amount of time elapsed between the emission of a note execution request and the actual execution of such note Latency is especially important in activities that demand real time responses such as live concerts In these tests we have focused mainly on analyzing the latency caused by wireless commu nications 1 e the total communication time when sending MIDI messages of different sizes Although International Journal of Distributed Sensor Networks Arduino Mega board Serial g Serial 1 Serial to USB v FIGURE 8 Latency test TABLE 4 Total latency values measured 1 byte Scenario Mean ms Variance Relay to coordinator Test 1 13 09 LIS Test 2 12 67 0 39 Coordinator to relay Test 1 12 40 25 36 Test 2 12 38 0 40 Total mean 12 63 6 88 the measurements performed are significant we should point out that further study is required in order to take into account all the factors that may influence the
10. used to perform different tests i Multiuser tests The implemented scheme is showed in Figure 7 where a small MCWN network was deployed All nodes are Arduino based with an XBee module Nodes A and B played the role of musicians each playing two different musical excerpts using the MIDI protocol making it necessary to develop a small program On the other hand node C was configured as wi conductor which in this example is responsible for collecting all the notes transmitted by nodes A and B and redirecting them to a software synthesizer As each node transmits its notes using different channels different synthesizer instruments can be configured In our test voice A was assigned to a violin and voice B was assigned to a piano The audio output from the synthesizer was very easy to verify as each voice was selected to be easily recognizable voice A is a broken chord whilst voice B is a static chord It is interesting to note that as there is no synchronization mechanism nodes start playing as soon as they are switched on the resulting output is also not synchronized ii Power consumption tests Table 1 shows prototype 5 Prototype Figure 6 shows the prototype we have built to test the WSN based collaborative paradigm proposed in this paper This patent pending design 39 is based on the popular consumption measured with a multimeter As can be seen power consumption depends on the node state for which there
11. which collaborative activities are feasible depending on the potential collaborators found in step 2 4 The device asks the user which collaborative activity it would like to perform 5 The device associates with one or several musical devices within the musical network to perform the collaborative activity 6 The collaborative activity is performed 7 When the collaborative activity ends go back to step pA 4 2 Collaboration Paradigm Collaborative activities among different musical devices are not usually offered as a feature by MIDI devices due to the previously mentioned limitations in MIDI communications Section 5 shows the benefits of performing such collaborations in several real scenarios In contrast this section presents a new musical instrument based on the concept of collaboration and WSN which could be regarded as a paradigmatic application of the ideas expressed in this article The instrument uses concepts of location algorithms and modular synthesis The idea consists in deploying a set of simple nodes which collaborate with other nearby nodes to play music in real time The performers could be for example visitors to a science museum or a contemporary art museum Visitors would move the nodes within the exposition room thereby changing the associations between the nodes Also the nodes could include some sort of basic controllers buttons sliders touch screens etc in order to allow real time contr
12. Hindawi Publishing Corporation International Journal of Distributed Sensor Networks Volume 2012 Article ID 314078 13 pages doi 10 1155 2012 314078 Research Article Enabling Collaborative Musical Activities through Wireless Sensor Networks Santiago J Barro Tiago M Fernandez Caram s and Carlos J Escudero Departamento de Electronica y Sistemas Universidade da Coru a 15071 A Coru a Spain Correspondence should be addressed to Carlos J Escudero escudero udc es Received 14 July 2011 Revised 9 December 2011 Accepted 10 December 2011 Academic Editor Yuhang Yang Copyright 2012 Santiago J Barro et al This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use distribution and reproduction in any medium provided the original work is properly cited In professional audio production the setup of electronic musical devices is a time consuming and error prone process as it involves manual operations like establishing local configurations and carrying cables for each device Such is the case of MIDI musical instrument digital interface which is widely used in the context of musical applications On the other hand the capabilities of WSN wireless sensor networks allow developers to build up more complex applications since nodes have the ability of autoidentifying autoconfiguring and establishing associations with other nodes behaving in a smarter way than o
13. are three different possibilities sleeping awake and idle and awake and transmitting It is worth mentioning that the consumption values 8 TABLE 2 Signal loss in free space Scenario Mean attenuation dB 50 cm 0 00 lm 8 16 2m 11 65 4m 19 91 8m 23 93 llm 29 61 TABLE 3 Signal loss with different obstacles Scenario Mean attenuation dB Window open metallic blinds 1 04 Window closed metallic blinds 3 95 Wall with open door 0 39 Wall with closed door 1 19 Brick wall 1 46 Between floors 13 08 shown can be easily reduced since some of the electronic components of the prototype are not strictly necessary For instance the design includes two processing units an AVR microcontroller at the Arduino board 40 and an EM250 microcontroller at the XBee ZigBee transceiver 41 Given the above results battery duration can be easily calculated For instance in the case of a 1 200 mAh lithium battery of 9 V node autonomy is 10 93 hours in the worst case It is important to note that power consumption in the awake and transmitting state is not constant each transmission causes a peak of power consumption that lasts a very short time thereby extending the calculated autonomy Also if an application does not need a continuous stream of music data it is possible to allow the node to enter the sleep state thus saving even more energy i Coverage tests This set of tests consisted of detecting the mean RSSI received
14. between WSN and MIDI network topolo gies can be observed when comparing Figures 1 and 2 with Figure 3 where examples of star and mesh network topolo gies are shown WSN networks try to maximize the number of possible connections between network nodes leading to more complex topologies such as the aforementioned star and mesh topologies In star networks all nodes interact with one central node which is called the coordinator Unfortunately the area covered by star networks is restricted by the transmission power of the coordinator This problem does not exist in mesh networks as all nodes within the network can provide network connectivity not only the coordinator Thus the area covered by mesh networks is wider than in a star network WSN technologies offer new possibilities for music appli cations high interconnection capacity bidirectional low powered communications low cost and a communication range in 2 4GHz of up to 120 meters depending on the environment but usually enough for a live concert stage Due to these characteristics WSN based music applications emerge as an interesting field whose optimization is critical due to tight restrictions on bandwidth and latency especially when performing live music Although MIDI has its own specific connectors the well known DIN 5 the fact is that MIDI protocol is independent from the transmission media being used to interconnect the musical devices In fact the MMA Associa
15. data transmission Figure 8 represents the testing scenario for the latency tests An Arduino Mega board 43 which is completely compatible with the Arduino Diecimila we used for the prototype was used because it facilitates the measurement process it has four serial interfaces more than enough to interconnect the two XBee modules needed to perform the latency experiment A USB connection was used to allow wireless data to be sent from and received by a computer where a Java based application was run to measure the communications latency The results obtained are shown in Tables 4 and 5 Depending on the test the communication direction varied from a ZigBee relay to a ZigBee coordinator and vice versa as did the number of bytes transmitted from one byte to three bytes the latter being the typical length of a MIDI message The distance between the two nodes was one meter the wireless modules were configured in AT mode and each test consisted of sending 100 000 messages of one of the two lengths specified One of our concerns was that the underlying WSN protocol could introduce timing differences as ZigBee was not initially thought to be a real time protocol but the results show that there are no significant differences between the measurements in both communication directions Values are very stable although sometimes there is a large delay in certain measures thus causing a high variance The cause of this behavior is being studi
16. devices like those described in this article Acknowledgments This paper has been supported by Xunta de Galicia 1OTICOO3CT and Ministerio de Ciencia e Innovaci n of Spain with FEDER funds of the European Union IPT 020000 2010 35 International Journal of Distributed Sensor Networks References 1 B Katz Mastering Audio The Art and the Science Focal Press 2nd edition 2007 2 A Pejrolo and R DeRosa Acoustic and MIDI Orchestration for the Contemporary Composer Focal Press 1st edition 2007 3 J Huntington Control Systems for Live Entertainment Focal Press 3rd edition 2007 4 M Pukkete The Theory and Technique of Electronic Music World Scientific 2011 5 P Cuadra A Master and C Sapp Efficient pitch detection techniques for interactive music in Proceedings of the Inter national Computer Music Conference pp 403 406 Havana Cuba 2001 6 E H Callaway Wireless Sensor Networks Architectures and Protocols Auerbach 2003 7 MMA Association http www midi org 8 M Wright A Freed and A Momeni Open sound control state of the art 2003 in Proceedings of the Conference on New Interfaces for Musical Expression NIME 03 Montreal Canada 2003 9 Yamaha Mlan http www mlancentral com mlan_info ml an_ppf php 10 HD Protocol 2011 http www midi org aboutus news hd php 11 Entertainment Technology USITT DMX512 A Asynchro nous Serial Dig
17. ed after one million heartbeats are collected iii Interactive WSN bar 37 A WSN network receives NS environmental data from outdoor sensors bright ness temperature and CO density This informa tion is used to make interactive flowers on a bar Also a WSN locating system detects the participant s movement being used to animate the flight of butterflies among different bushes according to the participant s moving between different rooms iFurniture WSN in a community 38 In this exhibi tion there are three types of furniture 1Boxes iChairs and iTables Each user that joins the artwork holds an 1Box which is a tag that stores his or her profile gender personal hobbies etc and mood When the user sits on the iChair the iBox transmits his or her profile and mood so the iChair displays LED colors and images matching the user s information When people gathered together in the same place have common interests the iTable not only displays a visual representation but also plays music allowing people to meet and start a conversation Voice A T CY wie A 4 L Sim i E o Ei i EE a _____ LUS FIGURE 7 Multiuser test Arduino board 40 uses a Digi XBee ZigBee module 41 and has three types of connectors MIDI in MIDI out and USB Additionally a special serial to MIDI driver was developed in order to connect the prototype to a PC that ran professional music software The prototype was
18. ed by the authors and may be International Journal of Distributed Sensor Networks TABLE 5 Total latency values measured 3 bytes Scenario Mean ms Variance Relay to coordinator Test 1 20 64 0 53 Test 2 21 01 81 10 Coordinator to relay Test 1 20 89 54 88 Test 2 20 58 53 27 Total mean 20 78 47 48 motivated by external factors such as intentionally delayed transmissions as a part of optimization techniques or even issues related with Java timing management Finally although latency values are not very high for simple applications they would definitely have to be lowered when working in more complex scenarios To optimize such latency it is important to take into account the configurations available in the XBee modules and the inner workings of the ZigBee protocol Similarly it is important to note that the results shown have been obtained without performing any optimization as this was not their initial objective 6 Applications In this section five different applications for the proposed scheme are discussed i Score management the wi interface is embedded into an interactive iPad like screen which allows musicians to make annotations in their orchestral scores and to share them with their colleges Each musician and the conductor would have a device placed on their music stand ii Stage rigging a series of multimedia devices lights smoke machines etc placed on a stage communi cate with each other t
19. end the coverage to a larger area using relay nodes 7 Bidirectional communications Commercial wireless adapters only allow one way communications thus limiting the range of the activities to be performed within the multimedia network e g devices could report problems in the configuration or malfunc tions 8 Auto identification and Auto configuration Tradi tionally the setup of multimedia systems has been performed manually using cables and local con figurations With a system of the characteristics proposed the automatic configuration of the devices is possible since devices can have enough knowledge about other devices within the network to make intelligent decisions as they are switched on 9 Low latency Musical performances require low latency devices since they occur in real time Further more sound delays can lead musicians to reject the use of the interface In the following section Subsection 3 2 currently avail able interfaces are analyzed in order to determine whether they fulfill the previous requirements International Journal of Distributed Sensor Networks Sequencer Synthesizer s 2 Figure 1 Network topologies available in MIDI technology the synthesizer is an E Mu Xboard49 while the sequencer is an Akai MPC2500 2 3 State of the Art 3 1 Traditional Multimedia Protocols versus WSN Protocols MIDI which stands for musical instrument digital interface
20. er for stagehands to deploy the multimedia network deployment configuration is much simpler because devices can self configure and even report abnormal situations to the technicians Finally musicians also have greater freedom of movement on stage thus benefiting the show 6 3 Computer Aided Learning Computer aided learning can be greatly benefited by the close relationship between music devices and computers which leads to a wide range of possibilities for interaction between students and International Journal of Distributed Sensor Networks Figure 11 Computer aided learning The software shown is Synthesia 45 left The pictures shown in are under Creative Com mons Attribution license Their author is woodleywonderworks right smart virtual teachers SVT One advantage of computer aided learning when compared to traditional systems is the possibility of a more effective training in practical subjects such as music theory harmony counterpoint ear training or instrument playing For instance there are pianos with a special circuit capable of emitting light in the correct keys to guide students helping them to play difficult passages Such a piano could also be able to interact with software that would challenge the student to perform short musical excerpts as if it was a video game shown in Figure 11 6 4 Collaborative Computer Aided Score Transcription The transcription of music scores to electronic versions u
21. fer to any association between two or more devices such as a score digitalization using a musical instrument and a score editor or a light show for a live concert where instruments and lighting system have to be synchronized As a proof of this idea the article also introduces a preliminary prototype of an adapter that enables any MIDI standard device to join an MCWN network easily The article is structured as follows Section 2 defines the requirements of the system to be developed Section 3 gives a detailed description of the background of multimedia protocols MIDI devices that can form part of MCWN networks and commercial wireless MIDI transceivers that could be used to implement some of the ideas put forward in this article Section 4 is devoted to describing the MCWN network architecture explaining how to extend a standard WSN network to support collaborative activities Section 5 presents a prototype that allows any standard MIDI device to join an MCWN network and test basic collaborative activities Section 6 describes real scenarios that could benefit from the inclusion of collaborative features in their normal operations Finally Section 7 is dedicated to conclusions and future work 2 Problem Statement This article describes a WSN based system that enables electronic musical devices to establish collaborative commu nications Our system has been designed according to the International Journal of Distributed Sensor Networks
22. gital art tech niques in Proceedings of the 4th Digital Archive Conference pp 71 78 2005 36 S Hsu J Lin C Chen Y Chen J Lin and K Chang One million heartbeats in Proceedings of the ACM International Conference on Multimedia pp 365 366 2007 37 J Lin S Hsu and Y Chen Interactive WSN Bar in Pro ceedings of the Ist International Conference on Arts and Technology ArtsIT 09 Lecture Notes of the Institute for Computer Sciences Springer 2009 S Liu and S Hsu iFurniture application of wireless sensor network in community interactive furniture in Proceedings of the International Conference on Advanced Information Technology DVD 2009 39 C J Escudero T M Fernandez and S J Barro M dulo de Comunicaci n Inal mbrica para Dispositivos Musicales Electr nicos Patent pending P2010 31523 Spain October 2010 40 Arduino Duemilanove Board Open Source Electronics Prototyping http www arduino cc es Main ArduinoBoard Duemilanove 41 XBeeR XBee ProR ZB OEM RF Modules Manual ver 11 15 2010 http ftp1 digi com support documentation 90 000976_G pdf 42 A Goldsmith Wireless Communications Cambridge Univer sity Press 2005 43 Arduino Mega Open Source Electronics Prototyping http arduino cc en Main ArduinoBoardMega 44 Capture Polar lighting design software http www capture sweden com 45 Synthesia piano learning software 2011 htt
23. hrough the wi interface in order to be synchronized with the music played In this case the lack of cables and automatic configura tion is particularly interesting because the stage set up requires a lot of manual work iii Computer aided learning technology can help music students to improve their musical skills by means of electronic musical instruments Sensors can also be useful in fields where traditional education is not very effective such as music theory harmony counterpoint ear training and certain aspects of instrument playing iv Computer aided score edition this application is aimed at digitalizing manuscript scores The notes can be obtained in real time from a group of mu sicians speeding up the score edition process the inputting of notes using a keyboard and mouse is a really time consuming task FIGURE 9 Score management in musical ensembles v Another application is Virtual Musician where we would build up a musician that is equivalent in func tions to a human musician but with the advantage of being available any time without agenda restrictions It can play with other human musicians adapting its performance as required by the musical discourse tempo dynamics etc 6 1 Score Management in Musical Ensembles One typical problem in large musical ensembles is the management of musical scores Wireless nodes can collaborate among themselves to facilitate the management of such musical
24. ications data travel only from the master to the slave To achieve bidirectional communications two MIDI cables are required It is possible to interconnect several slaves to one master by using the so called daisy chain technique shown in Figure 1 or a MIDI Hub shown in Figure 2 In the latter case the data sent by the master is received by all the slaves while the same MIDI cable is shared by all the devices Also note that MIDI defines channels which allows synthesizers to respond to only one specific channel Sequencer MIDI Hub Synthesizer Synthesizer lt lt e lt gt Su gt Rss P Synthesizer s ere Hees D Y FIGURE 2 Network topologies available in MIDI Technology the synthesizer is an E Mu Xboard49 while the sequencer is an Akai MPC2500 The MIDI hub is an M Audio MidiSport 4 x 4 F GURE 3 Network topologies available in WSN technology star network and mesh network Although bidirectional communications are not strictly necessary they can improve user experience as the master can receive feedback from its slaves to facilitate maintenance operations e g slaves can report hardware failures to enable device discovery within a MIDI network or even to solve configuration problems e g to fix incorrect MIDI channel configurations or to set up a synthesizer It must be said that this feature is already used by other multimedia protocols for example DMX512 11 The difference
25. ields like agriculture health industry and so forth Typically the sensors form ad hoc communication net works able to self organize and auto configure their nodes These features can be taken into account for professional audio production where as we have seen it is necessary to interconnect and configure a large number of musical devices By using WSN to provide connection support for control oriented protocols like MIDI we can obtain many benefits and new possibilities For example in the traditional MIDI connections devices are classified as masters or slaves with masters being responsible for initiating a communica tion However in WSN any device can initiate communica tions providing greater flexibility Additionally WSN offers the possibility of autoidentifying auto configuring and associating devices allowing designers to provide smarter musical applications The main limitation of WSN could be the low bandwidth usually available but this does not represent a real problem for this application since control oriented protocols are not highly demanding in terms of bandwidth Therefore this article proposes the use of WSN tech nology for interconnecting musical MIDI devices This kind of network named musical collaborative wireless network MCWN allows nodes not only to communicate wirelessly but also to auto identify and associate with other nodes in order to perform collaborative activities By collaboration we re
26. is one of the most widespread protocols found in the field of professional audio production 2 It was designed by a group of manufacturers under the name of MMA Association 7 in the late 80s with the main aim of ensuring compatibility between their products something that until then had proved almost impossible because each manufacturer had proprietary connectors cables and protocols As time went by MIDI became very popular among musicians and was extended to support more complex scenarios Although nowadays several academic and commercial alternatives can be found e g OSC 8 mLAN 9 or HD Protocol 10 MIDI remains the most popular as it is the protocol being used by most of the musical instruments devices and software currently available MIDI is a multimedia protocol characterized by being event oriented state based unidirectional and master slave MIDI does not carry any sound but it does carry the information that a synthesizer can use to produce sound MIDI was initially designed to solve the basic problem of intercommunicating one keyboard with one or more synthesizers Events are represented by MIDI messages which are created whenever a musician generates an event during his her performance MIDI can be regarded as a multimedia network protocol Figures 1 and 2 show typical connections between one sequencer and several synthesizers MIDI network topologies can be said to act like unidirectional one to one commu n
27. ital Data Transmission Standard for Control ling Lighting Equipment and Accessories http webstore ansi org RecordDetail aspx sku ANSI E1 11 2008 12 MIDI Media Adaptation Layer for IEEE 1394 Version 1 0 MMA Association http www midi org techspecs rp27v10 spec 1394 pdf 13 IEEE P1639 http dmidi l4l ie index html 14 IETF RFC 4694 RTP Payload Format for MIDI http tools etf org html rfc4695 15 MIDI Control of Stage Lighting http www innovateshow controls com support downloads midi dmx pdf 16 Yamaha WX 5 Owner s Manual http www2 yamaha co jp manual pdf emi english synth WX5E PDF 17 AKAI EWI4000S Electric Wind Instrument Reference Manual Revision D http www akaipro com stuff contentmgr files 0 6bcf69a9363e7bb0d784d6 12468 13646 file ewi4000s_refma nual_revd pdf 18 Steiner MIDI EVI Owner s Manual http www patchmanm usic com MidiEviManualV111 pdf 19 Morrison Digital Trumpet Owner s Manual http www dig italtrumpet com au MDTManual pdf 20 Zendrum ZX Digital MIDI Controller Owner s Manual http www zendrum com pdf manual_01 pdf 21 Sonalog Gypsy MIDI Motion Capture MIDI Controller Suit http www soundonsound com sos oct06 articles son alog htm 22 MakeMusic Finale Music Notation Software http www final emusic com 23 Sibelius Music Notation Software http www sibelius com 24 GVOX Encore Composition Software http www gvox com encore php 25
28. o the lack of musicians of a certain instrument These problems can be partially solved by using smart virtual musicians SVM who can play any musical instrument and are available at any time of day Although the use in professional environments requires the work of skilled technicians their use for group music International Journal of Distributed Sensor Networks Encore Opus 27 Fant Movement A fie fdt Notes woe AL Measures Score Yew Yindows Setup tiep le ba lms Q I SONATE SONATA QUASI UNA FANTASI A Fir das Pianoforte a iS fue 1 FIGURE 12 Collaborative computer aided score transcription The software shown GVOX Encore 24 left The pictures shown in are under Creative Commons Attribution license Their author is Ravpapa right FIGURE 13 Smart virtual musician The pictures shown in are under Creative Commons Attribution license Their authors are Funda o Cultural de Curitiba left and Chris 73 right i Ty MIDIJet Pro MIDI air MIDI stream WIDI XV 8 mo Frequency band 2 4GHz 2 4GHz Maximum communication range est 9m Communication T PETN pen Unidirectional Unidirectional MIDI channels 30 x Communication channel selection Manual Automatic Latency Low Low Coverage extension with relays Yes generic Yes generic Collaborative activity support No No 400 MHz and 868 870 MHz 2 4 GHz 2 4GHz 30m indoor
29. oftware shown is Capture Polar 44 iv Additional features As the score is in digital format it is possible to offer additional features that may help the musicians or the conductor in several situations For instance a musical excerpt could be displayed in different clefs which is very useful when working with transposing instruments Transposing instru ments are those for which the written and played sound does not match as happens with the B flat clarinet for instance which sounds one tone lower than written Visual transposition is especially useful for conductors music arrangers and composers and sometimes even for instrument players 6 2 Stage Rigging Figure 10 shows an MCWN network where several multimedia devices stage lights lasers smoke machines etc are interconnected in a live performance In this scenario it is necessary to synchronize a large number of multimedia devices in order to make them dance with the music which means that even musical instruments can join the MCWN network in order to set the main tempo e g guitar bass guitar or battery It is particularly easy to determine the tempo by monitoring percussion instruments In this specific application it is important to emphasize the advantage of using wireless interfaces as it greatly facilitates the assembling disassembling of the orchestra stage which can be repeated many times if the orchestra is on tour In addition to making it easi
30. ol of synthesis parameters for example general tempo filter depth vibrato amplitude or just note frequency Figure 5 shows a simple deployment of such an instru ment For the sake of simplicity only three different types of node are used The red node is the general input of the whole network and is the only node that cannot be moved by the visitors The other two types are as follows i Wave node which performs a note selected by another node The waveform can be customized and thus the resulting sound will have a different timbre International Journal of Distributed Sensor Networks Sequencer node Wave node A Sequencer node FiGuRE 5 WSN based collaborative musical instrument ii Sequencer node which generates a random sequence of notes as output with different duration and pitches When a sequencer node associates with a wave node the notes generated by the sequencer node are played in real time with the sound configuration selected at that moment in the wave node The association between nodes only occurs when visitors move two nodes and thus reduce the distance by up to half a meter for example Distances can be estimated by using the correlation between distance and the received signal power Figure 5 shows a scenario where the sequencer is moved from one point to another so the sequencer node is first associated with wave node A and then with wave node B Visitors can notice the change of associati
31. on easily as the notes being played have changed in timbre Of course more advanced types and interactions can be implemented but the general idea of collaboration is still valid In 34 four scenarios are shown where WSN based creative art was previously exhibited in two museums namely the National Museum of History and National Palace Museum both in Taiwan i Smart Museum 35 It uses a ZigBee WSN tag based system to identify museum visitors age sex day of the visit etc adapting the content of the pres entations in order to encourage learning by chal lenging them with interactive games Also the data collected by the WSN network are used to grow an e flower and thus make the visitor part of the artwork ii One Million Heart Beats 36 As in the case of the Smart Museum visitors are given a ZigBee tag which also acted as a game controller In the first phase International Journal of Distributed Sensor Networks Figure 6 Prototype design TABLE 1 Power consumption test Node state Consumption mA Sleeping 21 10 Awake Idle 69 80 Transmitting 109 80 visitors took control of virtual sperm fertilizing ova the sex blood type and Chinese astrological birth sign was determined based on the combined action of multiple participants In the second phase users held a bottle with an embedded WSN node that collected a heartbeat for the growing fetus The final resolution of the artwork is determin
32. ound data either in analog or digital format and contain all the information needed to play the sound In contrast control oriented protocols are used to intercommunicate musical devices internally usually to cause an action to be performed in response to the occurrence of an event This is the case of the MIDI protocol in which the information transmitted needs to be interpreted by a synthesizer before an audible waveform can be obtained 2 The combination of both wave oriented and control oriented protocols allows artists to create a complete multimedia performance through the interaction of various multimedia nodes which form a multimedia network Control oriented protocols have the advantage of being much easier to process because the musical events are explicit For instance with wave oriented protocols a signal processing analysis algorithm like the one needed to detect sound frequency 5 can work incorrectly due to the presence of different musicians playing simultaneously in live concerts as the harmonics of their instruments are mixed with the noise However control oriented protocols like MIDI detect the sound immediately simply by reading the corresponding value of a MIDI message Wireless sensor networks WSN are mainly used to collect data from a set of sensors distributed over a wide area without the need for a physical structure 6 By design these sensors are inexpensive and low powered and they are applied to f
33. p www synthe silagame com 38 13 ouralo Hindawi Publishing Corporation http Avww hindawi com 4 A Py TY A Witty A he as I 4 sy WN ae T BUI p 4 Advances in C i i E i i Hindawi Pt Corporation http w icom Volume 2014 The Scientific i wvwowhi r r A O O u n a N 7 a Hindawi Publishing Corporation Hindav poration http lindawicom Volume 2014 http w 3M Y LASGA ESS FN Journal of Sensors http www hindawi com SSO International Journal of Distributed Sensor Network Volume 2014 Journal of Control Scie and Enginee EA Hindawi Submit your manuscripts at http www hindawi com Journal of Robotics jon Advancesin OptoElectronics tay qT 4 lf i EF Ds Ne International Journal of Chemical Engineering fi Ta 4 VLSI Design International Journal of AeMORILOARIAO Observation International Journal of Antennas and Propagation Active and Passive Electronic Components Modelling amp Simulation in Engineering Journal of Electrical and Computer Engineering International Journal of Aerospace Engineerin Hindawi Publishing Corporation http www hindawi com Volume 2010
34. p of a typical WSN stack This special layer deals with device identification communication and asso ciation and therefore the ability to perform collaborative activities To enable all these features a software library has to be developed Such a library has to offer a list with several device categories score editing sound generating etc and the corresponding set of operations each device is able to perform The idea is similar to the ZDO Library used in ZigBee 33 which provides an easy configuration of wireless devices In addition devices could be programmed to behave in a smart way to help users detect hardware malfunctions or configuration errors e g an incorrect MIDI channel setup The CSL is an extra layer explicitly added by us since it is specific for the application and therefore is not included in the default WSN stack The CSL layer is placed between the network layer NWK and the host application Ideally communications with the host application are transparent regardless of being connected to an MCWN network Thus it is possible to reuse existing designs with minor changes The operation of the network from a node perspective is as follows 1 Network connection The musical device is switched on and joins the wireless network 2 Musical device discovery The musical device searches for other musical devices within the network 3 Definition of collaborative activities The musical device determines
35. score editors are MakeMusic Finale 22 Sibelius 23 or GVOX Encore 24 The second group also includes devices that are able to compose music by following several composition rules During the Baroque period the keyboard player was given a score with a bass line with certain numbers annotated with the objective of completing the given line with additional notes respecting the harmony 25 Nowadays technology allows us to build a harmonizing device able to complete the bass line automatically in the same way live musicians used to Finally there exists a third group known as synthe sizers which includes any device that transforms music information into something that can be perceived by one or more human senses The most obvious example is music synthesizers devices that generate audible signals as a response to MIDI events Light synthesizers firework firing systems and water machines fall into this category as well Of course all these categories are not mutually exclusive in practice most devices implement characteristics of two or even three of the categories mentioned 3 2 2 Wireless MIDI Devices In this section the features of some of the most relevant wireless devices currently on the market are analyzed in order to determine whether they are able to satisfy the requirements for constituting a musical collaborative wireless network such as that proposed in this article The first of these is the MIDIJet Pro Wireless
36. sing keyboard and mouse is a very slow process due to the abstract nature of music notation Figure 12 proposes a scenario where an MCWN network is used to collect notes played by a string ensemble using pickups and sensors capable of detecting the music performance All those notes are sent to music transcription software which is able to translate the musical performance to an electronic score with a certain error rate This collaborative activity enormously accelerates the transcription of musical scores Score editing software like MakeMusic Finale 22 Sibelius 23 and GVOX Encore 24 could easily integrate music transcription through a collaborative network saving the musical data using their own score formats 6 5 Smart Virtual Musician Technology allows us to have a virtual substitute for a musician either for individual re hearsals or even for public concerts That is the idea represented in Figure 13 where a virtual musician plays a trumpet concerto with a human pianist The idea of using a robot as a metaphoric representation of a synthesizer is very interesting since we visualize a synthesizer as a smart musician that can be asked to play any instrument The advantages of using virtual musicians are clear for anyone accustomed to the scheduling problems of musicians For instance music students often have agenda issues even in small ensembles duets quartets etc and sometimes parts of orchestras are unbalanced due t
37. ther networks In this paper we propose the use of an optimized WSN network for interconnecting MIDI devices This network has been named collaborative musical wireless network CMWN it eases device configuration enables musical collaboration and allows artists to explore new ways of expression The paper also presents the hardware and performance results of a prototype able to create CMWNs 1 Introduction The field of professional audio production refers to all those activities in some way related with the processing of sound using electronic means 1 that is musical performances composition and arrangement 2 artistic performances and multimedia spectacles 3 Technology plays an impor tant role in the world of professional audio production Several types of device are involved such as microphones to record voices and musical instruments audio processors to apply special effects echo delay etc and multitrack mixers to synchronize audio and video in soundtracks for documentaries and cinema amongst others The quality of the artistic results depends on the selection of the device settings which is usually done by a technician with special skills in music and technology either by using cables or software Communication between musical devices is possible thanks to musical protocols which can be classified into two categories wave oriented protocols and control oriented protocols 4 Wave oriented protocols carry the s
38. tion released documents dealing with USB IEEE 1394 or FireWire 12 Ethernet 13 and RTP Real Time Protocol 14 transmission media to use with MIDI As detailed in Section 3 2 2 there are also wireless MIDI interfaces available on the market Such interfaces are wireless replacements of one to one MIDI communications and therefore do not allow the one to multiple node communications that would support collaborations between several musical devices The features of multimedia WSN networks also include the fact that the intercommunication and integration of devices become easier Moreover a multimedia WSN net work would extend communication to other devices since it removes the natural boundary of cables and connectors and thus facilitates collaboration between musical and nonmusical devices For instance it is straightforward to translate musical notes into light movements and colors e g 15 helping artists find new ways of expression 3 2 MIDI Devices 3 2 1 Generic MIDI Devices The MIDI protocol was initially conceived to support musical keyboards and synthesizers but as time went by the range of devices that could be con nected to a MIDI network increased extending to a high pro portion of all the currently manufactured electronic musical devices Available MIDI devices can be divided into several groups The first group is called instrument like devices and includes any MIDI device that emulates at any le
39. ur prototype detailed in Section 4 It is important to note that none of the wireless MIDI devices studied allows one to multiple node communica tions as they all have been designed as transparent replace ments of the original MIDI one to one communications Only WIDI XV 8 permits a restricted one to multiple com munications allowing the user to switch among different slaves by pressing a button Observing such lack of support for collaborative musical activities we decided to implement our own prototype for performing the required one to multiple node communications This prototype is called wi making reference to wireless musician and can be defined in a nutshell as a WSN based MIDI interface 4 Architecture Description 4 1 Architecture Description Figure 4 presents the architec ture of a musical collaborative wireless musical network MCWN a specific WSN especially designed to pro vide optimized communications between musical devices thereby enabling collaborative activities among multiple participants Our definition of collaborative activity includes any association between two or more devices regardless of its complexity For instance a communication between a MIDI controller and a score editor is a good example of a simple collaboration More complex applications are described in Section 5 A star topology seems to be the most appropriate for musical applications since they typically use one to multiple e
40. vel of detail the physical appearance and touch feeling of an existing instrument Examples of such group are the Yamaha WX5 16 a monophonic wind controller with a fingering similar to a flute clarinet or saxophone the AKAI EW14000S 17 another wind controller the Steiner MIDI EVI 18 a trumpet style wind controller and the Morrison Digital Trumpet 19 a brass style controller designed by Steve Marshall with the Australian multi instrumentalist James Morrison The first group also includes novel and creative musical instruments that do not emulate physical instruments but nevertheless bear some resemblance to traditional instruments Examples are the Zendrum ZX 20 a percussion instrument or the Sonalog Gypsy MIDI 21 a performance instrument for controlling MIDI music through motion capturing The second group is the Music processing group which includes any device that deals with music as a piece of information producing some sort of music as an output One typical example is a sequencer which is able to record musical events a sequencer assigns a timestamp to each event and stores it into a file and vice versa being able to reproduce previously recorded time stamped events A score editor can be regarded as a kind of sequencer with International Journal of Distributed Sensor Networks graphical abilities that shows a graphic representation of a score in conjunction with additional information Examples of
Download Pdf Manuals
Related Search