Home

d2.2.5 scalable content-based indexing and ranking

1. This report is X public Change History Version Date Status Author Description Partner 0 1 13 01 2012 Draft PB CNR Initial draft 0 2 17 01 2012 Draft PB CNR Release for dev team review 0 3 01 02 2012 Draft ENG Peer reviewing by Massimiliano Nigrelli ENG 0 4 15 02 2012 Draft PB CNR Release for Approval 0 5 23 03 2012 Draft SG AIT Small test improvements suggestions for improvement 0 6 26 03 2012 Draft GA PB CNR Integration of SG suggestions 1 0 30 03 2012 Final LB ENG Approval and Release D2 2 5 V1 0 assets Table of Contents 1 INTRODUCTION 2 IMAGE INDEXING AND RETRIEVAL 2 1 Introduction 2 2 Software Requirements Overview 2 2 1 Requirements 2 2 2 Use Cases Image indexing Image searching 2 3 Technical Documentation 2 3 1 UML Diagrams 2 3 2 Service APIs REST services REST service for searching REST service for inserting 2 3 3 Service APIs Client API Client API for inserting Client API for searching 2 3 4 Software Packaging 2 3 5 Installation and configuration 2 3 6 Scientific foundations 2 4 User Manual 2 4 1 Text Similarity Search 2 4 3 Image URL Similarity Search 2 4 3 File Uploading Image Similarity Search 2 5 Concluding Remarks 3 3D MODEL INDEXING AND RETRIEVAL 3 1 Software Requirements Overview 3 1 1 Functionality overview 3 1 2 Requirements 3 2 Technical Documentation 3 2 1 3D search and retrieval use cas
2. My Europeana About us Communities Partners ThoughtLab Choose a language similarAudio http www europeana eu resolve record 920 Options v e share assets Eurovision Playing with fire as sound Title Eurovision Playing with fire as sound Date 30 05 2010 Related content Creator Eurovision youtube ShOxTI Items d Eurovision Todomondo Eurovision Todomondo as sound Eurovision Playing with fire Actions Add a tag a Description Eurovision Song Contest Final 2010 Romaia Paula Seeling and Ovi Format nothing to extent My Europeana Public Domain Data provider YOUTUBE com Provider ASSETS Europe Audio descriptors music happy E fast More View in original context Save to My Europeana Opens in a new window Embed Figure 20 Audio Description ASSETS Scalable Content based indexing and ranking 59 D2 2 5 V1 0 assots 4 4 5 Audio Search by Audio Description In order to search by audio descriptors 1 Mouse over the audio item result 2 Click on the i button It will display the complete track information including the audio descriptors 3 Click on any of the audio descriptors in order to search by one of these see figure above In the example below we clicked on happy for developers only Example relative URL http assetsdemo atc gr portal brief doc html start 1 amp view table amp assets amp query audioDescriptorSearch mood ha Logged in as
3. Organize v d Dre 1 Lig di Per assets ENS ASSETS 20new image jpg leaning tower of 2 logojpg pisajpg Libra Dol Jpg J My E Pic B vd E d 2jpg eeeeeeveveee Pablo 20Picasso jM Com jpg amp Loc l cx Loc cs Do Filename PNI AlFiles z assots 5 The path is automatically inserted in the search and the user selects Search Images Upload File J Images Upload URL C fakepath 2 jpg Basic Search 2 5 Concluding Remarks The image search and retrieval index was demonstrated during the first year review and published in the Europeana ThoughtLabs http pro europeana eu web guest thoughtlab new ways of searching and browsin ASSETS Scalable Content based indexing and ranking 23 D2 2 5 V1 0 assots 3 3D model indexing and retrieval 3 1 Software Requirements Overview Apart from the well known kind of data like text images video and sound Europeana intends to host 3D representations of cultural objects Thus a technology is required which can index large amounts of 3D models and enable the fast search and retrieval of 3D models A user would like to search for 3D models geometrically similar to a query model The 3D search interface will allow three types of queries models uploaded from the user models returned from a previous search and hand drawn sketches 3 1 1 Functionality overview In the following paragr
4. 2 Copy the content of XUGGLE HOME lib to CATALINA HOME shared lib folder Create the destination folder if necessary The copied content contains all the shared libraries required by Xuggler 3 Specify to Apache Tomcat where to look for shared libraries Edit the CATALINA HOME conf catalina properties configuration file and add the following lines or append the specified folders if the lines already exist shared loader S catalina home shared lib S catalina home shared lib jar java library pathz usr local xuggler lib Or wherever Xuggler libraries are 4 Ensure that the environment variable LD LIBRARY PATH is visible from Apache Tomcat If not or not sure it can be done by adding the following lines in the startup sh script for starting Apache Tomcat in Linux LD LIBRARY PATH SLD LIBRARY PATH SCATALINA HOME lib SCATALINA HOME shared li b export LD_LIBRARY_PATH 6 Be sure to include the required Xuggler s jar files in the WEB INF lib subfolder of your application when deploying it in Apache Tomcat 70 D2 2 5 V1 0 assots Additional notes The first important thing to note about using JNI under Tomcat is that we cannot place the native libraries or their JNI interfaces under the WEB INF lib or WEB INF classes directories of a web application and expect to be able to reload the webapp without restarting the server The class that calls System loadLibrary String must be loaded by a classloader that is not affected
5. 5 Click on the Search button ASSETS Scalable Content based indexing and ranking 57 D2 2 5 V1 0 My Europeana About us Communities Partners ThoughtLab Choose a language Upload File AU Upload URL Search Basic Search Something went wrong An email has been sent to inform our technical staff Please try another search Open go Ji My Documents ASSETS First Year Review g Search First Year Revie P ntacts Send us feedback unded by the E inion MN Organize New folder ET 0 Fr Favorites Documents library BE Desktop First Year Review T Downloads 35 Recent Places Arrange by Folder Name dminor 53 Libraries d kids 3 Documents i2 05 kids mp3 a Music 12 cello suite no 2 in d minor bwv 1008 vi gigue mp3 lis Pictures 18 12 26 Luxembourg 10 6 2011 pptx H Videos 12 26 Luxembourg story board 10 6 2011 pptx 3 Homegroup M Comouter Figure 18 Audio Search by Uploading a track It will take some time but results will eventually appear 4 4 3 Audio Search by url In order to perform a search by url 1 mouse over the options text beneath the search box and drop down menu will be displayed 2 Click on Upload and Search 3 Click on upload url on the left hand side of the text box 4 Enter url e g http audio bmat com audio 2 b bruce springsteen born to run 05 born to run mp3 5 Click on the Search button for de
6. Keyframe selected Figure 22 Video summarization use case Video Summarized in background User search Image search Search similar video Display search results Figure 23 Image similarity search use case ASSETS Scalable Content based indexing and ranking 62 D2 2 5 V1 0 assots Use case for image similarity search The actors are User Europeana Web Interface Europeana Video Service The flow of events is as follows see Figure 23 gom i The Europeana Video Service indexes videos in background The User searches for a Europeana image The Europeana Web Interface shows a list of videos that match the search criteria The User indicates that s he wants to search for videos similar to a specific Europeana image The Europeana Web Interface transfers the query to the Europeana Video Service The Europeana Video Service returns a list of similar videos ordered by similarity The Europeana Web Interface shows the User a list of videos that are similar to the provided image Use case for video similarity search The actors are User Europeana Web Interface Europeana Video Service The flow of events is as follows compe D b DES The Europeana Video Service indexes videos in background The User searches for a Europeana video The Europeana Web Interface shows a list of videos that match the search criteria The User indicates that s he want to search for videos
7. Provides methods associated to audio similarity search as well as audio description search 48 D2 2 5 V1 0 ASSETS Scalable Content based indexing and ranking assots Provided Interfaces void initIndex throws AudioIndexingException Index initialization void insertAudio Europeanald audiold URL audioURL Dictionary lt String String gt metadataObj throws AudioIndexingExceptionlnserts the audio content coming via URL for indexation void insertAudio Europeanald audiold InputStream audioObj Dictionary lt String String gt metadataObj throws AudiolndexingException Submits the audio content coming via InputStream for indexation Dependencies ASSETS Common Figure 15 Audio Searching Service Interfaces Audio Indexing Service Interfaces Service Name AudiolndexingService Responsibility Provides methods associated to audio similarity search as well as audio description search Provided Interfaces AudioDescriptors getAudioDescriptors Europeanald audioQueryld throws AudioSearchingException Given an existing track id it returns its audio descriptors void search Map lt String String gt filterObj throws AudioSearchingException Searches within the collection by audio descriptor void searchSimilar Europeanald audioQueryld Map lt String String gt filterObj throws AudioSearchingException Given an existing track id it returns similar tracks
8. Upload and Search Draw and Search Select Skin 3 The following interface appears e Draw a sketch Opraw Search and search for fase similar objects ill Reset Basic Search 4 Draw shape Draw a sketch a Search and search for NT similar objects ill Reset Basic Search 5 Choose Fill option Opraw Draw a sketch Cx and search for Pase similar objects il Reset Basic Search 6 Click on area to be filled and fill shape ASSETS Scalable Content based indexing and ranking 37 D2 2 5 V1 0 assots Draw a sketch 2 and search for Ori similar objects il Basic Search 7 Press Search button and search for similar 3D model 8 The search in the Assets portal returns 10 results Matches for similarSketch Ca i mum Results 1 10 of 10 Page 1 a share Church of the Savior on Luzhniki Stadium Stadion Zawiszy Soccer City Los Angeles Memorial Andrew Jackson statue Estadi Olimpic de Motoarena w Toruniu Kyiv Olympic Stadium Peoples Heroes Results 1 10 of 10 Page 1 share 3 3 2 Search by uploaded 3d model 1 Open the ASSETS portal H2 Click on Options Upload and Search Upload and Search Draw and Search Select Skin 3 The following interface appears ASSETS Scalable Content based indexing and ranking 38 D2 2 5 V1 0 assots Upload File Upload URL Basic Search H4 Click on the Browse button and choose a query 3d model file from local har
9. type XML JSON XML JSON XML JSON XML JSON XML JSON XML JSON XML JSON XML JSON Path Video Summarization assets ir video summarization rest component assets ir video summarization rest summarizedvideo Video Indexing and Retrieval assets ir video ir rest component POST assets ir video ir rest similarvideotourlimage assets ir video ir rest similarvideotoeuropeanaimage assets ir video ir rest similarvideotoeuropeanavideo Function Display the name of the current component This operation retrieves a previously summarized video from the MongoDB If the summary is not found it returns null This operation creates a summarized video that is stored in the MongoDB If the video summary already exists the method recreates the video summary Display the name of the current component This operation indexes the provided video Search videos similar to provided image Search videos similar to provided europeana image Search videos similar to provided europeana video Figure 27 Video service s index html page The summarization service provides methods to summarize a video item and retrieve a previously summarized video Table 3 gathers the information needed to call the video REST summarization services Prefix path of the service is assets ir video summarization rest Method Response Name Parameters Function type GET XML JSON getComponentNa Retu
10. 1 The keyframes of the video have a small button to search for videos similar to the selected keyframe portal In this case In addition the user can search for videos similar to an image that is not in the l E test for similar 1 The user selects the Upload and Search option which is below the main search box of the portal 2 The user uploads an example d image jpg jpeg gif png from Enter URL and Search his her local disk and moves the DRAW and Search 3D cursor over the Search button Figure 19 shows the image used in this example inca Videos Basic Search 3 The user selects the on videos pop up option 4 The portal returns a list of videos that are similar to the example image Figure 28 Image used for the search ASSETS Scalable Content based indexing and ranking 74 D2 2 5 V1 0 assots Matches for All Texts 0 Images 0 3D models 0 Videos 10 Sounds 0 zz Results 1 100f10 Page share Marie Lafor t Maine Joe DASSIN et IL ETAIT G rard Lenorman La Enrico Macias Melisa Click for more information Interview de Henri Rieben Eddy Mitchell Un Jacqueline Huet Quand Gianni Nazzaro Era o non Sylvie Vartan Tino Rossi Ajaccio zzz Results 1 10 of 10 Page share Enter URL iwww thebowesmuseum org uk uploads 5 The user can also http asia perform the operation Basic Search using the Upload URL op
11. Dependencies ASSETS Common In the following Figure the class diagram of the 3D Extraction service is shown 29 D2 2 5 V1 0 assots BaseAssetsService interface eu europeana assets common service api Extraction3DService extract3D String String extract3DToArray String int extract3DFromSketch String String extract3DFromSketchToArray String int Extraction3DServicelmpl configuration IRThreeDConfiguration setDataManagement DataManagement setConfiguration IRThreeDConfiguration getConfiguration IRThreeD Configuration extract3D String String extract3DToArray String int extract3DFromSketch String String extract3DFromSketchToArray String int o extract3D passes the url of a 3D model to the extractor the results are returned as an XML string o extract3DToArray does the same with the difference that the results are returned as a two dimensional int array o extract3DFromSketch passes the url of a hand drawn sketch to the extractor the results are returned as an XML string o extract3DFromSketchToArray does the same with the difference that the results are returned as an int array 3D indexing service interfaces Service Name Indexing3DService Responsibility Creation of the 3D index Provided String initIndex String objectsFile Interfaces throws Indexing3DException Initializes the 3D index and insert all objects that are
12. If the query is a hand drawn sketch it should be a monochrome PNG file of size either 100x100 or 400x400 TreeMap Double EuropeanaObject searchSimilar InputStream modelFile ASSETS Scalable Content based indexing and ranking 31 D2 2 5 V1 0 assots String extension int numOfResults throws Retrieval3DException Indexing3DException Extraction3DException Searches for 3D models similar to a model given by its input stream If the query is a 3d model it should be in one of the following formats VRML 3DS OBJ OFF X3D If the query is a hand drawn sketch it should be a monochrome PNG file of size either 100x100 or 400x400 Dependencies ASSETS Common In the following Figure the class diagram of the 3D Retrieval service is shown BaseAssetsService interface eu europeana assets common service api Retrieval3DService searchSimilar Europeanald int TreeMap Double EuropeanaObject gt searchSimilar URL int TreeMap lt Double EuropeanaObject gt searchSimilar InputStream String int TreeMap Double EuropeanaObject gt Retrieval3DServicelmpl dataManagement DataManagement configuration IRThreeDConfiguration extraction3DService Extraction3DService setDataManagement DataManagement setConfiguration IRThreeDConfiguration getConfiguration IRThreeD Configuration setExtraction3DService Extraction3DService searchSimilar Europeanald in
13. void deletelmage Europeanald imageld deletes an image from the index imageld Europeanald of the image to delete void commit Commits the indexing process void optimizelndex Optimizes the index void openIndex boolean append Opens the index for inserting append If false the index will be built from scratch void closelndex Commits the indexing process and close the index Dependencies ASSETS Common Client API for searching This API provides the needed methods to interface with the searching service Next table explains the available methods for searching API ImageSearching Responsibility Allows to perform image similarity searches Provided methods void searchByld Europeanald euld Searches similar images starting from an Europeanald already in the index imageQueryld Europeanald of the query image void searchByObj InputStream imgObj Searches similar images starting from a sample image A query image should have a size of at least 500x500 pixels and available in one of the following formats JPG PNG GIF BMP imageQueryObj InputStream of the query image void searchByUrl URL imageUrl Searches similar images starting from a sample image A query image should have a size of at least 500x500 pixels and available in one of the following formats JPG PNG GIF BMP imageQueryURL URL of the query Image 16 D2 2 5 V1 0 assots List lt E
14. 1 UML Diagrams The class diagrams of the domain objects used by the service are presented in Figure 4Figure 3 and Figure 4 e QueryResults contains the query results with the most similar images to the one the system was querying for as a list of Europeana IDs e mage2Features allows extracting the visual features necessary to perform an image similarity search from an image To extract them it makes use of LIRE an open source Java content based image retrieval library QueryResults results List lt Europeanald QueryResutts getResults int int ListsEuropeanald setResults String Figure 3 Images Indexing and Retrieval QueryResults http www semanticmetadata net lire ASSETS Scalable Content based indexing and ranking 8 D2 2 5 V1 0 assots Image2Features Image2Features IRImageConfiguration image2Features InputStream String image2Features File String image2Features URL String Instantiate Import Sends FeaturesExtractionException eu europeana assets service ir image exceptions FeaturesExtractionException Throwable FeaturesExtractionException String Figure 4 Images Indexing and Retrieval Image2Features The class diagram of the Image Searching Service is presented in Figure 5 In this diagram the available methods to perform an image similarity search are shown It is possible to perform a search by e Animage id Europeanald e T
15. 2 amd64 deb sudo dpkg i libqt4 network 4 5 1 2 amd64 deb sudo dpkg i gaia2 deb sudo aptitude install openjdk 6 jre sudo dpkg i pylucene 2 3 1 1 1 amd64 deb sudo aptitude install python mako sudo aptitude install libapache2 mod wsgi sudo aptitude install python formencode sudo dpkg i python webpy_0 310 1_all deb sudo aptitude install python simplejson sudo dpkg i python jsonpickle 0 2 0 trunk 20090715 all deb sudo dpkg i python mimerender 0 2 3 all deb sudo aptitude install python sglalchemy 2 Install Ella sudo dpkg i python ella core 2 deb ella ws 2 deb Ubuntu Just like Debian After installation do execute the following sudo pycentral rtinstall python2 6 Red Hat Enterprise Linux 5 CentOS 5 Note Ella is not compatible with SELinux Instructions for disabling it can be found at this link http www crypt gen nz selinux disable selinux html Also the default security settings for Postgres are too restrictive Password authentication must be enabled in var lib pgsql data pg_hba conf by adding the following lines right before local is for Unix domain socket connections only amp allow localhost connections to ella database 52 D2 2 5 V1 0 assots host ella ella 127 0 0 1 32 password sameuser 1 Add the RPMForge repository to your yum sources see http rpmforge net for further information wget http packages sw be rpmforge release rpmforge release 0 3 6 1 el5 rf xX86 64 rpm sudo rpm U
16. Ingestion subsystem The 3D descriptors are indexed along with the rest of the metadata Alternatively the Feature Extraction service can download the 3D model itself under the assumption that it is accessible Available 3D model formats are wrl x3d off obj and 3ds After the use case is completed the system has a set of 3D descriptors indexed along with the object metadata 3D model search and retrieval use case The following Figure describes the basic flow of events during the 3D model search and retrieval ASSETS Scalable Content based indexing and ranking 26 D2 2 5 V1 0 assots Browser lt gt Upload Query Sketch Query Select Query Feature Extraction Feature Retrieval Matching Result List The use case begins with the user accessing the Europeana site The user uploads a 3D model or draws a 2D sketch for using it as a query The model sketch is sent to the Feature Extraction Service which extracts the 3D descriptors After the matching procedure the search results are returned to the user In an alternative flow the use case begins with the user accessing the Europeana site The user selects a 3D model which was returned in a previous search for using it as a query The 27 D2 2 5 V1 0 assots features are retrieved from the index After the matching procedure the search results are returned to the user After the use case is completed the user receives a list of results in which th
17. KeyFrame instance is a representative picture of the video along with its timestamp in the original video Service Name Video Indexing and Retrieval Responsibility Indexing and searching for videos based on the visual similarity Provided JjRVideoService Interfaces Dependencies ASSETS Common Interface Name IRVideoService Key Concepts Europeanald Operations jndexVideo calculates the visual similarity indexes of the video so that it can be compared to the indexes of other videos for search purposes The method does not return any information but stores the Europeanid parameter and the indexes in the internal database jndexVideoCollection indexes the indicated video collection deletelndexedVideo deletes the indicated indexed video QeleteAllIndexes deletes all video indexes getVideoSimilarToURLImage retrieves a ranked list of videos that are similar to a given example The visually similar videos that this operation returns are obtained from those that our module has previously analyzed and indexed using indexVideo getVideoSimilarToEuropeanalmage retrieves a ranked list of videos that are similar to a given example JgetVideoSimilarToEuropeanalmageStream retrieves a ranked list of videos that are similar to a given example JgetVideoSimilarToEuropeanaVideo retrieves a ranked list of videos that are similar to a given example ASSETS Scalable
18. by reloading the web application itself Then the Xuggler jars should be placed in the SCATALINA HOME shared lib directory Note that in Tomcat the SCATALINA HOME shared lib directory may not exist In this case it has to be created The second important thing is that Tomcat does not take into account the SCLASSPATH or SDYLD_LIBRARY_PATH of the console So you will need to add this to the SCATALINA_HOME conf catalina properties shared loader catalina home shared classes catalina home shared lib jar java library path usr local xuggler lib Video content and summaries In order to point out where the video indexing and retrieval module should search for video content you have to configure the Spring configuration file named assets ir video properties Usually this value is portal content folder content folder which means that it has to search for content in the standard portal content folder In addition the indexed videos are generated in a subfolder of the content folder named generated Video indexes Video indexes are stored in two different places 1 In a folder that has to be specified in the assets ir video properties Spring configuration file Usually this value is global video index path video globalindex path 2 You need to create an additional PostgreSQL database for video indexes For this purpose the following script has to be executed cd assets services ir video src main resources mak
19. listed along with their metadata and 3D descriptors into the index Dependencies ASSETS Common In the following Figure the class diagram of the 3D Indexing service is shown ASSETS Scalable Content based indexing and ranking 30 D2 2 5 V1 0 assots BaseAssetsService interface eu europeana assets common service api Indexing3DService initindex String String A configuration IRThreeDConfiguration setDataManagement DataManagement setConfiguration IRThreeDConfiguration getConfiguration IRThreeD Configuration initIndex String String initIndex creates a new 3D index by passing the URL of a file that contains the URLs of the new 3D models to be indexed 3D retrieval service interfaces Service Name Retrieval3DService Responsibility Returns results similar to a given query to the user Provided Interfaces TreeMap Double EuropeanaObject searchSimilar Europeanald queryID int numOfResults throws Retrieval3DException Indexing3DException Extraction3DException Searches for 3D models similar to a model given by its Europeanald TreeMap Double EuropeanaObject searchSimilar URL modelURL int numOfResults throws Retrieval3DException Indexing3DException Extraction3DException Searches for 3D models similar to a model given by its URL If the query is a 3d model it should be in one of the following formats VRML 3DS OBJ OFF X3D
20. scalable Content based indexing and ranking 47 D2 2 5 V1 0 assots Domain object QueryResultsMarshaller collectionObjects ListsEuropeanaldAdapter getEuropeanaldAdapter ListsEuropeanaldAdapter setEuropeanaldAdapter List Europeanald7 Figure 14 Audio Domain Object 4 3 3 Service APIS Rest Interfaces Method Response Name Parameters Function type GET XML JSON Component Returns the component display name GET XML JSON searchByld audioQueryld id of It returns a set of musically similar the audio to be songs to the seed audio searched by GET XML JSON Search q a query string It returns those tracks which matched including a set of the search audio descriptors with the value to be searched e g mood happy GET XML JSON getAudioDescript audioQueryld id of It returns the audio descriptors of a ors the audio whose given audio Typically this are audio description is displayed in the interface detail view to be retrieved GET XML JSON searchByUrl audioQueryUrl It returns a set of musically similar http url of a remote songs to the seed audio audio to be search by POST XML JSON searchByObj audioFile Audio It returns a set of musically similar Binary data audio songs to the seed audio stream to be searched by 4 3 4 Service APIs Client Interfaces Audio Searching Service Interfaces Service Name AudioSearchingService Responsibility
21. similar to a specific video item The Europeana Web Interface transfers the query to the Europeana Video Service The Europeana Video Service returns a list of similar videos ordered by similarity The Europeana Web Interface shows the User a list of videos that are similar to the provided video ASSETS Scalable Content based indexing and ranking 63 D2 2 5 V1 0 assots l Video Summarized in background User search Video search search Search similar video Display search results Figure 24 Video similarity search 5 2 Technical Documentation 5 2 1 UML Diagrams Service description Figure 25 shows the UML class diagram of the domain object for the summarization service The method createSummarizedVideo of the VideoSummarizationService class is intended to index the video and return a corresponding VideoData object The same original video can be summarized several times with different values in the percentage parameter Each instance of the VideoData class represents a summarization request This data is associated to the Europeanald object by means of the getOriginalVideo method The getVideoSummary and getStoryboard methods do not create new Europeanald instances but a URL or a collection of KeyFrame objects respectively The KeyFrame class contains information for the timestamp of the keyframe in the original video The timestamp units are milliseconds and 0 means the beginning of the video The Key
22. the index 5 Image Indexing Image indexing is finalized Image searching This use case describes the activities that are performed during the image content based searching The main goal of the service is to offer powerful yet easy to use image search ASSETS Scalable Content based indexing and ranking 5 D2 2 5 V1 0 assots functionalities to the portal end users Actors Europeana Ingestion Manager Repox2SIP who harvests the metadata that will be used as input for the ingestion workflow see also Europeana Requirements for integration of ingestion tools ASSETS developers who develop the metadata enrichment services ASSETS portal administrators who use the ingestion workflow to insert content into the ASSETS database and to visualize analyze the ingestion logs Stakeholder End users want to find images similar to one he she has identified as interesting or wants to identify an image whose motif is unknown to him her Europeana wants to offer powerful image search based on non textual metadata and actual content Content providers are unable to develop themselves this type of search by and instead wish to offer the capability to their users via Europeana ASSETS Scalable Content based indexing and ranking 6 D2 2 5 V1 0 assots Basic Flow of Events WNFE Oo 4 Select an image 1 Browsing site 2 Home page 3 Generic results page with images 5 Query image uploading 7 Features Ext
23. 2 Requirements Usability The service should be self explanatory and easy to use even from a non expert user ASSETS Scalable Content based indexing and ranking 43 D2 2 5 V1 0 assots Reliability The audio search functionality should have a 24 7 availability and work every time within acceptable time limits for a user search The returned results should be musically similar to the queried track Performance The feature should have no adverse effects on Europeana eu s web site performance Look amp Feel Font and colors should be in line with Europeana brand guidelines Layout and Navigation Search similar button upload interface audio descriptors Communications Interfaces The interface will use stateless HTTP for the communication with other services or clients Licensing The ir audio uses a java library jElla which complies with the licence used by Europeana artifacts EUPL However this java library invokes functionality of the ella service which is BMAT s copyrighted This back end interface is REST based and supports well known standards like HTTP XML and JSON Documentation Documentation for developers will be delivered as code documentation and as a detailed document outlining the architecture and the interaction between the used components 4 3 Technical Documentation 4 3 1 Audio Service Use Cases The main use case we considered for the audio service are indexing and retrieval services Indexing use cas
24. Content based indexing and ranking 69 D2 2 5 V1 0 assots 5 2 4 Installation and configuration Memory requirements The video summarization module requires certain memory In our internal tests it exceeds 2 GB of memory Windows 32 allocates a maximum of 2GB for the user space This means that 32 bit JVMs are not eligible for executing our module You need a 64 bit JVM Additional software For video summarization the Xuggler library should be installed The following instructions are for the official Ubuntu Linux environment of the ASSETS project The installation process could differ for other environments Section O references sources with additional installation instructions Prerequisites This section assumes that an instance of Apache Tomcat is installed We also assume that CATALINA HOME environment variable points to the Tomcat directory Installation Instructions 1 Download and install Xuggler from http www xuggler com xuggler The installation script describes the environment variables to set up In particular be sure to setup the following environment variables Configure XUGGLE HOME pointing to the Xuggler installation directory Include XUGGLE HOME in the LD LIBRARY PATH environment variable Include XUGGLE HOME in the PATH environment variable All of these variables are not required in all cases but they will maximize the possibilities of making Xuggler work in different situations and environments
25. Frame class also stores the JPEG copy of the keyframe ASSETS Scalable Content based indexing and ranking 64 D2 2 5 V1 0 interfaces videosummarizationService eu europeana assets service ir video api VideoDatalmp e VF originalvideo Figure 25 Video summarization client model Figure 26 shows the UML classes diagram for the indexing and retrieval service The getVideosSimilarToXXX methods of the RVideoService class are intended to obtain a collection of visually similar videos represented as instances of the SimilarityRanking class The operation receives as a parameter a Europeanald object that can be either an image or a video The visually similar videos the operation returns are obtained from those ones that the module has previously analyzed and indexed i e the videos that were received through the indexVideo method 9 SimilarityRanking 9 IRVideoServicelmpl f SimitarityRankings su europeana assets service ir videa api Qf addtSimitarityRankingltem aa SE tantie Impoem deleteAilindexes g SimilarityRankingltem lt SimilarityRankingltem gt deletelndexedVideo String milarityRankingltem getConfiguration IRVidecConfiguration getGlobalVideolndexPath String getVideoSimilarToFurageanalmage String SimilarityRanking getVidecSimilarToEurogeanalmageStream InputS ream String Simila
26. IF BMP Please note that RAW photo camera files are not accepted Suggested resolution for the images is 500x500 pixels or more The size of each image file should not exceed 10 Mega Bytes to reduce downloading and features extraction times System Documentation a Code is commented in a professional manner so that API documentation can be automatically generated and b service documentation detailing the installation configuration and use of the service is object of this deliverable 2 2 2 Use Cases Image indexing This use case describes the activities that are performed during the images indexing Actors e Europeana Ingestion Manager Repox2SIP who harvests the metadata that will be used as input for the ingestion workflow see also Europeana Requirements for integration of ingestion tools e ASSETS developers who develop the metadata enrichment services and use the ingestion workflow to insert content into the ASSETS database e ASSETS portal administrators who use the ingestion workflow to insert content into the ASSETS database and to visualize analyze the ingestion logs Stakeholder e End user wants to find images similar to one he she has identified as interesting or wants to identify an image whose motif is unknown to him her e Europeana wants to offer powerful image search based on non textual metadata and actual content e Content providers are unable to develop themselves this type of search by and instead
27. Personal galleries bluk gt OnlineGallery Home gt Cnlre exhibitions gt Asia Pacific and Africa Collections gt Encamoment at Biehnupur Benga elephar s bulleck carts horses and an Englishman on horseback in the foreground 29 January 1825 Encampment at Bishnupur Bengal elephants Search within this bullock carts horses and an Englishman on poliectio horseback in the foreground 29 January 1823 fa Artist D Oyly Sir Charles 1781 1845 Favourites What is this Medium Pen and ink on paper i You must log in to sse Dete 1823 your Favourites Interactive 720mable n F4 neue aeei Hashi Personal galleries x is thi B Full size printable What is this el You must log in to create More metadata and edit galleries 2 The user right clicks on the image and select Copy image URL Artist D Oyly 1845 VIC mee E Copy image Open image in new tab Chrome to Phone Search Image on TinEye Pen and ink drewing E m m een The copied URL is http ogimages bl uk images 019 019WDZ000002060U00007000 SVC1 jpg ASSETS Scalable Content based indexing and ranking 20 D2 2 5 V1 0 assots 3 The user goes to the Europeana portal and click on the Options menu Logged in as Taichi saved tems 0 Saved searches 0 My Europeana About us Communities Partners TheughtLap Paintings music films and books from Europe s galleries libraries archives and mu
28. String sendlnitLinkToService sends the URL of a file containing 3D model URLs to the index creator In the following Figure the class diagram of the 3D Retrieval client is shown 33 D2 2 5 V1 0 assots interface Retrieval3D eu europeana assets ir threed getComponentNameF romRest String searchByld Europeanald ListsEuropeanald searchByObj InputStream String List lt Europeanald gt searchByUT URL List lt Europeanald gt getResults int int List Europeanald gt Retrieval3DImpl configuration IRThreeDClientConfiguration getComponentNameFromRest String getConfiguration IRThreeD ClientConfiguration searchByld Europeanald List lt Europeanald gt searchByObj InputStream String List lt Europeanald gt searchByUrn URL List lt Europeanald gt getResults int int List Europeanald gt The following query forms can be used e a3D model id Europeanald e the stream of a 3D model e a3D model URL or a hand drawn sketch URL By calling getResults the system returns the results of the 3D search ranked by similarity 3 2 3 Software Packaging The 3D search and retrieval framework consists of two parts e the3D search and retrieval services packaged in a war file e the3D search and retrieval clients packaged in a jar file The services package also contains the 3D index 3 2 4 Installation and configuration The installation of the 3D search and retrieval fram
29. Then execute redhat centos sudo etc init d httpd restart debian sudo etc init d apache2 restart 3 Please note that the first call to ELLA will last because it takes sometime to load all the necessary datasets 4 You may test it by executing the following query such as curl localhost ella ws collections tags search q rock NOTE 1 replace SCUSTOMER by your collection name NOTE 2 In order to execute the above curl application is needed You may install it by doing sudo yum install curl 4 if the previous query fails probably there is a configuration issue See the troubleshooting section Troubleshooting 1 Ifthe test query fails please try the following sudo tail f var log httpd or sudo tail f var log apache if you use debian enter enter enter enter curl localhost ella ws collections tags search q rock You ll notice in the error logs what went 2 If the problem persists please contact Support bmat com and attach the following information var log httpd log see above or var log apache log fetc httpd or etc apache e etc ella e tree var lib ella collections gt collections tree txt e Query that fails to be executed 4 3 8 Scientific foundations Here are some of the references to papers and PhD which are behind the music search provided by BMAT 1 Cano P Koppenberger M and Wack N 2005 Content based music audio ASSETS Scalable Content based indexi
30. a new one Then by calling insertlmage the index can be populated by inserting images URLs or streams 2 3 2 Service APIs REST services Among the REST services some are needed to build an image index and some to perform image similarity searches Figure 9 shows a table with the available services ASSETS Scalable Content based indexing and ranking 12 D2 2 5 V1 0 assots IRImage Services type IRimage Search Service allows users to build an image index and GET XML JSON i viet assets ir image Searches to performs visually image similar searches searching similar images rest searchByld starting from an Europeanald image already in the index POST MULTIPART XML JSON assets ir image Searches searching similar images rest searchByObj starting from an uploaded sample image GET XML JSON assets ir image Searches searching similar images rest searchByUrl starting from an URL of a sample image GET assets ir image Opens the indexing index for rest openindex inserting GET assets ir image Commits the indexing indexing rest closelndex process POST MULTIPART assets ir image Inserts an indexing image into the Irest insertlmageObj index GET assets ir image Inserts an indexing image into the Irest insertlmageUr index Figure 9 IRImage REST services REST service for searching The search service provides methods to perform a visual similarity search They allow the user to perform a s
31. admin Saved items 0 Saved searches 0 Saved tags 0 Log out My Europeana About us Communities Partners ThoughtLab Choose a language CO sad Options amp Matches for audioDescriptorSearch mood happy pease ones Al assets zs Results 1 3 of 3 Page 1 OG share Refine your search a By provider By language By country By dat Europeana the WHAT Eurovision Playing with Eurovision Playing with y date GaGa By type Actions Save this search No image Click through the colour block to see the item We are stil loading Figure 21 Audio Search by Audio Description 4 5 Concluding Remarks The audio search and retrieval service was demonstrated on the first year review of the ASSETS project and was tested for usability within the user evaluation ASSETS Scalable Content based indexing and ranking 60 D2 2 5 V1 0 assots 5 Video summarization adaptation indexing and retrieval 5 1 Software Requirements Overview 5 1 1 Requirements Usability The video services are provided through a Web User Interface which should be self explanatory and easy to use even for basic users Reliability The video services should be deployed in a high availability server with at least 4 GB of principal memory The user should recognize the summarized videos and find results as conceptually similar to the original videos Performance Video summarization and indexing are time consuming
32. ailable in the default package repositories So rpmforge should be added to the list of repositories and the packages for tbb and fftw3f should be installed wget http packages sw be rpmforge release rpmforge release 0 3 6 1 el5 rf x86 64 rpm sudo rpm Uhv rpmforge release 0 3 6 1 el5 rf x86_64 rpm sudo yum localinstall fftw3 3 1 2 5 el5 x86 64 rpm nogpgcheck sudo yum localinstall tbb 2 0 4 20070927 x86 64 rpm nogpgcheck Install the extractor sudo yum localinstall ella extractor x y z 1 x86 64 rpm nogpgcheck 4 3 6 1 Redhat Centos 5 5 onwards 50 D2 2 5 V1 0 assots 1 Most dependencies will be automatically resolved by yum Some of the packages might not be available in the default package repositories So rpmforge should be added to the list of repositories and the packages for tbb and fftw3f should be installed wget http packages sw be rpmforge release rpmforge release 0 3 6 1 el5 rf x86 64 rpm sudo rpm Uhv rpmforge release 0 3 6 1 el5 rf x86_64 rpm sudo yum localinstall fftw3 3 1 2 5 el5 x86_64 rpm nogpgcheck sudo yum localinstall tbb 2 2 1 20090809 x86 64 rpm nogpgcheck sudo yum install imlib2 sudo yum localinstall libavutil49 0 5 2 33 el5 x86 64 rpm nogpgcheck 2 Install the extractor sudo yum localinstall ella extractor 1 1 0 1 x86_64 rpm nogpgcheck 3 Redhat 5 5 provides ffmpeg 0 6 but ella extractor requires ffmpeg 0 5 Therefore remove ffmpeg 0 6 and dependencies sudo rpm e ffmpe
33. aph the different types of search will be described Upload a 3D model and search for similar models The user accesses the interface by selecting the appropriate tab S he uploads a 3D model by clicking a Browse button and selects the model file which is located on his her local hard disk The model is uploaded and the search similar function returns a list of similar results Select one of the existing 3D models and search for similar models The user accesses the interface by clicking on a Search similar link while browsing through the existing models The search similar function returns a list of similar results Create a sketch and search for similar models The user accesses the interface by selecting the appropriate tab on the user interface S he draws a sketch an performs a search on the 3D content The search similar function returns a list of similar results 3 1 2 Requirements Usability The service should be self explanatory and easy to use even from a non expert user Reliability The 3D search functionality should have a 24 7 availability and work every time within acceptable time limits for a user search The returned results should be geometrically similar to the queried 3D object Performance The feature should have no adverse effects on the performance of the Europeana platform Look amp Feel Font and colors should be in line with Europeana brand guidelines 24 D2 2 5 V1 0 assots Layout and Naviga
34. assots D2 2 5 SCALABLE CONTENT BASED INDEXING AND RANKING Advanced Search Services and Enhanced Technological Solutions for the European Digital Library Grant Agreement Number 250527 Funding schema Best Practice Network Deliverable D2 2 5 WP2 2 Deliverable V 1 0 30 March 2012 Document ref ASSETS D2 2 5 CNR WP2 2V1 0 D2 2 5 V1 0 assots Programme Name ICT PSP Project Number 250527 Project Title sssss ASSETS Partners sssssseeeeeeee Coordinator ENG IT Contractors Document Number D2 2 5 Work Package WP2 2 Deliverable Type Prototype Contractual Date of Delivery Actual Date of Delivery Title of Document Author s seesssss Approval of this report 31 January 2012 30 March 2012 Scalable Content based indexing and ranking Giuseppe Amato Paolo Bolettieri Fabrizio Falchi CNR Michalis Lazaridis CERTH Oscar Paytuvi BMAT Fernando L pez UAM APPROVED Luigi Briguglio ENG Summary of this report see Executive Summary HistOry oie nso tecto teceea ceci see Change History Keyword List sss ASSETS similarity indexing services search Availability
35. ball for the data containing similarity information and metadata and one py file configuration per collection A md5sum for all files is also provided to validate data consistency 1a The tarballs must be uncompressed in var lib ella collections The usual directory layout is the following ASSETS Scalable Content based indexing and ranking 53 D2 2 5 V1 0 assots var lib ella collections CUSTOMER timestamp datasets enriched index CUSTOMER CUSTOMER timestamp this is a symbolic link TAGS timestamp datasets index TAGS gt TAGS timestamp this is a symbolic link In order to create the symbolic links you may do the following In s var lib ella collections CUSTOMER timestamp var lib ella collections CUSTOMER In s var lib ella collections TAGS timestamp var lib ella collections TAGS 1b The provided py configuration files must be copied to etc ella collections NOTE Minimal setup will comprise two collections a tags collection and a customer collection However another collection called BMAT may be provided to offer recommendation based on artists tracks not available in the customer collection NOTE Please comment out the database part in the config py if you don t require User Recommendation database database dict 4 connection mysql ella ella amp 9localhost ella H 2 OPTIONAL Only required if you make use of user recommendatio
36. configuration IRAudioConfiguration setDataManagement dataManagement setlRAudioDao componentNameDao initIndexd insertAudio audiold audioURL Dictionary lt String metadataOb insert amp udio audiald audioObj DictionarysString metadataObj Figure 12 Audio Indexing Class Diagram Audio Searching Class Diagram lt sinterface gt gt AudioSearchingService searchSimilar audioQueryld filterObj searchSimilar audioQueryOhj filterObj searchSimilar audioQueryURL filterObj getResults startFrom numResults List lt Europeanald gt lt lt implementation gt AudioSearchingServicelmpl configuration IRAudioConfiguration setDataManagement dataManagement setlr amp udioDao componentNameDao searchSimilar audioQueryld filterObj searchSimilar audioQueryObj filterObj searchSimilar audioQueryURL filterObj getResults startFrom numResults List Europeanald get amp udioDescriptors audioQueryld AudioDescriptors search Map lt string string AudioDescriptors getKeyModed String setKeyMode String getSpeedd String setSpeed String getTimbred String setTimbre String getSound String setSound String getKeyd String setkey String getTonalityQ String setTonality String getMoodso String addMood String getstyles String addStyle String setDescriptors HashMap lt String String getDescriptorsQ List String Figure 13 Audio Searching Class Diagram JassETS
37. cts of all the software services that have been defined analyzed implemented and tested during ASSETS WP2 2 This document provides the following information e The software requirements overview e The technical documentation UML diagrams services description and API documentation the software packaging and installation e The user manual Part of the content of this deliverable already appears in Deliverable 2 0 4 The ASSET APIs ASSETS Scalable Content based indexing and ranking 1 D2 2 5 V1 0 assots 1 Introduction The goal of the ASSETS WP2 2 is to enhance the usability of Europeana portal the European Digital Library platform with innovative services that aim at improving the existing search functionality These services allow users to search multimedia objects based on content similarity They are designed to be reused by any digital library This deliverable provides the technical documentation for the ASSETS content based indexing and ranking services needed to install configure and use them The document is divided into four parts which neatly describe the four ASSETS content based indexing and ranking services It introduces technical aspects of the services such as the software requirements the UML diagrams the API documentation the software installation and configuration and the user manuals The Section 2 Image indexing and retrieval concerns the image similarity service that allows users searchin
38. d disk or click on the Upload URL link and give the URL of a 3d model file to be placed into the input field In this example the 3d model stadium wrl was used as a query which is accessible through the following address http www europeanalabs eu svn assets trunk services ir threed src test resources stadium wrl stadium wrl model vrml Object Mozilla Firefox Cortona Remark Valid 3D model file extensions for this service are wrl x3d 3ds obj off H5 The search in the Assets portal returns 10 results ASSETS Scalable Content based indexing and ranking 39 D2 2 5 V1 0 assots Matches for Results 1 10 of 10 Page 1 Ataturk Olimpiyat Stadi King Fahd International Kyiv Olympic Stadium Los Angeles Memorial Stadion Slaski Narodowy Olympiastadion Berlin Stadion Narodowy Stadion Zawiszy Cape Town Stadium Motoarena w Toruniu Resuts1 100f10 Page 1 share 3 3 3 Search by id In any of the result pages of the last two examples the user can click on the Similar search button located under all the 3d results and search for 3d models similar to that result As an example we continue after step 5 of the last section 6 Click on the Similar search of the result Olympiastadion Berlin ASSETS Scalable Content based indexing and ranking 40 D2 2 5 V1 0 assots Matches for Results 1 10 of 10 Los Angeles Memorial Olympiastadion Berlin Google 3D warehouse conte
39. d to know the steps to follow during their installation and configuration process ASSETS Scalable Content based indexing and ranking 76 D2 2 5 V1 0
40. e The indexing use case explains the indexation process an audio track undergoes when ingested on assets system The main point here is that audio editorial metadata is enriched with data features coming from audio itself through Music Information Retrieval algorithms ASSETS Scalable Content based indexing and ranking 44 D2 2 5 V1 0 assots b 6 Exception Handler Figure 10 Audio indexing Use case Retrieval use case The retrieval use case explains the retrieval search process both when searching based on a track which is already in the assets system or based on a uploaded track sample Results are based on music similarity to the seed track ASSETS Scalable Content based indexing and ranking 45 D2 2 5 V1 0 assots 1 Browsing site 2 Home Page 3 Generic results page with tracks artists and albums 4 Query track by uploading 5 Select a track artist or album 6 Feature Extraction 7 Hybrid Similarity Search 8 Content based similarity Search 9 Display results Figure 11 Audio Searching Use case ASSETS Scalable Content based indexing and ranking 46 assots 4 3 2 Audio Class Diagrams Audio Indexing Class Diagram interface2 AudiolndexingService initlndexQ insert amp udio audiold audioURL Dictionary String metadataObj insert amp udio audiold audioObj Dictionary String metadataObj lt lt implementation gt AudiolndexingServicelmpl
41. e Extraction3DService 28 D2 2 5 V1 0 assots Responsibility Extraction of the 3D low level features out of the 3D objects or hand drawn sketches Provided Interfaces String extract3D String url throws Extraction3DException Extracts 3D low level feature vector from a 3D model given by its url The 3D model should be in one of the following formats VRML 3DS OBJ OFF X3D It returns an xml representation of the extracted 3D low level descriptors int extract3DToArray String url throws Extraction3DException Extracts 3D low level feature vector from a 3D model given by its url The 3D model should be in one of the following formats VRML 3DS OBJ OFF X3D It returns the extracted 3D low level descriptors in a 2 dimensional array of integers String extract3DFromSketch String url throws Extraction3DException Extracts 3D low level feature vector from a hand drawn sketch given by its url The sketch should be saved as a png file The size of the sketch should be either 100x100 or 400x400 It returns an xml representation of the extracted 3D low level descriptors int extract3DFromSketchToArray String url throws Extraction3DException Extracts 3D low level feature vector from a hand drawn sketch given by its url The sketch should be saved as a png file The size of the sketch should be either 100x100 or 400x400 It returns the extracted 3D low level descriptors in an array of integers
42. e clean create populate The make command can receive different options that configure its behavior make clean Crop the DB make create Create the DB make populate Populate the tables of the existing DB make dump Dump the DB content into a file named dumped test data sql 71 D2 2 5 V1 0 assots 5 3 User Manual This section highlights the integrated video summarization indexing and searching features and demonstrates the usability improvements from the end user point of view Video summarization 1 The user searches for europeana and obtains several video items There is an icon at the bottom right of each item indicating that these items contain video The user can also switch to the video tab in which only video items are displayed europeana Search Refine SearchAdvanced Search Upload and Search Related searches judaica wien nice rhein manchester united venice typografie la dolce vita livres cinema Matches for europeana All Texts 0 Images 0 3D models 0 Videos 3 Sounds 0 Page 1 o share 29 Select Language Y Bite Saint Europeana Europeana the WHAT WHERE Europeana Master and WHY sos esult of Page 1 share ASSETS Scalable Content based indexing and ranking 72 D2 2 5 V1 0 assots 2 The user selects an item in example we used the Europeana the WHAT WHERE and WHY object He she obtains the original video along with a storyboard to navigate the video The storyboard contai
43. e ir video module provides the following services through the corresponding interfaces Service Name Video Summarization Responsibility The generation of reduced length versions of original videos and extraction of representative keyframes Provided Interfaces VideoSummarizationService VideoData Dependencies ASSETS Common Interface Name VideoSummarizationService Key Concepts Europeanald Operations MongoDB createSummarizedVideo summarizes a video and produces a video summary that is stored in the MongoDB The caller can optionally indicate the desired percentage for the video summary createSummarizedVideoCollection summarizes a video collection and produces the video summaries that are stored in the ASSETS Scalable Content based indexing and ranking 68 D2 2 5 V1 0 assots deleteSummarizedVideo deletes the video from the MongoDB deleteSummarizedVideoCollection deletes the summarized video collection getSummarizedVideo retrieves a previously summarized video from the MongoDB Interface Name VideoData Key Concepts URL KeyFrame Operations JgetOriginalVideo gets the description of the original video getVideoSummary gets the description of the summarized video getStoryBoard Returns a collection of KeyFrame objects according to the percentage that was indicated for the video summary Each
44. e retrieved objects are geometrically similar to the given query object Available object formats are wrl x3d off obj and 3ds for 3D model queries as well as 100x100 monochrome png images 3 2 2 Service APIs The Search and Retrieval framework consists of the following services e the Extraction3D service responsible for extracting the 3D low level features out of the 3D objects or hand drawn sketches e the Indexing3D service responsible for creating the 3D index e the Retrieval3D service responsible for returning to the user results similar to a given query object 3D search and retrieval domain objects In the following Figure the class diagrams of the domain objects used by the 3D services are shown QueryResults Index3DResponse results List lt Europeanald gt results List lt String gt QueryResults getResults List lt String gt getResults int int List amp Europeanald setResults String QueryResultsMarshaller getEuropeanaldAdapter List lt EuropeanaldAdapter gt setEuropeanaldAdapter List lt Europeanald gt o QueryResults contains the query results with the most similar 3D models to the query object as a list of Europeanalds o QueryResultsMarshaller marshals the 3D search results in a list of EuropeanalDAdapter objects Rest Service purpose o Index3DResponse contains the 3D search results as a list of EuropeanaUris 3D extraction service interfaces Service Nam
45. eaich oveccccece Opticns 2 Text search results page The user clicks on the similar b n of one of the results Fes nanc 0 Save ag D Log cur Togas imone a wnquaie e ASSETS uec ene tecccid conga iecgzid a exped Sg wed Rewts 1 123111398 Page d Fefine your search nomm p e Leopold I inwijcing van het standbeel Ey language a Ed Geerts Ey county u b auluac Ovgensaros Ey country 2 Move Ey dae a a i ns dd un B B L f s e CA mac 2 Actiow 1 amar Search Save tus search Ho gage Cici through tre COUT DUCK ID See ma zem rve Manse Stamberg Freire are sti Eadeg 3 Similar images are retrieved Similar to All HH Results 1 12 cf 24 Page 1 G shae Leopold lt Inwijding van Leopoli Il 50 jaar Lecpold Toneelkundig Leopold Eerste amp Jabileumsmedalle Leopoldi Leopold 1 SSETS Scalable Content based indexing and ranking 19 D2 2 5 V1 0 Lecpold 1 assots 2 4 2 Image URL Similarity Search Description the user is interested in Europeana objects similar to the one he she has found on a web page 1 The user finds an image while navigating the British Library s web site http www bl uk onlinegallery onlineex apac other 019wdz000002060u00007000 html uj ONLINE GALLERY seo sone ton See 30 000 items from our collection Online Gallery Home Virtual books Online exhibitions Highlights tour
46. earch by e Animage id Europeanald e The stream of an image e Animage URL The search methods return a list of Europeanald containing the ids of the most similar images for the query Prefix path of the service is assets ir image searching rest In Table the main service information needed to call it are shown Method Response Name Parameters Function type GET XML JSON searchByld imageQueryld Id of the query Searches image similar images numResults number of results to starting from return Default value 100 if the an Image Id parameter is missing already in the index lAssETS Scalable Content based indexing and ranking 13 D2 2 5 V1 0 assots POST XML JSON searchByObj imegFile InputStream of the query Searches MULTIP image similar images ART numResults number of results to Starting from return Default value 100 if the an uploaded parameter is missing sample image GET XML JSON searchByUrl imageQueryURL URL of the query Searches Image numResults number of results to return Default value 100 if the parameter is missing similar images starting from a URL of a sample image Table 1 REST Search methods REST service for inserting The insert service provides the needed methods to build a new image index and or to insert new images Prefix path of the service is assets ir image indexing rest In Table the main s
47. ervice information needed to call it are shown Method Respons Name Parameters Function e type GET openindex append boolean value If false the Opens the index index will be built from scratch for inserting GET closelndex Commits the indexing process and close the index POST insertlmageObj imageObj InputStream of the Inserts an image MULTIP image to insert into the index ART imageld id of the image to insert GET insertlmageUrl imageuUrl URL of the image to Inserts an image insert into the index imageld id of the image to insert POST updatelmageObj imageObj InputStream of the Updates an MULTIP image to update image into the ART imageld id of the image to update index GET updatelmageUrl imageuUrl URL of the image to Updates an update image into the imageld id of the image to update index GET deletelmage imageld id of the image to delete Deletes an image from the index GET optimizelndex Optimizes the index GET commit Commits the indexing process ASSETS Scalable Content based indexing and ranking 14 D2 2 5 V1 0 assots Table 2 REST methods for management of the image index 2 3 3 Service APIs Client API Client API for inserting This API provides the needed methods to interface with the inserting service Next table explains the available methods for inserting API Imagelndexing Responsibility Allows bui
48. es 3D model indexing use case 3D model search and retrieval use case 3 2 2 Service APIs 3D search and retrieval domain objects 3D extraction service interfaces 3D indexing service interfaces 3D retrieval service interfaces 3D search and retrieval client interfaces 3 2 3 Software Packaging 3 2 4 Installation and configuration 3 2 5 Scientific foundations 23999 Do o Scalable Content based indexing and ranking D2 2 5 V1 0 np PW Ww oo 12 13 14 15 15 16 17 17 18 19 19 20 22 23 24 24 24 24 25 25 25 26 28 28 28 30 31 32 34 34 35 assets 3 3 User Manual 3 3 1 Search by hand drawn sketch 3 3 2 Search by uploaded 3d model 3 3 3 Search by id 3 4 Concluding Remarks 4 AUDIO INDEXING AND RETRIEVAL 4 1 Introduction 4 2 Software Requirements Overview 4 2 1 Functionality overview 4 2 2 Requirements 4 3 Technical Documentation 4 3 1 Audio Service Use Cases Indexing use case Retrieval use case 4 3 2 Audio Class Diagrams Audio Indexing Class Diagram Audio Searching Class Diagram Domain object 4 3 3 Service APIS Rest Interfaces 4 3 4 Service APIs Client Interfaces Audio Searching Service Interfaces Audio Indexing Service Interfaces 4 3 5 Software Packaging 4 3 6 Audio feature extractor installation Technical Requirements Installation Instructions for Redhat Centos 5 X Installation instructions for debian ubuntu 4 3 7 Audio Search Engine installation and configuration Technical require
49. etadata based search can be used as CBIR queries to refine the search external documents can be used to access the Europeana content or even the Europeana content can be used to get information concerning documents owned by the users for instance who is the author of the painting contained in this picture The objective of the Image indexing and retrieval service is to provide a system able to perform effective and efficient similarity search on image documents including images of scanned manuscripts The proposed system offers functionality that uses the real content of the images rather than their metadata only to search for other documents 2 2 Software Requirements Overview 2 2 1 Requirements Usability The web user interface should allow similarity search in a self explanatory and easy way So no particular settings weights or parameters should be required during a search Reliability The image similarity search engine will be based on three MPEG 7 Visual Descriptors that can be automatically extracted from images ScalableColor EdgeHistogram ColorLayout The similarity or dissimilarity function used to compare any two images will be a linear combination of the distances suggested by the MPEG group for each of these features Given the similarity function used for comparing any two images the index data structures will return approximate results The relative quality of the results will be evaluated with the Recall and Er
50. ework is straightforward The services war package needs to be deployed on a web server servlet container like Apache Tomcat or Jetty Two folders need to be accessible for the service the index folder and the temp temporary folder The index folder contains the 3D index in the form of files as they appear in the following list 34 D2 2 5 V1 0 assots association dat bulk load done dat database dat EuropeanaReferenceObjects dat offsets dat pl eO sorted dat pl ei sorted dat pl e2 sorted dat pl e3 sorted dat pl e4 sorted dat pl e5 sorted dat pl e6 sorted dat pl amp e7 sorted dat pl e8 sorted dat pl amp e9 sorted dat pl ei sorted dat pl eii sorted dat pl ei2 sorted dat pl e13 sorted dat pl ei4 sorted dat Wa pl e15 sorted dat Mz pl e16 sorted dat 3 pl e17 sorted dat Mz pl e18 sorted dat 3 pl ei9 sorted dat Wal sort_done dat The temp folder is initially empty and serves for storing the uploaded queries and other feature extraction related files This folder needs to be accessible both for reading and writing AMAN URSCISUIRU SURE SUR FURSUSUSURU RU SU sts After the installation the configuration file assets ir threed properties needs to be updated in order to point to the correct location paths of the two folders The initial configuration may look like threed_index_path services ir threed src main resources index threed_content_folder services ir threed s
51. g nodeps sudo rpm e x264 nodeps sudo rpm e libavutil49 nodeps 4 Andinstall the following packages sudo rpm i libtheora 1 0alpha8 1 x86_64 rpm sudo rpm i x264 0 0 0 0 5 20090708 el5 rf xX86 64 rpm sudo rpm i ffmpeg 0 5 2 2 el5 rf x86 64 rpm Installation instructions for debian ubuntu 1 You have to install the following dependencies sudo apt get install ffmpeg libboost filesystem1 34 1 libboost program options1 34 1 libcrypto 7 libcurl3 libfftw3 3 libsamplerateO libsndfile1 libtag1c2a libtbb2 libyaml 0 1 2 Install the extractor sudo dpkg i ella extractor x y z amd64 deb NOTE The Ella extractor requires an Internet connection to validate its license in order to work Also make sure to exchange the x y z with the current version of the packageOperation Just Execute ella extractor path to the input audiotrack path to the output signature 51 D2 2 5 V1 0 assots 4 3 7 Audio Search Engine installation and configuration Technical requirements RedHat CentOS 5 34 Debian 5 Lenny Ubuntu e Pentium Dual Core 1 86 GHz e RAM Memory 4GB Storage space 100GB Note Storage and Memory requirements depends on the amount of tracks Installation Instructions Debian 5 Lenny 1 Install dependencies sudo aptitude install curl sudo aptitude install python sudo aptitude install python central sudo aptitude install python yaml sudo aptitude install libglib2 0 0 sudo dpkg i libqtcore4 4 5 1
52. g between Europeana images using an image as query The Section 3 3D model indexing and retrieval describes the 3D Model service which allows users to search for 3D models similar to a 3D model selected as query The Section 4 Audio indexing and retrieval refers to the audio service that provides advanced music search and recommendation functionalities Finally the section 5 Video summarization adaptation indexing and retrieval describes the video service that aims at enhancing the functionalities of Europeana for searching browsing pre visualizing and accessing video content ASSETS Scalable Content based indexing and ranking 2 D2 2 5 V1 0 assots 2 Image indexing and retrieval 2 1 Introduction Content based image retrieval CBIR is becoming a popular way for searching digital libraries as the amount of available multimedia data keeps on increasing CBIR applications are becoming popular for accessing cultural heritage information as they represent a complement to metadata based search In fact in some cases metadata associated with images do not describe the content with enough details to satisfy the user queries or sometimes metadata are completely missing Images containing reproductions of works of art contain a lot of implicit information that is not generally described in manually generated metadata Various search paradigms can be supported by means of this functionality In the Europeana context results of a m
53. he stream of an image e Animage URL Then by calling getResults the system returns the results of the query ranked by similarity BaseAssetsService interface eu europeana assets common service api ImageSearchingService getResults int int List lt Europeanald searchSimilar Europeanald searchSimilar URL searchSimilar InputStream ImageSearchingServicelmpl dataManagement DataManagement irmageDao IRImageDao ImageSearchingServicelmpl IRimageConfiguration getResuttstint int List lt Europeanald searchSimilar URL searchSimilar InputStream searchSimilar Europeanald setDataManagement DataManagement setlrimageDao IRImageDao Figure 5 Images Indexing and Retrieval Searching Service Class Diagram ASSETS Scalable Content based indexing and ranking 9 D2 2 5 V1 0 assots interface ImagelndexingService BaseAssetsS ervice eu europearia assets common service api closelndex commit deletelmage String initindex insertCollection EuropeanacCollection insertlmage String InputStream insertlmage String URL openindex optimizelndex updatelmage String InputStream updatelmage String URL ee ImagelndexingServicelmpl configuration IRImageConfiguration dataManagement DataManagement ImageindexingServicelmpl iIRimageContiguration closelndex commit deletelmage String getConfiguration IRImageConfiguration initi
54. hods used to perform an image similarity search are shown They allow performing a search by e Animage id Europeanald e The stream ofan image e Animage URL Then by calling the getResults method the system will return the results of the query ranked by similarity ASSETS Scalable Content based indexing and ranking 11 D2 2 5 V1 0 assots interface Imagelndexing eu europeana assets ir image closelndex commit deletelmage Europeanald getComponentNameFromRest String initindexQ insertlmage Europeanald URL insertimage Europeanald InputStream openindex boolean optimizelndex updatelmage Europeanald InputStream updatelmage Europeanald URL t tt t Imagelndexinglmpl Imagelndexinglmpl closelndex commit deletelmage Europeanald getComponentNameFromRest String getConfiguration Y ImageRetrievalClientConfiguration initlndex insertlmage Europeanald InputStream insertlmage Europeanald LIRL openindex boolean optimizelndex updatelmage Europeanald URL updatelmage Europeanald InputStream 4 4 4 4 Figure 8 Images Indexing and Retrieval Indexing Client Model In this diagram the client side methods used to create and insert images into the image similarity search index are shown initlndex is the method to be invoked in order to create a new image index it destroys the previous index if any and then builds
55. hv rpmforge release 0 3 6 1 el5 rf xX86 64 rpm 2 Install dependencies from official repositories sudo yum install httpd libyaml python yaml python simplejson curl 3 Install 3rd party software packaged by BMAT sudo yum nogpgcheck localinstall python jsonpickle 0 2 0 1 noarch rpm sudo yum nogpgcheck localinstall python mako 0 2 4 1 noarch rpm sudo yum nogpgcheck localinstall python webpy 0 31 2 1 noarch rpm sudo yum nogpgcheck localinstall PyLucene 2 3 2 1 3 x86_64 rpm sudo yum nogpgcheck localinstall mod wsgi 2 1 2 el5 x86 64 rpm sudo yum nogpgcheck localinstall qt45 4 5 2 1 el5 pp x86 64 rpm sudo yum nogpgcheck localinstall python sqlalchemy 0 4 9 1 x86_64 rpm sudo yum nogpgcheck localinstall python mimerender 0 2 2 1 noarch rpm sudo yum nogpgcheck localinstall python formencode 1 0 1 1 x86_64 rpm sudo yum nogpgcheck localinstall libyaml 0 1 2 3 el5 kb x86 64 rpm Note libyaml rpm installation might fail due to newer package in the repositories In this case please do sudo rpm Uhv force libyaml 0 1 2 3 el5 kb x86_64 rpm 4 Install BMATs recommendation engine sudo yum nogpgcheck localinstall libgaia2 5 Install Ella sudo yum nogpgcheck localinstall python ella core 2 2 x86 64 rpm ella ws 2 2 x86 64 rpm NOTE you may install it more quickly by doing sudo yum nogpgcheck localinstall V rpm over the rpm files provided by BMAT Configuration 1 The data received from BMAT is one compressed tar
56. lding and populating the image index Provided void initIndex methods Creates a new image index It destroys the previous index if it exists to build a new one void insertlmage Europeanald imageld InputStream imageObj Inserts an image into the index Images to be indexed should have a size of at least 500x500 pixels and available in one of the following formats JPG PNG GIF BMP imageObj InputStream of the image to insert imageld Europeanald of the image to insert void insertlmage Europeanald imageld URL imageURL Inserts an image into the index Images to be indexed should have a size of at least 500x500 pixels and available in one of the following formats JPG PNG GIF BMP imageUrl URL of the image to insert imageld Europeanald of the image to insert void updatelmage Europeanald imageld InputStream imageObj Updates an image into the index Images to be indexed should have a size of at least 500x500 pixels and available in one of the following formats JPG PNG GIF BMP imageObj InputStream of the image to update imageld Europeanald of the image to update void updatelmage Europeanald imageld URL imageURL Updates an image into the index Images to be indexed should have a size of at least 500x500 pixels and available in one of the following formats JPG PNG GIF BMP imageUrl URL of the image to update imageld Europeanald of the image to update 15 D2 2 5 V1 0 assots
57. ments Installation Instructions Configuration Operation Troubleshooting 4 3 8 Scientific foundations 4 4 User Manual 4 4 1 Audio Search by existing track 4 4 2 Audio Search by uploading track 4 4 3 Audio Search by url 4 4 4 Audio Description 4 4 5 Audio Search by Audio Description 4 5 Concluding Remarks 37 37 38 40 42 43 43 43 43 43 44 44 44 45 47 47 47 48 48 48 48 49 50 50 50 50 51 52 52 52 53 55 55 55 57 57 57 58 59 60 60 5 VIDEO SUMMARIZATION ADAPTATION INDEXING AND RETRIEVAL 61 5 1 Software Requirements Overview 5 1 1 Requirements 5 1 2 Use cases 23999 DO o Scalable Content based indexing and ranking 61 61 61 assots Use case for video summarization Use case for image similarity search Use case for video similarity search 5 2 Technical Documentation 5 2 1 UML Diagrams 5 2 2 Service APIs REST interfaces 5 2 3 Services APIs Client interfaces 5 2 4 Installation and configuration Memory requirements Additional software 5 3 User Manual Video summarization Video similarity search 5 4 Bibliography 6 CONCLUDING REMARKS 23999 _ Scalable Content based indexing and ranking D2 2 5 V1 0 61 63 63 64 64 66 68 70 70 70 72 72 74 75 76 assots Executive Summary This is a technical document detailing the ASSETS architecture and APIs for scalable content based indexing and ranking components It introduces technical aspe
58. n Setup your favorite sglalchemy compatible database tested with Postgres and MySQL install their python connectors python psycopg2 for postgres and MySQL python for mysql and start the server Note If the db server does not run on the server where ELLA is installed then the following sql commands will have to be executed by hand on the DB 2 a Create the database For PostgreSQL sudo u postgres psql c CREATE USER ella WITH PASSWORD ella sudo u postgres psql c CREATE DATABASE ella sudo u postgres psql c GRANT ALL PRIVILEGES ON DATABASE ella TO ella For MySQL mysql u root p e CREATE USER ella localhost IDENTIFIED BY ella mysql u root p e CREATE DATABASE ella DEFAULT CHARACTER SET utf8 mysql u root p e GRANT ALL PRIVILEGES ON ella TO ella 2 b initialize the database model SCHEME is postgres or mysql ASSETS Scalable Content based indexing and ranking 54 D2 2 5 V1 0 assots python m ella core db model create SCHEME ella ella localhost ella 2 c set the connection string in etc ella config py accordingly SCHEME is postgres or mysql sudo vi etc ella config py set database connection to SCHEME ella ella 9localhost ella Operation It is important to install Ella on a clean server since some problems may arise if conflicting apache configurations are present 1 Before starting up the server please make sure that the configuration is completed 2
59. n moods blue party furious happy amongst others The audio description is combined with editorial information artist release date country as well as information distilled from blogs reviews or other external data This supports implementation of a hybrid music search The search engine will then base the search criteria on metadata extracted from the song and its editorial content and will allow the users to search inside the collections for music that sounds like Pixies or to query music like what are the tracks from this repository which sound like Chet Baker or to select I want relaxed jazzy music Once the user has reduced the scope of search via editorial metadata or audio descriptors search the service will provide the means for retrieving similar tracks to a given track so to obtain more user relevant results The search must scale to millions of audio segments and have a simple yet powerful API 4 2 Software Requirements Overview 4 2 1 Functionality overview The basic functionality of the audio service is the similarity search based on either an uploaded track or an existing track in the catalog Please see the comprehensive list of use cases displayed below Upload a music track or a track url and search for similar tracks Select one of the existing audio tracks and search for similar tracks Retrieve an audio description of a given track 4 Search by audio descriptors UN ES 4 2
60. ndext insertCollection EuropeanaCollection insertlmage String InputStream insertlmage String URL openindex optimizelndex setConfiguration IRimageConfiguration setDataManagement DataManagement setiRimageDao IRImageDao updatelmage String URL updatelmage String InputStream r o BR B ox Figure 6 Images Indexing and Retrieval Indexing Service model In this diagram the available methods to create and insert images into the image similarity search index are shown The initlndex method creates a new image index it destroys the previous index if any and then builds a new one By calling the insertlmage method the index can be populated by inserting images URLs or streams ASSETS Scalable Content based indexing and ranking 10 D2 2 5 V1 0 assots interface ImageSearching eu europeana assets ir image getComponentNameFromRest String getResults int int List lt Europeanald gt searchByld Europeanald searchByObj InputStream searchByUrl URL ImageSearchinglmpl configuration ImageRetrievalClientConfiguration results ArrayList lt Europeanald ImageSearchinglmpl getComponentNameFromRest String getConfiguration Y ImageRetrievalClientConfiguration getResults int int List lt Europeanald searchByld Europeanald searchByObj InputStream searchByUrl URL Figure 7 Images Indexing and Retrieval Retrieval Client Model In this diagram the client side met
61. ng and ranking 55 D2 2 5 V1 0 assots recommendation In ACM International Conference on Multimedia ACMMM pages 211 212 2 Cano P Celma O Koppenberger M and Buldu J M 2006 The topology of music recommendation networks Chaos An Interdisciplinary Journal of Nonlinear Science http arxiv org abs physics 0512266v1 3 Cano P Koppenberger M and Wack N 2005 Content based music audio recommendation In ACM International Conference on Multimedia ACMMM 4 Celma O PhD Thesis Music Recommendation and Discovery in the Long Tail 2008 5 Bogdanov D Serra J Wack N and Herrera P 2010 Hybrid music similarity measure Music Information Retrieval Evaluation eXchange MIREX Abstract 6 Bogdanov D Serra J Wack N and Herrera P 2009 From low level to high level Comparative study of music similarity measures In Inter national Workshop on Advances in Music Information Research AdMIRe Co Located with the IEEE International Conference on Multimedia and Expo ICME 7 Bogdanov D Serra J Wack N and Herrera P 2010 Hybrid music similarity measure Music Information Retrieval Evaluation eXchange MIREX Abstract 56 D2 2 5 V1 0 assots 4 4 User Manual 4 4 1 Audio Search by existing track The user searches for Eurovision and obtains several items which include video and audio items The icons indicate whether it is an audio video image or3d model for devel
62. ns are obtained from those that our module has previously analyzed and indexed using indexVideo GET XML JSON getVideoSimilarT oEuropeanalmag e europeanauUriStr The image to be used as search pattern This operation retrieves a list of videos that are similar to the Europeanald image indicated in the europeanaUriStr parameter The visually similar videos that this operation returns are obtained from those that our module has previously analyzed and indexed The visually similar videos that this operation ASSETS Scalable Content based indexing and ranking 67 D2 2 5 V1 0 assots returns are obtained from those that our module has previously analyzed and indexed using indexVideo GET XML JSON getVideoSimilarT oEuropeanaVide o europeanaUriStr The video to be used as search pattern This operation retrieves a list of videos that are similar to the Europeanald video indicated in the europeanaUriStr parameter The visually similar videos that this operation returns are the result of aggregating the visual similarity of the keyframes of the video indicated in the europeanaUriStrVideo parameter The visually similar videos that this operation returns are obtained from those that our module has previously analyzed and indexed using indexVideo 5 2 3 Services APIs Client interfaces Table 4 REST video indexing services Th
63. ns the more significant parts of the video according to our summarization algorithm Matches for europeana T Return to results eae share summary run video t r Europeana the WHAT WHERE and WHY Ary i j Title Europeana the WHAT WHERE and WHY agi 4 COB D Date 23 06 2008 LE ur gt Ae 3 Et Creator Europeana youtube theeuropeanalibrary g P A X T gt b siad i Description Europeana EDLnet conference Users expect the EN A a Tim i interoperable bd h Europeana Promo on Youtube ND C od Format nothing to extent 2 4 X My Europeana Public Domain Data provider YOUTUBE com Provider ASSETS Europe 01 36 01 545 lt More test for similar View in original context Create summary 0 100 10 Create Opens in a new window 3 The user clicks on any keyframe and the video starts playing from that position 4 The user presses the Summary button to obtain a summary of the original video Our algorithm has removed redundant information in order to speed up video browsing ASSETS Scalable Content based indexing and ranking 73 D2 2 5 V1 0 assots Video similarity search D Summary C3 Full video User Expectations Once the video has been indexed the user can search for videos in the portal whose R What is that piece of jewelery worn by Mary Stuart on content Is similar toan example image the occasion of her marriage to William 1
64. nt provider 2011 Google 3D warehouse ASSETS sg 858 Stadion Slaski Narodowy Stadion Harodowy Stadion Zawiszy Cape Town Stadium Motoarena w Toruniu zz Results 1 10 of 10 Page 1 a 18 The search in the Assets portal returns 10 results u Options Similar to Results 1 10 of 10 Page 1 Olympiastadion Berlin Kaftanzoglio stadio in Stadion Zawiszy Kyiv Olympic Stadium Estadi Olimpic de Kosevo Stadium in Soccer City Athens Olympic Stadium E Motoarena w Toruniu King Fahd International HH Results 1 10 of 10 Page 1 a ASSETS Scalable Content based indexing and ranking D2 2 5 V1 0 assots 3 4 Concluding Remarks The 3D search and retrieval framework was demonstrated during the first year review and published in the Europeana ThoughtLabs http pro europeana eu web guest thoughtlab new ways of searching and browsin 42 D2 2 5 V1 0 assots 4 Audio indexing and retrieval 4 1 Introduction The service has two main objectives e Providing a service for the automatic description and content enrichment of music audio data e Providing a service for fast indexing and retrieval of music audio content Music search core engine will listen to understand and interpret music like humans do It will describe audio in musical and perceptual terms like rhythm bpm percussiveness harmony key mode chords timbre instrumentation production qualities genre of the song and eve
65. operations and are executed in background Video search is a faster operation that can be executed online Look amp Feel Web pages layout should be in line with the Europeana web application guidelines Applicable standards The video service is able to process MPEG 2 and MPEG 4 video profiles including Flash Videos The video search service is able to process also images in JPEG PNG GIF and BMP format Documentation The interfaces of the services include standard Javadoc comments This document includes additional conceptual level comments 5 1 2 Use cases Use case for video summarization The actors are User Europeana Web Interface Europeana Video Service The flow of events is as follows see Figure 22 The Europeana Video Service summarizes videos in background 2 The User searches for a Europeana video item 3 The Europeana Web Interface shows a list of videos that match the search criteria 4 The User indicates that s he wants to see a summary of a video item 5 The Europeana Web Interface retrieves the selected video from the Europeana Video Service along with the keyframes of the summarized video 6 The Europeana Web Interface shows the video summary to the User The User selects a keyframe The Europeana Web Interface skips the video to the position of this keyframe 61 D2 2 5 V1 0 assots Video Summarized in background User search Mund aia Display search results
66. opers only Example relative URL http assetsdemo atc gr portal brief doc html startz1 amp view table amp assets http963A962F962 Fi2 ytimg com962Fvi962FmOUYgPmLK 0k 2Fdefault jog amp query similarAudio http 3A 2F 2Fwww europeana eu 2Fresolve 2F record962F92001962FD219B15D18424E5616DB5AO0B606COAED6D200005 Logged in as admin Saved items 0 Saved searches 0 Saved tags 0 Log out My Europeana About us Communities Partners ThoughtLab Choose a language eurovision Options v D Matches for eurovision All assets HH Results 1 4 0f 4 Page 1 Refine your search By provider By language By country Eurovision Playing with ovision Todomondo Eurovision Todomondo By date By type Eurovision Playing with fire as sound Eurovision youtube ShOxTI 2010 YOUTUBE com ASSETS Actions Save this search No image Click through the colour block to see the item We are still loading Figure 17 Audio Search by Existing Track The icon indicates that the retrieved content is an audio track In order to search similar tracks click on the sear similar i e symbol 4 4 2 Audio Search by uploading track In order to perform an example Upload and search query 1 mouse over the options text beneath the search box and drop down menu will be displayed 2 Click on Upload and Search 3 Click on upload file on the left hand side of the text box 4 Select an mp3 wav or wma file
67. p pivots to be used in the query e maxpivsQuery 50 Maximum number of pivots to use during the searching process 2 3 6 Scientific foundations Amato G Savino P Approximate Similarity Search from another Perspective SEBD 2008 Bolettieri P Falchi F Lucchese C Mass Y Perego R Rabitti F Shmueli Scheuer M Searching 100M Images by Content Similarity IRCDL 2009 Gennaro C Amato G Bolettieri P Savino P An approach to content based image retrieval based on the Lucene search engine library ECDL 2010 Amato G Bolettieri P Falchi F Gennaro C Rabitti F Combining local and global visual feature similarity using a text search engine CBMI 2011 Gennaro C Amato G Bolettieri P Savino P An approach to Content Based Image Retrieval based on the Lucene search engine library Extended Abstract SEBD 2011 18 D2 2 5 V1 0 assots 2 4 User Manual In the next pages the functionality of the image search service will be demonstrated through some search samples on the project portal http assetsdemo atc gr portal 2 4 1 Text Similarity Search Description the user first searches for leopold as text query then select one of the results as a query for searching similar objects with the respect to the visual content 1 The user searches for leopold Paintings music films and bocks from Europe s galleries libraries archives and museums Find out mcre eed 77 0 S e S
68. raction 8 Content Based Similarity Search 9 Display Search Results Page Figure 2 Image searching use case 6 Select an image Start The use case begins when the users access the site Browsing site The user is browsing the site Home Page The user is viewing the home page Generic results page with images The user is viewing the results page of a previous generic search Select an image The user selects an image for searching between the ones listed in the home page Query image uploading The user selects an image between the ones in his device and uploads it in order to search for similar images Select an image The user selects an image between the ones displayed in the results page ASSETS Scalable Content based indexing and ranking 7 D2 2 5 V1 0 assots 7 Features Extraction Features are extracted from the image 8 Content Based Similarity Search Content based similarity search is performed 9 Display Search Results Page Results in a web page are displayed to the users Requirements for Content Provision Images to be indexed must be available through URL at medium resolution at least 500x500 pixels and must have one of the following formats JPG PNG GIF BMP Content requirements for Europeana portal Images to be indexed must be available through URL at medium resolution at least 500x500 pixels and must have one of the following formats JPG PNG GIF BMP 2 3 Technical Documentation 2 3
69. rc main resources temp Since this package structure will not be available on the production server the paths need to be updated in order to point to the correct location threed_index_path ir threed index threed_content_folder ir threed 3 2 5 Scientific foundations The description of the scientific foundations on which the 3D search and retrieval framework relies is out of the scope of this deliverable The feature extraction algorithm is based on 1 while the index structure used in the framework is based on 2 1 P Daras A Axenopoulos A 3D Shape Retrieval Framework Supporting Multimodal 35 D2 2 5 V1 0 assots Queries International Journal of Computer Vision Springer Volume 89 Issue 2 2010 09 2 G Amato P Savino Approximate similarity search in metric spaces using inverted files In Proceedings of the 3rd International Conference on Scalable Information Systems InfoScale 2008 pp 1 10 ICST 2008 36 D2 2 5 V1 0 assots 3 3 User Manual In the following sections the functionality of the 3D search and retrieval framework will be demonstrated through an example of a search request in the Assets portal http assetsdemo atc gr portal 3 3 1 Search by hand drawn sketch The user wishes to draw a sketch and search for similar 3d models This can be done through the dedicated interface on the portal 1 Open the ASSETS portal 2 Click on Options Draw and Search
70. rityRanking ge VidecSimilarToEurogeanaVideolS ring SimilarityRanking getVideoSimilarToURLimage String SimilarityRanking indexVideolString int indexVideoCollectioniString int setConfiguration IRVideoConfiguration setDataManagement DataManagement setirVideoDaaiIRVideoDso setVideoSummarizationService VideoSummarizationService Figure 26 Similarity search client model ASSETS Scalable Content based indexing and ranking 65 D2 2 5 V1 0 assots 5 2 2 Service APIs REST interfaces The following subsections list the REST interfaces for video indexing and retrieval Below we provide an example of these REST interfaces invocation GET http localhost 8983 assets ir video summarization rest summarizedvideo europeanaUriStr http www europeana eu r esolve record 07794 337F0F2113547F5F8BB28B41432A93F324B312BO0 In this example GET is the posting method summarizedvideo is the path or the service and europeanaUriStrz is a query parameter In addition to using the unit tests these services can be tested using the service s index html page see Figure 27 Video indexing and retrieval service ERES Indexing and Retrieval dad 5 4 4 This service aims at i il H supporting the summarization GET indexing and retrieval of video objects These specific functionalities are GET Video summarization Video indexing Video retrieval i retrieval POST GET POST GET GET GET Method Response
71. rns the component me name GET XML JSON getSummarizedVi europeanauUriStr This method retrieves a deo The video to be previously summarized retrieved video from the MongoDB ASSETS Scalable Content based indexing and ranking 66 D2 2 5 V1 0 assots POST XML JSON createSummarize dVideo europeanauUriStr The video to be summarized This method summarizes a video and produces a video summary that is stored in the MongoDB Table 3 REST video summarization services Table 4 gathers the information needed to call the video REST indexing services Prefix path of the service is assets ir video ir rest Method Response type Name Parameters Function GET XML JSON gerComponentNa me Returns the component name POST XML JSON indexVideo europeanauUriStr The video to be indexed The method calculates the visual similarity indexes of the video so that it can be compared to the indexes of other videos for search purposes MPEG 1 MPEG 2 and FLV videos are currently supported GET XML JSON getVideoSimilarT oURLImage url An image URL This operation retrieves a list of videos that are similar to the image file indicated in the url parameter The visually similar videos that this operation returns are obtained from those that our module has previously analyzed and indexed The visually similar videos that this operation retur
72. ror on Position measures Expected values are 0 3 for Recall and less than 0 002 for Error on Position Performance Should be able to answer queries within a second User Interfaces A link for searching similar images should be added near each image returned by the Europeana in the web user interface independently from the type of query performed In particular the button or link for similarity searching should be displayed only for those images for which it would be possible to perform the content based similarity search A text reporting the similarity score of the results images could be added in the results list of ASSETS Scalable Content based indexing and ranking 3 D2 2 5 V1 0 assots similar images However this is not mandatory It should not be necessary to have a specific user interface for Content Providers The standard methods and user interface used for ingestion of the images metadata and thumbnails should be used Look amp Feel All the links to the similarity services should be self explanatory linked text Licensing Requirements The image similarity search service is written in Java and requires the Java Runtime Environment There are no requirements to acquire a licence for commercial third party software Third party components are several open source Java libraries like LIRE Applicable Standards The image features extraction software will accept most of the image formats used nowadays i e JPG PNG G
73. seums Find out more Options E MN 4 The user selects Upload and Search Search Upload and Search and Search Select Skin v Upload an item and search for similar 4 The user pastes the of the image and press the Search button Paintings music films and books from Europe s galleries libraries archives and museums Find outmore Images Uoced URL apacioiler 019 dz000002060u00007000 hunb ecco Basic Search 4 The very same object togethe similar items in Europeana are retrieved Matches for All Page 1 2 share ASSETS Scalable Content based indexing and ranking 21 D2 2 5 V1 0 assots 2 4 3 File Uploading Image Similarity Search Description the user has a photo and wants to find similar content in Europeana 1 The user goes to the Europeana portal and click on the Options menu Loaged in as ffalchi Saved tems 0 Saved searches 0 My Europeana ADOUI Us Communities Partners TheugniLap Paintings music films and books from Europe s galleries libraries archives and museums Find out more e Options Y RM 2 The user selects Upload and Search ED Upload and Search Dray and Search Selekt skin y Upload an item and search for similar 3 The user clicks on the box Upload File Upload URL Basic Search 4 The user navigates its own computdr file system eventually selecting an image
74. t TreeMap Double EuropeanaObject gt searchSimilar URL int TreeMap lt Double EuropeanaObject gt searchSimilar InputStream String int TreeMap Double EuropeanaObject gt The following query forms can be used e a3D model id Europeanald e the input stream of a 3D model e a3D model URL or a hand drawn sketch URL The results are ranked by similarity 3D search and retrieval client interfaces In the following Figure the class diagram of the 3D Extraction client is shown ASSETS Scalable Content based indexing and ranking 32 D2 2 5 V1 0 assots interface Extraction3D eu europeana assets ir threed getComponentNameF romRest String sendLinkToService String String Extraction3DImpl configuration IRThreeDClientConfiguration getComponentNameF romRest String getConfiguration IRThreeD ClientConfiguration sendLinkToService String String sendLinkToService sends the 3D model file URL to the extractor and receives the extracted features as an XML string In the following Figure the class diagram of the 3D Indexing client is shown interface Indexing3D eu europeana assets ir threed getComponentNameF romRest String sendinitLinkToService String String Indexing3DImpl configuration IRThreeDClientConfiguration getComponentNameFromRest String getConfiguration IRThreeD ClientConfiguration sendinitLinkToService String
75. t contains the following parameters e image index home S image index location that is the file system path of the image index The index folder contains the image index files e fx config file S image fx config location ASSETS Scalable Content based indexing and ranking 17 D2 2 5 V1 0 assots that is the file system path of the image features extractor configuration file assets ir image fx properties contains the settings for the image features extraction At the moment it mustn t be modified indices properties sets the similarity implementation to be used currently ScalableColor EdgeHistogram and ColorLayout MPEG 7 descriptors LIRE MP7ALL properties is the main image index configuration file This file is used to configure the index It contains the following parameters Hpath syntax means parent path of this configuration file path of the image index folder e archive 0O binaries FCArchive testset lire dat Path of the binary files of the visual features Himage index e lucenelndexPath LuceneLire filtered Path of the image index Pivot file e PivotsPath binaries LireObjectPivots filtered 10k dat Pivot file supplied with the index to use during the indexing and the searching process Num of pivots e num_of_pivots 10000 Number of pivots in the pivot file Top pivots to be used in the index e maxpivsindex 50 Maximum number of pivots to use during the indexing process To
76. tion In this case an Internet URL of the example image has to be provided For instance you can use the following URL http www thebowesmuseum org uk uploads collections fullsize b m 868 JPG 5 4 Bibliography 1 Apache Tomcat http tomcat apache org 2 Xuggler http www xuggle com xuggler 3 Additional information in the xuggle wiki http wiki xuggle com Frequently Asked QuestionsHl get an 22UnsatisfiedLinkError 22 when run Xuggler based Applications in Tomcat 4 Additional information about Apache Tomcat and JNI http wiki apache org tomcat HowTottl 27m encountering classloader problems when u sing JNI under Tomcat ASSETS Scalable Content based indexing and ranking 75 D2 2 5 V1 0 assots 6 Concluding Remarks In this deliverable we have described the ASSETS services for the content based indexing and ranking components implemented and tested in ASSETS WP2 2 The technical aspects of the following components have been explained in detail Image indexing and retrieval service 3D model indexing and retrieval service Audio indexing and retrieval service Video summarization adaptation indexing and retrieval service The software requirements the technical documentation UML diagrams services description and API documentation the software packaging and installation and the user manual have been provided for each service in order to allow developers to understand how to use these services an
77. tion Search similar button upload interface sketch interface Communications Interfaces The interface will use stateless HTTP requests for the communication with other services or clients Licensing The 3D search service will be implemented in Java and requires the Java Runtime Environment There are no requirements to acquire a license for commercial third party software Third party components are a Java virtual machine and several open source Java libraries like Log4J The implementation uses well known standards like HTTP and XML Documentation Documentation for developers will be delivered as code documentation and as a detailed document outlining the architecture and the interaction between the used components 3 2 Technical Documentation 3 2 1 3D search and retrieval use cases Two major use cases are identified in relation to the 3D search and retrieval service the 3D model indexing UC1 and the 3D search and retrieval UC2 In the following paragraph a brief description of each use case will be given 3D model indexing use case The following Figure describes the basic flow of events during the 3D model indexing Download Object Extract Descriptors Indexing 25 D2 2 5 V1 0 assots The use case begins with the Europeana Ingestion subsystem downloading the 3D model from the Content Provider The 3D model is sent to the Feature Extraction service which extracts the 3D descriptors and sends them back to the
78. uropeanald gt getResults int startFrom int numResults Returns the results of the query startFrom index to start QnumResults number of results to return If the value is set to 1 it returns all the query results It returns a List of Europeanald containing the ids of the query results Dependencies ASSETS Common 2 3 4 Software Packaging The image index software is 10096 Java code It consists of a jar library file some configuration files and some library dependences The software needs the following libraries Image index library melampo jar This is the main image index module Dependence libraries lire jar messif jar mifile jar conja jar trove jar mtree jar jama jar lucene jar vir jar Feature extraction and various utilities used in the main index module Configuration files assets ir image properties assets ir image fx properties indices properties LIRE MP7ALL properties 2 3 5 Installation and configuration The image index installation and configuration steps are quite simple The software consists of a set of jar libraries and some configuration files In order to install the image service the following configuration files will need to be set assets ir image properties assets ir image fx properties indices properties LIRE MP7ALL properties assets ir image properties sets the image index home path and the feature extraction configuration file I
79. velopers only Example relative URL http assetsdemo atc gr portal brief doc html query uploadAudio amp uploadSearchURL http audio bmat com audio 2 b bruce springsteen born to run 05 born to run mp3 ASSETS Scalable Content based indexing and ranking 58 D2 2 5 V1 0 assots Logged in as admin Saved tems U Saved searches U Saved tags U Log out My Europeana About us Communities Partners ThoughtLab Choose a language w Upload File Upload URL Basic Search D Similar to assets a ei see Ss Results 1 6 of 6 Page 1 share Refine your search By provider By language By country By date By type Eurovision Playing with Eurovision Playing with Europeana the WHAT Eurovision Todomondo Actions Save this search No image Click through the colour block to see the item We are still loading Eurovision Todomondo Europeana Master Figure 19 Audio Search by URL 4 4 4 Audio Description In order to see the audio descriptors 1 Mouse over the audio item result 2 Click on the i button It will display the complete track information including the audio descriptors for developers only Example relative URL http assetsdemo atc gr portal brief doc html query uploadAudio amp uploadSearchURL http audio bmat com audio 2 b bruce springsteen born to run 05 born to run mp3 Logged in as admin Saved items 0 Saved searches 0 Saved tags 0 Log out
80. void searchSimilar InputStream audioQueryObj Map lt String String gt filterObj throws AudioSearchingException Given an existing track InputStream usually not in the collection Query by Example it returns similar tracks 49 D2 2 5 V1 0 assots void searchSimilar URL audioQueryURL Map lt String String gt filterObj throws AudioSearchingException Given an existing track url usually not in the collection Query by Example it returns similar tracks Dependencies ASSETS Common 4 3 5 Figure 16 Audio Indexing Service Interfaces Software Packaging The audio search service is provided as follows 1 2 3 4 4 3 6 Ir audio service is packaged in a war library Ir audio client is packaged in a jar library Ir audio uses a jar library implemented for ASSETS and called jElla that wraps the calls to BMAT s Audio search engine Installation instructions for audio search engine and audio feature extractor which are both stand alone are documented below Audio feature extractor installation Technical Requirements Linux RedHat Centos 5 x Linux Debian Lenny Ubuntu Pentium Dual Core 1 86 GHz RAM Memory 4GB Storage space 100GB Note Storage and Memory requirements depends on the amount of tracks Installation Instructions for Redhat Centos 5 X 1 Most dependencies will be automatically resolved by yum Some of the packages might not be av
81. wish to offer the capability to their users via Europeana Preconditions e Images to be indexed are available through internet by URLs Basic Flow of Events ASSETS Scalable Content based indexing and ranking 4 D2 2 5 V1 0 assots 5 Finalize Image Indexing 6 Exception Handler Figure 1 Image indexing use case 0 Start The use case begins when the image search system is requested to add an image to the index 1 Image Download The image is downloaded using its given URL The image URL to be indexed is made available through the metadata of complex objects inserted in Europeana by the Content Providers 2 Features Extraction The requested content based features are extracted Feature extraction is a time consuming operation and is performed off line 3 Features Storage Automatic extracted features are stored The index will be built using the features previously stored For this reason in case of index rebuilding it will not be necessary to extract the features again However features are necessary only for the content based image searching and indexing and it is not necessary to store them with the other metadata 4 Features Indexing Features are indexed for content based searching An index is necessary to avoid comparison of the query image with all the images in the library Thus index is built for efficiency and it is typically not dynamic Dynamically updating the index may result in rebuilding

d2.2.5 scalable content-based indexing and ranking

Contents

Download Pdf Manuals

Related Search

Related Contents