(N)ERD service manual
Contents
1. Stri shal lt password gt ee Type of request URL Request input type ted application x www form urlencoded admin shal lt pwd gt shal application x www form urlencoded shal shal lt input string gt application x www form allProperties encoded hal Administration sick ies allProperties GET ehangePropetty Value application x www form urlencoded changeProperty Value Strin xml lt some xml gt 8 Response output Description type Request to get parameters of nerd properties and nerd_service properties formatted in html table Request to get an input string hashed using shal Request to get all properties key value type as xml Sent xml follow the following schema lt properties gt lt property gt text xml lt key gt key lt key gt lt value gt value lt value gt lt type gt type lt type gt lt property gt lt property gt lt property gt lt properties gt Change the property value from the property key passed in the xml input Xml input has to follow the following schema lt changeProperty gt lt password gt pwd lt password gt text xml lt property gt lt key gt key lt key gt lt value gt value lt value gt lt type gt type lt type gt lt property gt lt changeProperty gt text html 5 2 Language identification For preliminary check a language identification service is available2. Requesting Parameter MIME Type Type of request URL ype eek Response output Description Request input type type Identify the language of a fragment of text Language processLIdText POST PUT text multipart form data application json eer ON grr late agag OR identification and confidence score lang en conf 1 gt curl X POST http localhost 8090 service processLIdText text Bonjour gt lang fr conf 0 7142846023142064 5 3 Sentence segmentation This service segments a text into sentences It is useful in particular for the interactive mode for indicating that only certain sentences need to processed for a given query Beginning and end of each sentence are indicated with offset positions with respect to the input text Requesting Parameter MIME Type type name gae Type of request URL Response output Description Request input type type Segment a text into sentences Sentence p ate Answer JSON with offsets for each sentences processSentenceSegmentation POST PUT text String application json sentences segmentation offsetStart 0 offsetEnd 6 offsetStart 6 offsetEnd 21 gt curl X POST http localhost 8090 service processSentenceSegmentation text I eat Then I spleep gt sentences offsetStart o offsetEnd 6 offsetStart 6 offsetEnd 21 5 4 NERD tex
3. 5 4 1 processNERDText and processERDText This is the base NERD ERD service to be used on raw text fragment Note that the language identifier will be applied as a starting point of the process to ensure that the language of the text is supported by the system In the current version only English French and German are supported by the ERD services and only English is supported by the NERD services If the language of the text is not supported an http error 406 request not acceptable is sent back The different possible parameters are described bellow The two services processNERDText and processERDText use exactly the same arguments parameters and differ only with respect to the explanations given in 5 4 Requesting type Parameter MIME Type Type of request Description Request input type ae onse output text onlyNER processNERDText nbest processERDText sentence format customisation Perform a Named Entity Recognition and Disambiguation on a text identify entities and classify application json them in term of NER types and when possible subtypes Entity resolution against Wikipedia amp FreeBase Named Entity Recognition and Disambiguation The request is an HTTP GET or POST with the following query parameters text Text to be processed in UTF 8 Mandatory Default none onlyNER Boolean indicating if the process should be limited to Named Entity Recognition without disambiguation and resolution a
4. It will generate 2 artifacts 1 for the text mining data nerd data lt version gt zip and the deployable war fr inria nerd lt version gt war nerd target nerd data lt version gt zip nerd target fr inria nerd lt version gt war Copy these 2 artifacts to your remote server nerd data lt version gt zip contains the needed native libraries the models lexicons gazetteers and a config directory that contains 2 properties files nerd properties and nerd_service properties You have to unzip nerd data wherever you want on your server unzip nerd data lt version gt zip In fr inria nerd lt version gt war the file web xml has 3 parameters to set before starting the server e fr inria nerd property path to nerd property e fr inria nerd property service path to nerd_service properties e fr inria nerd home path to nerd home These properties are filled by the following variables _NERD PROPERTY NERD SERVICE PROPERTY and NERD HOME so that it is possible to fill these values with a script given the environment It is also possible to set manually these variables 4 Use of N ERD console Welcome page is available at http lt server instance name gt lt root context name gt i e for local tomcat http localhost 8080 lt name of the war deploy in webapp gt MIME Type Requesting Parameter type name Request input Response type outputype Description Gives a very brief descripti
5. Computer scientists invent algorithm ic processes that create describe and transform information and formulate suitable abstraction computer science abstraction s to model complex systems source wikipedia en lang en term engine score 0 1 entities rawName engine preferredTerm Engine nerd_score 0 001837127251406647 prob 1 0 wikipediaExt ernalRef 9640 freeBaseExternalRef m 02mk9 definitions definition An engine or motor is a machine designed to convert energy into useful Motion physics mechanical motion Devices converting heat energy into motion are referred to as engines which come in many types A common type is a heat engine such as an internal combustion engine which typically burns a fuel with air and uses the hot gases for generating power External combustion engine s such as steam engine s use heat to generate motion via a separate working fluid source wikipedia en lang en language lang fr conf 0 857140003447354 5 6 ERD search query processing 5 6 1 processERDSearchQuery The N ERD service can disambiguate a search query expressed as a short text Search query disambiguation uses a special model optimized for a small number of terms and trained with search queries The difference between standard text and short text is similar to the one of the
6. ERD 2014 challenge http web ngram research microsoft com erd2014 Docs Detail 20Rules pdf Requesting Parameter MIME Type type name noe Type of request YP Response output Description Request input type ies Perform a disambiguation on a search query entity processERDSearchQuery GET POST application json application json resolution against Wikipedia amp FreeBase of the search terms in the global context of the search query Disambiguation of a search query The JSON format for the query parameter to be sent to the service must follow the following template text concrete pump sensor language lang Van nbest 0 format JSON customisation generic The fields nbest language format customisation resultLanguages are optional and defined similarly in section 5 4 2 and are set to their default values gt curl X POST http localhost 8090 service processERDSearchQuery d text concrete pump sensor language lang en conf 1 0 nbest 0 WwW e gt runtime 84 onlyNER false nbest false text concrete pump sensor language lang conf 1 0 entities rawName concrete preferredTerm Concrete offsetStart 0 offsetEnd 8 ged 0 988974165793762 ner_conf 0 8 prob 1 0 wikipediaExternalRef 5371 freeBaseExternalRef m oimxf definitions definition Concrete is
7. N ERD service manual I RIV CITC W manene snien a tute cue A tapes Meaenah aed E a poe E 1 2 Build and set up environment for local deployment s onsesseeseeeeseesseeseessseesssesseeseresseessrssersssessees 1 3 Build and set up environment for remote deployment snsesseeesesseeseeseessressesreesetssresseesersssessres 1 di Useof NERD Console hena a tea E E A G e A aT 2 J NERD RPS TAP r a ah A E n Oak Utd Eada a eed e ai 4 6 REST API response examples eres aa e TERE tig E O EER A 18 Authors Patrice Lopez Last update 02 10 2015 1 Overview N ERD Named Entity Recognition and Disambiguation includes a RESTful service implementation for consuming the entity recognition and disambiguation processes 2 Build and set up environment for local deployment To build the N ERD service for local deployment you just have to go to the root of the project and run the following command gt cd nerd gt mvn clean install Then deploy the generated war to the server The artifact is under nerd target fr inria nerd lt version gt war As an alternative it is also possible to quick start and test the service with jetty gt mvn Dmaven test skip true jetty run war 3 Build and set up environment for remote deployment To build N ERD for remote deployment you have to go to the root of the project and run the following command mvn clean install PgenericBuild
8. a fieompodte eel construction material composed of cement commonly Portland cement and other cementitious materials such as fly ash and slag cement construction aggregate aggregate generally a coarse aggregate made of gravel or crushed rocks such as limestone or granite plus a fine aggregate such as sand water properties water and Chemistry chemical admixtures source wikipedia en lang en domains Materials Engineering Aviation Architecture categories source wikipedia en category Pavements page_id 1297865 source wikipedia en category Concrete page_id 2238414 source wikipedia en category Building materials page_id 3962842 source wikipedia en category Masonry page_id 3975663 source wikipedia en category Sculpture materials page_id 10308784 rawName concrete pump preferredTerm Concrete pump offsetStart o offsetEnd 13 Service to call ERD search query 0 example_1 Reuters_1 text concrete pump sensor example_2 Reuters 2 language b peg zi example_3 Reuters_3 example_4 Reuters_4 Submit Annotations Response concrete Conf 0 98 wW Concrete is a composite construction material composed of cement commonly Portland cement and other cementitious materials such as fly ash and slag cement a
9. _1 example_4 Reuters_4 sent Armcatons Response Fig 6 1 Viewing service response Named Entity Recognition service Input Output
10. ase for term vector each term in the global context of the vector The JSON format for the query parameter to be sent to the service must follow the following template termVector term computer science score 0 3 term engine score 0 1 language lang Yan hy resultLanguages de nbest 0 format JSON customisation generic The fields nbest language format customisation are optional and defined similarly in section 5 4 2 and are set to their default values The termVector field is required for having a well formed query resultLanguages can be set to get wikipedia pages for languages in addition to the language of the input terms gt curl X POST http localhost 8090 service processERDQueryTerms d termVector term computer science score 0 3 term engine score 0 1 nbest 0 gt runtime 33 nbest false termVector term computer science score 0 3 entities rawName computer science preferredTerm Computer science nerd_score 0 20149253731343286 prob 1 0 wikipediaExternalRef 5323 freeBaseExternalRef m O1lmkq definitions definition Computer science or computing science abbreviated CS is the study of the theoretical foundations of information and computation and of practical techniques for their implementation and application in computer systems
11. d grobid home config grobid propert ies fr inria nerd proxy_port null fr inria nerd elasticSearch_host localhost fr inria nerd lang impl CybozuLanguageDetectorFactory fr inria nerd grobid_properties Fig 4 3 Service administration The web page Service Fig 4 2 allows to test the different REST requests quickly In the source code the class NerdRestService is the entry point for each rest service of the N ERD services The administration section Admin Fig 4 3 allows to manage dynamically the properties contained in nerd properties and nerd_service properties Finally the Doc section gives access to the present manual 5 N ERD REST API 5 1 Administration services These services make possible the modification the property values dynamically while the server is deployed For instance it is possible to modify the nerd_home path or the ElasticSearch instance when new models and resources are available indexed This is a standard production requirement The table below shows the provided resources corresponding to the HTTP verbs to use the administration services All url described bellow are relative path the root url is Attp lt server instance name gt lt root context gt service The console web application calls these services and can be used to administrate the N ERD deployment or can be exploited as javascript reference client implementation Parameter Requesting MIME Type
12. es the Allies based on the Triple Entente of the United Kingdom France and the Russian Empire and the Central Powers of Germany and Austria Hungary description Customisation for World War 1 domain Name is the identifier of the customisation The context will be build based on Wikipedia articles FreeBase entities and raw texts which are all optional Wikipedia articles are expressed as an array of Wikipedia page ID FreeBase entities are given as an array of mid the FreeBase Machine IDs Finally texts are represented as an array of raw text segments Response status codes HTTP Status Code Reason 200 Successful operation 400 Wrong request 404 Indicates that the customisation resource was not found 500 Indicate an internal service error In case of error status the service returns a JSON object with an error message gt curl X POST http localhost 8090 service createNERDCustomisation wwit d wikipedia 4764461 gt curl X GET http localhost 8090 service NERDCustomisations gt ww1 gt curl X GET http localhost 8090 service getNERDCustomisation ww1 nn gt name ww1 wikipedia 4764461 6 REST API response examples It is possible to view the service response with the Web Service console as shown by Figure 6 1 example_2 Reuters_2 example_3 Reuters_3 Austria invaded and fought the Serbian army at the Battle of Cer and example_1 Reuters
13. ey exist currently only English German and French wikipedia are supported In this example the processNERDText service optional parameters onlyNER nbest sentence format customisation are set to their default values they remain optionals The entities attribute is here empty which means that there is no pre defined annotation Apart from the source language which is pre set consequently the language is considered certain and the language identifier will not be used and additional target languages of results this example is similar to a processNERDText request but in a JSON format 2 In the following example a pre defined entity typically pre annotated by a user is present in the entities attribute text Austria invaded and fought the Serbian army at the Battle of Cer and Battle of Kolubara beginning on 12 August language lang Yan entities rawName Austria type LOCATION offsetStart 0 offsetEnd 7 wikipedia 26964606 l3 In a typical interactive scenario an application client first sends a text to be processed via the processNERDText service and receives a JSON response with some entities The annotated text is displayed to a user which might correct some invalid annotations The client updates the modified annotations in the first JSON response and can send it back to the service now as new query via the processNERDQuery The corrected annotations will
14. gainst Wikipedia and Freebase Performing only NER results in much faster processing time Optional Default false nbest Boolean indicating if only the best disambiguation results should be returned for an identified entity or several best hypotheses Optional Default false sentence Boolean indicating if a sentence segmentation should be present in the returned result Optional Default false format String indicating if the response format is JSON only or JSON TEI Optional Default JSON customisation Indicate the name of a domain customisation or generic if no particular customisation is used Optional Default generic gt curl X POST http localhost 8090 service processNERDText text John Smith amp onlyNER true gt text John Smith runtime 3 language lang de conf 0 5714286566545531 entities rawName John Smith type PERSON offsetStart o offsetEnd 10 conf 0 8 prob 1 0 5 4 2 processNERDQuery and processERDQuery This NERD service supports interactive usage where a user can manually pre annotate or correct some annotations and the client application might sends the corrected annotated text to the service several time There is only a unique parameter which is a JSON string representing a query This query is similar to the base response of the NERD ERD service When annotations are present in the query the NERD system will con
15. ggregate generally a coarse z aggregate made of gravel or crushed rocks such as limestone or granite plus a fine aggregate such as sand water and chemical admixtures concrete pump Conf 0 98 W A concrete pump is a tool used for transferring liquid concrete by pumping There are two types of concrete pumps The first type of concrete pump is attached to a truck It is known as a trailer Dorlas pam me so Ean it uses a remote controlled aaa oe am called a Fig 5 Search query disambiguation console 5 7 Customisation API It is possible to use a customisation to specialise the entity recognition disambiguation and resolution for a particular domain This API allows to manage customisations for the N ERD instance which can then be used a parameter by the N ERD services Requesting type Parameter MIME Type Type of request Description Request input type ne onse output List the existing aoe Return the list of existing customisations as a JSON ae NERDCustomisations fe customisations array of customisation names Get the NERDCustomisation Return the JSON profile of an existing customisation information of a GET N A N A application json p 8 application json aes name identifi its name as path parameter customisation j dentified by its name as path paramete Create a customisation as defined in the input JSON named following the path parameter Create a createNERDCustomisation ee Be phase The JSON
16. ionary Force under Sir John French 1 A series of engagements known as the Battle of the Frontiers ensued 2 Key battles included the Battle of Charleroi and the Battle of Mons 3 In the former battle the French 5th Army was almost destroyed by the German 2nd and 3rd Armies and the latter delayed the German advance by a day 4 A general Allied retreat followed resulting in more clashes such as the Battle of Le Cateau the Siege of Maubeuge and the Battle of St Quentin Guise After marching through GERI and the GEES the G GY advanced in the latter BRITISH EXPEDITIONARY FORCE half of into northern GIN where they met both the army under and the initial six divisions of the E7 SSS under Sl EES Type ORGANISATION conf 0 8 Ennewnss Branch in Wenner oF oun Cownanes Fig 4 2 Test Rest Interface NERD About Services Admin Doc Doc Enter administrator password Welcome to the admin console Property value fr inria nerd service admin pw d033e22ae348aeb5660fc2140aec35850c4da997 fr inria nerd KnowledgeBaseESName erd2014 fr inria nerd grobid_home Users lopez grobid segmentat ion work grobid grobid home fr inria nerd elasticSearch_port 9200 fr inria nerd nerd_home fr inria nerd cache_rest true fr inria nerd mongodb_host localhost fr inria nerd language_detector_facto ry fr inria nerd proxy_host null fr inria nerd mongodb_port 27017 Users lopez grobid segmentat ion work grobi
17. ms of fundamentals said of EN n CEE pronounced is a Federation federal constitutional republic Now we re at the point where the fundamentals are with us The history is now falling out of view in North Americal It is bordered on the north by the United States on the south and west by the Pacific Ocean on the That history is one etched into the minds of all investors in Wg an economy in crisis since southeast by Guatemalal Belize and the Caribbean Seal a free falling peso and stubbornly high interest rates NERD About Services Admin Doc Service to call NERD text sson Generic onlyNER J sentence segmentation short text _ nbest After marching through Belgium Luxembourg and the 5 the German Army advanced in the latter half example_1 Reuters_1 of August into northern France where they met both the French army under Joseph Joffre and the initial six divisions of the British Expeditionary Force under Sir John French example_2 Reuters_2 example_3 Reuters_3 Aseries of engagements known as the Battle of the Frontiers ensued Key battles included the Battle of j example_4 Reuters_4 Submit Annotations Response 0 After marching through Belgium Luxembourg and the Ardennes the German Army advanced in the latter half of August into northern France where they met both the French army under Joseph Joffre and the initial six divisions of the British Expedit
18. nenberg But the failed Russian invasion causing the fresh German troops at the First Battle of the Marne processSentence Sentences offsetStart 0 1 rd offsetEnd 163 offsetStart 163 offsetEnd 319 to move to the east allowed the tactical Allied victory entities rawName Russian type NATIONAL offsetStart 179 offsetEnd 186 etc gt curl X POST http localhost 8090 service processNERDQuery d text John Smith onlyNER true gt text John Smith runtime 3 language lang de conf 0 5714286566545531 entities rawName John Smith type PERSON offsetStart o offsetEnd 10 conf 0 8 prob 1 0 5 5 ERD term vector processing 5 5 1 processERDQueryTerms The N ERD service can process a weighted vector of terms Each term will be disambiguated when possible in the context of the complete vector The client must respect a JSON format encoding the weighted term vector as query as described bellow Requesting Parameter MIME Type type name eae Type of request yP i Response output Description Request input type ene Disambiguation Perform a disambiguation on a weighted term vector of a weighted processERDQueryTerms GET POST application json application json entity resolution against Wikipedia amp FreeB
19. on about the N ERD service nerd text html This welcome resource is accessed by the About section Fig 4 1 From there it is possible to access the service interface Service Fig 4 2 and the administration section Fig 4 3 NERD About Services Admin Doc About NERD Named Entity Recognition and Disambiguation Contact Patrice Lopez Fig 4 1 About NERD About Services Admin Doc Service to call NERD text z yson Generic _ ony NER sentence segmentation shorttext _ nbest MEXICO Recovery excitement brings Mexican markets to life example_1 Reuters_1 Hony nons example_2 Reuters_2 MEXICO CITY e3 R z Emerging evidence that Mexico s economy was back on the recovery track sent Mexican markets into a buzz oon euters_ of excitement Tuesday with stocks closing at record highs and interest rates at 19 month lows example_4 Reuters_4 Submit Annotations Response CE Recovery excitement brings markets to life Tricks Type LOCATION CITY Sense country N1 Emerging evidence that QEIZJ s economy was back on the recovery track sent markets into a buzz conf 0 8 of excitement EMESIS with stocks closing at record highs and interest rates at month lows CE has been trying to stage a recovery since the beginning of this year and it s always been getting ahead The United Mexican States commonly known as Mexico of itself in ter
20. profile specifies a context via the customisation name POST eve proie a i epp caion jaon oe ofa list of Wikipedia article IDs FreeBase entity mid and text fragments A text describing informally the customisation can be added optionally following the path parameter by additional context information extendNERDCustomisation POST PUT profile application json application json The additional context is given by the combination of a name list of Wikipedia article IDs FreeBase entity mid and text fragments which will be merged with the existing one A text describing informally the customisation can be added optionally Delete a NERDCustomisation byes Delete an existing customisation identified by its name hee DELETE name String application json customisation name provided as path parameter Extend the definition of a customisation named Extend a customisation The JSON profile of a customisation to be sent to the server for creation and extension has the following structure wikipedia 4764461 51499 1014346 freebase m Ocm2xh m 0d14z m 02kxg_ m O06v9th texts World War WWI or WW1 or World War One also known as the First World War or the Great War was a global war centred in Europe that began on 28 July 1914 and lasted until 11 November 1918 The war drew in all the world s economic great powers which were assembled in two opposing allianc
21. sider them certain and ensure that the user annotations will be present in the output response without inconsistencies with other annotations exploit the user annotations to improve the context for identifying and disambiguating the other possible entities The client must respect the JSON format of the NERD ERD response as new query as described bellow Requesting Parameter MIME Type type name ere Type of request YP Response output Description Request input type eine Perform a Named Entity Recognition and Disambiguation on a text identify entities and classify application json application json them in term of NER types and when possible subtypes Entity resolution against Wikipedia amp FreeBase Named Entity Recognition and Disambiguation processNERDQuery processERDQuery 1 The JSON format for the query parameter to be sent to the service is identical to a response of the service The parameters of the processNERDText are attribute of the JSON which typically follow a template like the following one text The text to be processed language Alang ken hy Wentities resultLanguages fr de onlyNER false nbest 0 sentence false format JSON customisation generic An additional available parameter is resul tLanguages This parameter is a list of language codes and permits to get the wikipedia pages in additional languages if th
22. t processing Two services performing named entity recognition and disambiguation are available processNERDText for processing raw text fragment resulting in a structured response containing the different annotations processNERDQuery dedicated to interactive applications which consumes as input a structured JSON query following the format of the N ERD annotated text and which produces a re processed structured response These services are restricted to a set of 26 classes of names entities see https github com kermitt2 grobid ner wiki Grobid NER classes and senses If covered by Wikipedia FreeBase the service will try to disambiguate the recognized named entities Entities not covered by these knowledge bases will be characterized by an entity class a word sense estimation and a confidence score In addition two similar services are provided performing free disambiguation Without entity class restriction the service tries to disambiguate all the entities covered by Wikipedia FreeBase However in this case the entities not present in the knowledge base will not be recognized and characterized at all processERDText for processing raw text fragment resulting in a structured response containing the different annotations processERDQuery dedicated to interactive applications which consumes as input a structured JSON query following the format of the ERD annotated text and which produces a re processed structured response
23. then be exploited by the N ERD system to possibly improve the other annotations and disambiguations 3 To support addition of text by a user e g note taking environment where the N ERD service is called continuously in background it is possible to indicate the N ERD system to process only certain sentences of the input text The entity disambiguation and resolution will still consider the entire text and the previous annotations when processing only the indicated sentences For this purpose an additional attribute processSentence is possible when calling the service processNERDQuery The parameter attribute provides the list of sentences to be processed text The army led by general Paul von Hindenburg defeated Russia in a series of battles collectively known as the First Battle of Tannenberg But the failed Russian invasion causing the fresh German troops to move to the east allowed the tactical Allied victory at the First Battle of the Marne processSentence 1 When processSentence is set a sentence segmentation always occur whatever the value of the attribute sentence In this example only the second sentence will be the object of the NERD processing It is possible to express sentences as interval e g 2 5 10 text The army led by general Paul von Hindenburg defeated Russia in a series of battles collectively known as the First Battle of Tan
Download Pdf Manuals
Related Search