WQLoader USER MANUAL
Contents
1. a User a valid QueueMetrics user holding the key WQLOADER a Password the access password for the user above a Partition or Token a logical place in the QueueMetrics database that your data will be written under Each Asterisk instance MUST be uploaded to a separate partition or subsequent data analysis will be incorrect 2 4 2 Running in Basic mode Running the loader in basic mode is quite simple When running manually all parameters should be specified from the command line The valid parameters are Known flags A KOS host name default 127 0 0 1 port port number default 8080 C context application context default queuemetrics W USE user name default webgloader w password password default gloader wgLoaderd pl flags my gueue log file partition name my activity log Flags not explicitly defined are set to the default values specified above As example pushing data into partition P001 to a cluster gueuemetrics live com host running on port 80 with context client4 with default robot robot username and password and logging to standard output would create a command line like wgloaderd pl a cluster gueuemetrics live com 0 80 e Clients A var log asterisk gueue log P001 2 4 3 Running in Advanced mode By running in Advanced mode you can feed multiple AueueMetrics instances from one single Asterisk box This way each instance can be in a physically separate loca2. sure the script runs on boot 2 3 1 Perl prerequisites Some perl extra packages are needed but they usually are available through yum installer on CentOS distributions Here is the list of needed extra packages fon CentOS 5 6 josie LMM ES perl JSON On Debian Ubuntu systems dependencies are available from libwww perl libmime tools perl libjson perl 2 3 2 Example installing on Ubuntu 14 04 The following seguence downloads and installs wgloader on an Ubuntu 14 04 server system apt get install libwww perl libmime tools perl libjson perl wget http downloads loway ch gm wgloaderd 1 4 tar gz eee AvE Wea A segus GFZ cd wgloaderd 1 4 chmod a x wgloaderd pl wqloaderd pl h cluster gueuemetrics live com p 80 c system13 u robot w hello var log asterisk queue_log POOL See below for the meaning of configuration parameters 2 4 Configuring the loader Depending on the user needs the loader runs in two different modes basic and advanced The basic mode is targeted at small single user systems where the advanced mode is targeted at large multi tenant systems 2 4 1 A bit of terminology You define a QueueMetrics system to be a target for receiving data by defining Installation e a Host the name or IP address of the machine that runs QueueMetrics a Port the TCP port QueueMetrics listens on a Context the name of the folder QueueMetrics appears to be in Then you would define
3. N sViLS RDS SD YE token null name Check High WaterMark where the field result is populated with the timestamp value 1379435567 that is Tue 17 Sep 2013 16 32 47 GMT Chapter 4 Frequently Asked Questions FAQs 4 1 How do you handle multi client clusters You may have two possible scenarios e You have multiple Asterisk servers and each client is handled on one Asterisk server You have multiple Asterisk servers and each client may be handled by any machine on the cluster In the first case you simply set up a one to many cluster as explained in the Advanced mode section of this manual In the second case you need to set up a partition for each Asterisk server on the client s machine As this data is processed by different Asterisk boxes it is important that no data from two distinct boxes is sent to the same partition 10
4. TTP port already has to be exposed and you do not need to handle TCP connections to MySQL Centralizing data loading into QueueMetrics also acts as a choke point to apply further security restrictions Additionally the wqloader offers automatic splitting of gueue log files in order to feed multiple separate QueueMetrics instances from the same Asterisk instance so that you can host multiple independent clients on the same Asterisk system or cluster The wqloader was built for e Maximum data safety your data is never lost even in case of catastrophic failures on the net or on the receiving QueueMetrics instance e High availability and continuous unattended operation Good performance even when having high or intermittent network latency e Running in parallel with an existing qloaderd The wgloader works with all versions of QueueMetrics after 14 06 3 Chapter 2 Installation 2 1 Prerequisite a valid QueueMetrics user In order to run wgloader you need to have a valid user in QueueMetrics holding the key WQLOADER that must be used for remote authentication Recent versions of QueueMetrics already include a user named webgloader with password gloader It is disabled by default so it has to be manually enabled from the GUI select Edit settings then Edit users edit it and sent Enabled to Yes While you are at it make sure you change the default password If your QueueMetrics instance does not have such a user defi
5. WQLoader USER MANUAL L Emilitri M Signorini Loway WQLoader USER MANUAL L Emilitri M Signorini Loway Table of Contents 1 Web Ol ader Ashot user guide 22 bsa Su asa kangen aa an aa needa sa In 0 1 2 Installation anna Kana bea bana ban Ta bana kang aan kau baba Kanaan 2 2 1 Prerequisite a valid AueueMetrics user iklan 2 2 2 Automated 1 1 aa a em gann anang 2 2 3 Manual installatiOM sedeln len senen anta ja coup aeda ad aknad db neue jk nene AR MAN KUMMA jan e adia A vi 3 2 4 Configuringithe loadef 3 3 31 nana 1111 11 3 6 API 3 ana hana nan banana banana Ban ba an java jaa 6 3 1 JSON Vectors fields na Aada ka 6 3 3 3 ma 3 131311 113 111 1 11 1 1 1 111111111 131 8 4 Freguently Asked Questions FAQS 4 sv kavade ne ja Levene ge 111111 60 10 4 1 How do you handle multi client clusters 020 10 Chapter 1 Web Qloader A short user guide The web qloader wgloader for short is a daemon written in Perl that uploads log events to a QueueMetrics instance over HTTP This has several advantages over the plain MySQL protocol used by the old gloaderd tool especially for physically distant systems or hosted systems the QueueMetrics H
6. dld a string populated with insertRow token See Section 3 1 1 Common fields e version see Section 3 1 1 Common fields e token a string containing the name of the token identifying the dataset actually partition set in QueueMetrics e timeld a string containing the row data timestamp as reported by asterisk callld a string containing the asterisk unique id for the row data queue a string containing the queue name field for the row data agent a string containing the agent field present in the row data verb a string containing the verb field present in the row data parameters string array containing other unnamed parameters present in the row data The expected array size is 5 meaning that parameters not available should be always present as empty strings QueueMetrics answers with a JSON vector containing the following fields commandid a string populated with insertRow token See Section 3 1 1 Common fields name a string containing a user readable name for this vector In this case is Insert Row e version see Section 3 1 1 Common fields e resultStatus see Section 3 1 1 Common fields e result a string containing a not null internal technical id associated to this event 1 5 Check Heartbeat This entry asks for an insetion of an heartbeat data line in the database Heartbeat events should be sent periodically to QueueMetrics to allow remote process survey The interval between di
7. fferent Heartbeat events is not defined but suggested as 900 seconds QueueMetrics expects to receive a JSON vector containing the following fields commandld a string populated with checkHb token See Section 3 1 1 Common fields e version see Section 3 1 1 Common fields token a string containing the name of the token identifying the dataset actually partition set in QueueMetrics e timeld a string containing the timestamp of this query in seconds starting from 01 01 1970 QueueMetrics answers with a JSON vector containing the following fields commandld a string populated with checkHb token See Section 3 1 1 Common fields e name a string containing a user readable name for this vector In this case is Check Heartbeat The network protocol API version see Section 3 1 1 Common fields e resultStatus see Section 3 1 1 Common fields e result a string containing a not null internal technical id associated to this event 3 2 HTTP Example As example below is shown a simple transaction dump Firts the Web Qloader sends a POST query to QueueMetrics In this example we can see the authorization field containing the base64 encoding of the username password pair and the COMMANDSTRING parameter populated with the HTTP encoded JSON object commandid checkHWM Our UTOKE ARO The reply is POST queuemetrics jsonQLoaderApi d
8. g containing the result of the current operation Valid values are specific for each operation type Values associated to this field should be discarded for events marked as KO in resultStatus commandid See paragraph Since protocol version checkHWM Check High Watermark 1 3 1 2 Check High Watermark This entry asks for the maximum timestamp already available on remote database for the specified token 0 0 0 0 QueueMetrics expects to receive a JSON vector containing the following fields commandld a string populated with checkHWM token See Section 3 1 1 Common fields e version see Section 3 1 1 Common fields token a string containing the name of the token identifying the partition to be used in QueueMetrics QueueMetrics answers with a JSON vector containing the following fields commandld a string populated with checkHWM token See Section 3 1 1 Common fields name a string containing a user readable name for this vector In this case is Check High WaterMark e version see Section 3 1 1 Common fields e resultStatus see Section 3 1 1 Common fields e result a string containing the highest timestamp found in the database for the specified token or the string null if no data are present 3 1 3 Check Row This entry asks for the presence of queue_1og data line in the database QueueMetrics expects to receive a JSON vector containing the following fields commandld a string pop
9. mple example of single rule is reported below ruleName gt Queues starting with Customer gueueStarts gt 1 queueEnds gt 0 queueToken gt Customer agentStarts gt 0 agentEnds gt 0 agemcloken gt taccgeclosic gt VIO 10 5 110 targetPort gt 8084 Installation BERGE targetUser gt robot targetPass gt robot targetToken gt P08 timezoneOffset gt 3600 This example generates a process able to handle queues where name is starting with Customer Events associated to that queues as either generic unassociated events are sent to the 10 10 5 110 8084 customer URL In this case gueueStarts should be set to 1 and the queueToken should be populated with Customer Tokens are case sensitive A different rule with gueueStarts set to 0 and queueEnds set to 1 will populate only queues with names ending in Customer Filtering by agents requires to properly set the keys agentStarts agentEnds agentToken in the same way As soon as the configuration file is set up the loader will be run by issuing the following command wgloader pl var log asterisk queue_log X The parameters are taken from the configuration file In this example a dummy parameter X in this case is needed as a placeholder if you want to specify a custom output log When the default output file var log asterisk qloader 1oq is used there is no need to specify any additi
10. ned create a new user with minimum privileges and give it the O key WQLOADER Test it by logging on and off manually 2 2 Automated yum installation On systems that run CentOS and derived systems including at the moment the all popular AsteriskNOW FreePBX and Elastix systems the wqloader can be installed by issuing the following commands as root wget P etc yum repos d http yum loway ch loway repo yum install y wgloaderd If the installation fails on older CentOS 5 systems you may have to install a missing package manually you could need to disable gpgcheck from yum s configuration such as in wget http pkgs repoforge org perl JSON perl JSON 2 50 1 e15 rf noarch rpm Kals ta joSsicl VSON 2Z 50 ISS iit moOaicela Lis When the runner starts you will have to edit its configuration located etc sysconfig wgloaderd to point it to your QueueMetrics instance and then restart it A valid configuration file will look like a AOS 127 005 1 HTTPPORT 8080 HTTPCONTEXT queuemetrics OMUSER webgloader OMPASS gloader PARTITION P001 OUEUELOG var log asterisk gueue log LOGFILE var log asterisk wqloaderd log LOCKFILE var lock subsys wqloaderd PIDFILE var run wqloaderd pid You usually have to edit the HTTP parameters and the user name and password 2 2 1 Starting and stopping To start the wqloaderd you simply type etc init d wgloaderd start And to
11. o HTTP 1 1 TE deflate gzip g 0 3 Connec ons Tit Close Authorization Basic ZGVtb2FkbWluOmR1bW8 Hosts 10 105 LILO s SO User Agent libwww perl 5 833 Content Length 106 Content Type application x www form urlencoded COMMANDS TRING 3 7B 22command1d 22 3A 22checkHWM 22 2C 22version S22 3A 221 0 22 2C S22tokens22 3AS22P08S22 7D QueueMetrics answers with the following JSON vector ETTEV IST 2090 Ox Server Apache Coyote 1 1 Set Cookie JSESSIONID 94C0990E61F67440D95 Path queuemetrics HttpOnly Content Type application json encoding UTF 8 charset ISO 8859 1 Content Length 119 Date Thu 07 Aug 2014 09 36 37 GMT Connection close commandId checkHWM Lesson Ov SE SS LB NE 117 token null name Check High WaterMark In this example the field result is populated with null meaning that there is no data for the required token If data are already present for the reguired token the answer resembles to the following HTTP 1 1 200 OK Server Apache Coyote 1 1 Set Cookie JSESSIONID F517168E427059693B73 Path queuemetrics HttpOnly Content Type application json encoding UTF 8 charset ISO 8859 1 Content Length 127 Date Thu 07 Aug 2014 09 50 27 GMT Connection close commandid checkHWM The network protocol API SS ES IS RS OF SC SE B
12. onal command line parameters wqloader pl var log asterisk queue_log Chapter 3 The network protocol API Conversations between wgloader and QueueMetrics used JSON as high level protocol In this section is a list of JSON vectors involved in the process The Web Gloader process is responsible for starting all transactions Each transaction is wrapped into a single HTTP POST with Basic Authentication targeted to an end point named jsonOLoaderApi do Basic Authentication requires to have a first unauthenticated HTTP transaction followed by a HTTP 401 Not Authorized server response code and a second client HTTP request containing authorization headers QueueMetrics requires to have a single client transaction with authorization headers already present 3 1 JSON Vectors fields 3 1 1 Common fields QueueMetrics does not support the standard Basic Authentication method but a slightly modified version Where standard All JSON vectors contain some common fields used by QueueMetrics and or the Web Qloader to persist information like for example the protocol version and answer results and statuses commandld a string containing the command identifier Valid values are specified in the table below version a string containing current protocol version Valid values are 1 0 e resultStatus a string containing the operation result status Valid values are OK or KO followed by an error message description result a strin
13. stop it you type etc init d wqloaderd stop You may also force a restart e g after a change of configuration by issuing etc init d wgloaderd restart When you install it using yum the wqloader is scripted to start automatically on boot Installation 2 2 2 Monitoring To check that the system is running as expected just enter tail f var log asterisk wgloaderd log It will print out something like Thu Aug 7 09 30 56 2014 OueueMetrics Web loader Revision 1 7 Thu Aug 7 09 30 56 2014 Evaluating wgloader regexp pl Thu Aug 7 09 30 56 2014 Evaluating wgloader cluster pl Ta Ang 7 09230256 ZOLA 1 elvuseez sulas lt Han Atg 7 99230255 20M Nend 1069 Wokeme POOL TA Orrsete Q So 90 Thu Aug 7 09 30 56 2014 PID 1068 Target host URI demo gueuemetrics live com 80 mys Thu Aug 7 09 30 56 2014 Ignoring all timestamps below 0 It will also show any errors it should encounter while running In general when the loader encounters any error it just waits a few seconds and retries 2 3 Manual installation The general procedure you should follow is Download and unpack the latest version of wgloader from the downloads section of the QueueMetrics website Make sure all Perl requisites are met Create a script to run the wgloader passing all required parameters see the Configuring section Make
14. tion and can be configured and possibly resold independently In order for this to work the prerequisite is that you use a common naming schema for all agents and queues be it gueue1 client1 and Agent 100 client1 or client1 queue1 and Agent client1 100 Then you tell wgloaderd which prefixes or suffixes apply to each QueueMetrics instance and give it all the details to connect to it You do this by configuring a set of rules by editing the file wgloader cluster pI1 that resides in the same folder where the main script is stored Within the wgloader cluster pI file is a set of structures that are to be uncommented and or edited according to your needs For each defined rule the loader starts a process that will be in charge of pushing data to the defined host for that rule There is no limit on the number of rules that can be defined except the CPU and RAM amount needed to handle all the processes that will be generated Each rule defines a set of filtering criteria Two types of filters are availables filters by gueue and filters by agent The filters are applied to each loader row as a first match iptables style starting by queue information Filters are to be individually enabled through configuration Rows without queue or agent information i e containing NONE on these fields are pushed to all servers This allows to split single multi tenant log into several single tenant QueueMetrics instances A si
15. ulated with checkRow token See Section 3 1 1 Common fields e version see Section 3 1 1 Common fields e token a string containing the name of the token identifying the dataset actually partition set in QueueMetrics e timeld a string containing the row data timestamp as reported by asterisk The network protocol API 3 callld a string containing the asterisk unique id for the row data queue a string containing the queue name field for the row data agent a string containing the agent field present in the row data verb a string containing the verb field present in the row data parameters a string array containing other unnamed parameters present in the row data The expected array size is 5 meaning that parameters not available should be always present as empty strings QueueMetrics answers with a JSON vector containing the following fields commandld a string populated with checkRow token See Section 3 1 1 Common fields name a string containing a user readable name for this vector In this case is Check Exsisting Row e version see Section 3 1 1 Common fields e resultStatus see Section 3 1 1 Common fields result a string containing the number of matching data lines found in the database 1 4 Insert Row This entry asks for the insertion of queue_log data line in the database QueueMetrics expects to receive a JSON vector containing the following fields comman
Download Pdf Manuals
Related Search