WQLoader USER MANUAL
Contents
1. You usually have to edit the HTTP parameters and the user name and password 2 2 1 Starting and stopping To start the wqloaderd you simply type etc init d wgloaderd start And to stop it you type etc init d wqloaderd stop You may also force a restart e g after a change of configuration by issuing etc init d wgloaderd restart When you install it using yum the wqloader is scripted to start automatically on boot Installation 2 2 2 Monitoring To check that the system is running as expected just enter tail f var log asterisk wgloaderd log It will print out something like Thu Aug 7 09 30 56 2014 OueueMetrics Web loader Revision 1 8 Thu Aug 7 09 30 56 2014 Evaluating wgloader regexp pl Thu Aug 7 09 30 56 2014 Evaluating wgloader cluster pl Wau Woe 7 09530255 2014 1 Elusese sule Ss Thu Aug 7 09 30 56 2014 PID 1068 Token P001 TZ offset 0 s Heartbeat after 90 Thu Aug 7 09 30 56 2014 PID 1068 Target host URI demo gueuemetrics live com 80 mys Thu Aug 7 09 30 56 2014 Ignoring all timestamps below 0 It will also show any errors it should encounter while running In general when the loader encounters any error it just waits a few seconds and retries 2 3 Manual installation The general procedure you should follow is e Download and unpack the latest version of wgloader from the downloads section of the QueueMetrics website2. Make sure all Perl requisites are met Create a script to run the wgloader passing all required parameters see the Configuring section Make sure the script runs on boot 2 3 1 Perl prereguisites Some perl extra packages are needed but they usually are available through yum installer on CentOS distributions Here is the list of needed extra packages fon CentOS 5 6 DE LAMM OOLE perl JSON On Debian Ubuntu systems dependencies are available from libwww perl libmime tools perl libjson perl 2 3 2 Example installing on Ubuntu 14 04 The following seguence downloads and installs wgloader on an Ubuntu 14 04 server system apt get install libwww perl libmime tools perl libjson perl wget http downloads loway ch gm wgloaderd 1 4 tar gz eee EXE WL 1 5 4 Cena GFZ cd wgloaderd 1 4 chmod atx wgloaderd pl wgloaderd pl h cluster gueuemetrics live com p 80 system13 u robot w hello var log asterisk gueue log P001 See below for the meaning of configuration parameters 2 4 Configuring the loader Depending on the user needs the loader runs in two different modes basic and advanced The basic mode is targeted at small single user systems where the advanced mode is targeted at large multi tenant systems Installation 2 4 1 A bit of terminology You define a QueueMetrics system to be a target for receiving data by defining e a Host the name or IP address of the machine
3. UVE nS TOn 3 On EE 208 The reply is POST queuemetrics jsonQLoaderApi do HTTP 1 1 TE deflate gzip q 0 3 Connection 12 Close Authorization Basic ZGVtb2FkbWluOmR1bW8 Hoses 10 105 0 User Agent libwww perl 5 833 Content Length 106 Content Type application x www form urlencoded COMMANDS TRING 3 7B 22command1d 22 3A 22checkHWM 22 2C 22version 22 3A 221 0 22 2C 22token 2233A 22aleph 2237D QueueMetrics answers with the following JSON vector kuu i 290 Ox Server Apache Coyote 1 1 Set Cookie JSESSIONID 94C0990E61F67440D95 Path queuemetrics HttpOnly Content Type application json encoding UTF 8 charset ISO 8859 1 Content Length 119 Dasss Wau 07 Aug 2914 99336537 CNIL Connection close commandId checkHWM SC SEO TA Ou WieesilheSicene wie 3 oki ES SE ETON I Wie Olesya STL IL A name Check High WaterMark In this example the field result is populated with null meaning that there is no data for the required token If data are already present for the required token the answer resembles to the following Iai i 200 OK Server Apache Coyote 1 1 Set Cookie JSESSIONID F517168E427059693B73 Path gueuemetrics HttpOnly Content Type application json encoding UTF 8 charset ISO 8859 1 Content Length 127 Date Thu 07 Aug 2014 09 50 27 GMT Connection close commandIid checkHWM Vera sil Ow ES
4. scenes s MOU ur 5 DSS token null name Check High WaterMark where the field result is populated with the timestamp value 1379435567 that is Tue 17 Sep 2013 16 32 47 GMT 10 Chapter 4 Frequently Asked Questions FAQs 4 1 How do you handle multi client clusters You may have two possible scenarios e You have multiple Asterisk servers and each client is handled on one Asterisk server You have multiple Asterisk servers and each client may be handled by any machine on the cluster In the first case you simply set up a one to many cluster as explained in the Advanced mode section of this manual In the second case you need to set up a partition for each Asterisk server on the client s machine As this data is processed by different Asterisk boxes it is important that no data from two distinct boxes is sent to the same partition 11
5. that runs QueueMetrics a Port the TCP port QueueMetrics listens on a Context the name of the folder QueueMetrics appears to be in Then you would define a User a valid QueueMetrics user holding the key WQLOADER a Password the access password for the user above a Partition or Token a logical place in the QueueMetrics database that your data will be written under Each Asterisk instance MUST be uploaded to a separate partition or subseguent data analysis will be incorrect 2 4 2 Running in Basic mode Running the loader in basic mode is guite simple When running manually all parameters should be specified from the command line The valid parameters are Known flags A Os 2 mesic name Geraec2 127 00 por port number default 8080 c context application context default gueuemetrics U USET user name default webgloader w password password default gloader wgLoaderd pl flags my gueue log file server name my activity log Flags not explicitly defined are set to the default values specified above As example pushing data for server aleph to a cluster gueuemetrics live com host running on port 80 with context client4 with default robot robot username and password and logging to standard output would create a command line like Juedloedere jal a elusees epsusmsestes liva eem a 80 e clients var log asterisk gueue log aleph by specifying an empty server nam
6. GUI select Edit settings then Edit users edit it and sent Enabled to Yes While you are at it make sure you change the default password If your QueueMetrics instance does not have such a user defined create a new user with minimum privileges and give it the O key WQLOADER Test it by logging on and off manually 2 2 Automated yum installation On systems that run CentOS and derived systems including at the moment the all popular AsteriskNOW FreePBX and Elastix systems the wgloader can be installed by issuing the following commands as root wget P etc yum repos d http yum loway ch loway repo yum install y wgloaderd If the installation fails on older CentOS 5 systems you may have to install a missing package manually you could need to disable gpgcheck from yum s configuration such as in wget http pkgs repoforge org perl JSON perl JSON 2 50 1 e15 rf noarch rpm ym Kals OS SS ON 50 1 iit moOaicela Lis When the runner starts you will have to edit its configuration located in etc sysconfig wgloaderd to point it to your QueueMetrics instance and then restart it A valid configuration file will look like a POS 1 HTTPPORT 8080 HTTPCONTEXT queuemetrics OMUSER webgloader OMPASS gloader PARTITION P001 OUEUELOG var log asterisk gueue log LOGFILE var log asterisk wqloaderd log LOCKFILE var lock subsys wqloaderd PIDFILE var run wqloaderd pid
7. WQLoader USER MANUAL L Emilitri M Signorini Loway WQLoader USER MANUAL L Emilitri M Signorini Loway Table of Contents 1 Web Aloader A short user guide 1111 1 1 0 1 2 Installation anna Kana bea bana ban Ta bana kang aan kau baba Kanaan 2 2 1 Prerequisite a valid AueueMetrics user iklan 2 2 2 Automated yum instalation sisisi benar ana aan memanen anna 2 2 3 Manual installatiOM 4 344 2s senen nana aa an ada ana AR ERA ea MA A KUMA jana Tia aia A vi 3 2 4 Gonfiguring the loader 1591 rik cece Tenan lowe ceeds bag ade bea nana 7 1 3 3 The network protocol 3 3 33 3131 sa nan basa aan Ban Ba an jan vms jaa 7 3 1 JSON Vectors fields aan nan laduda 7 3 2 HTTP Example sesi 3 3 3 3 3 33 3 1311313111 111 11 1 1 1 11111 3030 0 3 9 4 Freguently Asked GuestionS FAAS 4 15 7777 1 3 1 11 4 1 How do you handle multi client clusters 222 11 Chapter 1 Web Qloader A short user guide The web qloader wgloader for short is a daemon written in Perl that uploads log events to a QueueMetrics instance over HTTP Since version 1 6 the web gloader daemon is able to connect to a set of asterisk servers through AMI and perform some basic actions like log in log out pause and unpause agents under the supe
8. b token See Section 3 1 1 Common fields e version see Section 3 1 1 Common fields token string type containing the name of the token identifying the dataset actually the cluster server name or empty for single server set in QueueMetrics results array of last feedback results defined by the following structure Last Feedback Results commandld string type populated with checkFbResCon token See Section 3 1 1 Common fields e version see Section 3 1 1 Common fields token string type containing the name of the token identifying the dataset actually the cluster server name or empty for single server set in QueueMetrics e id string containing the unique ID identifying the executed feedback operation e resultStatus string containing the status of the executed feedback operation Status should be either OK or KO result string containing an human readable description of the result status QueueMetrics answer with a JSON vector containing the following fields commandld string type populated with checkFb token See Section 3 1 1 Common fields name a string containing a user readable name for this vector In this case is Check Feedback e version see Section 3 1 1 Common fields e actions array of AMI commands to be sent to the remote asterisk Actions are defined by the following structure e resultStatus see Section 3 1 1 Common fields result string type contain
9. e QueueMetrics takes the default configuration values from the QueueMetrics configuration properties file Use this option when running QueueMetrics in single server configuration 2 4 3 Running in Advanced mode By running in Advanced mode you can feed multiple AueueMetrics instances from one single Asterisk box This way each instance can be in a physically separate location and can be configured and possibly resold independently In order for this to work the prerequisite is that you use a common naming schema for all agents and queues be it gueue1 client1 and Agent 100 client1 or client1 gueue1 and Agent client1 100 Then you tell wgloaderd which prefixes or suffixes apply to each QueueMetrics instance and give it all the details to connect to it You do this by configuring a set of rules by editing the file 1 cluster pI1 that resides in the same folder where the main script is stored Within the wqloader_cluster pl file is a set of structures that are to be uncommented and or edited according to your needs For each defined rule the loader starts a process that will be in charge of pushing data to the defined host for that rule There is no limit on the number of rules that can be defined except the CPU and RAM amount needed to handle all the processes that will be generated Each rule defines a set of filtering criteria Two types of filters are availables filters by queue and filters b
10. he information needed to generate AMI transactions like for example the asterisk host port AMI username and password are automatically retrieved by QueueMetrics by looking at the configuration properties file for the server specified in the web qloader token field An empty token field is interpreted by QueueMetrics to use the default asterisk server that s the case for a QueueMetrics operating on a single server mode It s important to properly configure QueueMetrics parameters in order to refer to remote asterisk server like if they re local For example if the web qloader runs on the same box where the asterisk server is located QueueMetrics should define oca host in the AMI connection configuration key For a single server this will reflect as default webloaderpbx tru default gueue log file sgl P001 callfile dir tcp admin password 127 0 0 1 For a simple two asterisk clustered QueueMetrics instead Installation default webloaderpbx tru default gueue log file cluster cluster servers trixl trix2 cluster trix1 manager tcp admin elastix456Glocalhost cluster trix1 gueuelog sgl P101 cluster trix2 manager tcp admin elastix456Glocalhost cluster trix2 gueuelog sgl P102 Chapter 3 The network protocol API Conversations between wgloader and QueueMetrics used JSON as high level protocol In this section is a list of JSON vectors involved in the process The Web Gloader process is responsible for starti
11. in the same way As soon as the configuration file is set up the loader will be run by issuing the following command wgloader pl var log asterisk queue_log X The parameters are taken from the configuration file In this example a dummy parameter X in this case is needed as a placeholder if you want to specify a custom output log When the default output file var log asterisk qloader 1oq is used there is no need to specify any additional command line parameters wqloader pl var log asterisk queue_log 2 4 4 Enable asterisk actions Since Web qloader revision 1 6 and QueueMetrics 14 10 is possible to trigger some basic actions on remote asterisk servers This is very useful when QueueMetrics needs to log in log out pause unpause agents transfer calls etc as requested by an agent through the Icon web interface or by an administrator through the realtime page In order to enable this feature some configurations changes are needed on the QueueMetrics server side These changes affect the QueueMetrics configuration properties file By setting the key default webloaderpbx tru and restarting the QueueMetrics server the feature is enabled The web gloader agent asks periodically to QueueMetrics if there are actions to be performed The polling frequency is defined by a parameter in the web qloader perl script Feel free to modify the variable my Sfeedback delay 5 in seconds in order to change the default All t
12. ing a not null internal technical id associated to this event Feedback Action Container structure e id string type populated with the unique id of the operation to be performed host string type populated with the name or IP identifying the asterisk server port integer type populated with the remote asterisk server port where AMI is listen to actions array of string each one populated with a single AMI transaction to be sent to the remote asterisk server are automatically retrieved by QueueMetrics by looking at the configuration properties file for the server specified in the token field of the checkFbResCon query An empty token field is interpreted by QueueMetrics to use the default asterisk server that s the case for a QueueMetrics operating on a single server mode It s important to properly configure QueueMetrics parameters in order to refer to remote asterisk server as explained in Section 2 4 4 Enable asterisk actions 3 2 HTTP Example As example below is shown a simple transaction dump All the information needed to generate the AMI transactions like for example the host port AMI username and password The network protocol API Firts the Web Qloader sends a POST query to QueueMetrics In this example we can see the authorization field containing the base64 encoding of the username password pair and the COMMANDSTRING parameter populated with the HTTP encoded JSON object commandIid checkHWM
13. k High Watermark This entry asks for the maximum timestamp already available on remote database for the specified token 1 0 1 0 1 0 6 QueueMetrics expects to receive a JSON vector containing the following fields commandld a string populated with checkHWM token See Section 3 1 1 Common fields e version see Section 3 1 1 Common fields token a string containing the name of the token identifying the dataset actually the cluster server name or empty for single server set in QueueMetrics QueueMetrics answers with a JSON vector containing the following fields commandld a string populated with checkHWM token See Section 3 1 1 Common fields name a string containing a user readable name for this vector In this case is Check High WaterMark e version see Section 3 1 1 Common fields e resultStatus see Section 3 1 1 Common fields e result a string containing the highest timestamp found in the database for the specified token or the string null if no data are present 3 1 3 Check Row This entry asks for the presence of queue_1og data line in the database QueueMetrics expects to receive a JSON vector containing the following fields commandld a string populated with checkRow token See Section 3 1 1 Common fields version see Section 3 1 1 Common fields The network protocol API token a string containing the name of the token identifying the dataset act
14. ly the cluster server name or empty for single server set in QueueMetrics timeld a string containing the row data timestamp as reported by asterisk callld a string containing the asterisk unique id for the row data queue a string containing the queue name field for the row data agent a string containing the agent field present in the row data verb a string containing the verb field present in the row data parameters string array containing other unnamed parameters present in the row data The expected array size is 5 meaning that parameters not available should be always present as empty strings QueueMetrics answers with a JSON vector containing the following fields commandid a string populated with insertRow token See Section 3 1 1 Common fields name a string containing a user readable name for this vector In this case is Insert Row version see Section 3 1 1 Common fields resultStatus see Section 3 1 1 Common fields result a string containing a not null internal technical id associated to this event 1 5 Check Heartbeat This entry asks for an insertion of an heartbeat data line in the database Heartbeat events should be sent periodically to QueueMetrics to allow remote process survey The interval between different Heartbeat events is not defined but suggested as 900 seconds QueueMetrics expects to receive a JSON vector containing the following fields commandid a string populated with checkHb
15. ng all transactions Each transaction is wrapped into a single HTTP POST with Basic Authentication targeted to an end point named jsonOLoaderApi do Basic Authentication reguires to have a first unauthenticated HTTP transaction followed by a HTTP 401 Not Authorized server response code and a second client HTTP request containing authorization headers QueueMetrics requires to have a single client transaction with authorization headers already present 3 1 JSON Vectors fields 3 1 1 Common fields QueueMetrics does not support the standard Basic Authentication method but a slightly modified version Where standard All JSON vectors contain some common fields used by QueueMetrics and or the Web Qloader to persist information like for example the protocol version and answer results and statuses commandld a string containing the command identifier Valid values are specified in the table below version a string containing current protocol version Valid values are 1 0 e resultStatus a string containing the operation result status Valid values are OK or KO followed by an error message description result a string containing the result of the current operation Valid values are specific for each operation type Values associated to this field should be discarded for events marked as KO in resultStatus commandid See paragraph Since protocol version checkHWM Check High Watermark 10 checkFb Check for Feedbacks 3 1 2 Chec
16. rvision of AueueMetrics This has several advantages over the plain MySAL protocol used by the old gloaderd tool especially for physically distant systems or hosted systems the QueueMetrics HTTP port already has to be exposed and you do not need to handle TCP connections to MySQL Centralizing data loading into QueueMetrics also acts as a choke point to apply further security restrictions Additionally the wqloader offers automatic splitting of log files in order to feed multiple separate QueueMetrics instances from the same Asterisk instance so that you can host multiple independent clients on the same Asterisk system or cluster The wgloader was built for e Maximum data safety your data is never lost even in case of catastrophic failures on the net or on the receiving QueueMetrics instance e High availability and continuous unattended operation Good performance even when having high or intermittent network latency e Running in parallel with an existing qloaderd The wgloader works with all versions of QueueMetrics after 14 06 3 Chapter 2 Installation 2 1 Prerequisite a valid QueueMetrics user In order to run wgloader you need to have a valid user in QueueMetrics holding the key WQLOADER that must be used for remote authentication Recent versions of QueueMetrics already include a user named webgloader with password gloader It is disabled by default so it has to be manually enabled from the
17. token See Section 3 1 1 Common fields version see Section 3 1 1 Common fields token a string containing the name of the token identifying the dataset actually the cluster server name or empty for single server set in QueueMetrics The network protocol API e timeld a string containing the timestamp of this query in seconds starting from 01 01 1970 QueueMetrics answers with a JSON vector containing the following fields commandld a string populated with checkHb token See Section 3 1 1 Common fields name a string containing a user readable name for this vector In this case is Check Heartbeat e version see Section 3 1 1 Common fields e resultStatus see Section 3 1 1 Common fields e result a string containing a not null internal technical id associated to this event 3 1 6 Check for Feedbacks This entry asks for possible AMI actions to be performed for the specified target token The same entry can be used to report to QueueMetrics the results of already processed feedbacks if retrieved by the previous call Check for Feedbacks events should be periodically sento to QueueMetrics only if feedback events are enabled see Section 2 4 4 Enable asterisk actions The interval between different Check for Feedback events is not defined but suggested as 5 seconds QueueMetrics expects to receive a JSON vector containing the following fields commandld string type populated with checkF
18. ually the cluster server name or empty for single server set in AueueMetrics timeld a string containing the row data timestamp as reported by asterisk callld a string containing the asterisk unique id for the row data gueue a string containing the gueue name field for the row data agent a string containing the agent field present in the row data verb a string containing the verb field present in the row data parameters a string array containing other unnamed parameters present in the row data The expected array size is 5 meaning that parameters not available should be always present as empty strings QueueMetrics answers with a JSON vector containing the following fields commandid a string populated with checkRow token See Section 3 1 1 Common fields name a string containing a user readable name for this vector In this case is Check Exsisting Row version see Section 3 1 1 Common fields resultStatus see Section 3 1 1 Common fields result a string containing the number of matching data lines found in the database 1 4 Insert Row This entry asks for the insertion of a queue_log data line in the database QueueMetrics expects to receive a JSON vector containing the following fields commandld a string populated with insertRow token See Section 3 1 1 Common fields version see Section 3 1 1 Common fields token a string containing the name of the token identifying the dataset actual
19. y agent The filters are applied to each loader row as a first match iptables style starting by queue information Filters are to be individually enabled through configuration Rows without queue or agent information i e containing NONE on these fields are pushed to all servers This allows to split a single multi tenant log into several single tenant QueueMetrics instances A simple example of single rule is reported below ruleName gt Queues starting with Customer Installation gueueStarts gt 1 queueEnds gt 0 queueToken gt Customer agentStarts gt 0 agentEnds gt 0 2 1104 targetPort gt 8084 Bango EE targetUser gt robot targetPass gt robot targetToken gt trix timezoneOffset gt 3600 This example generates a process able to handle queues where name is starting with Customer Events associated to that queues as either generic unassociated events are sent to the 10 10 5 110 8084 customer URL In this case queueStarts should be set to 1 and the queueToken should be populated with Customer Tokens are case sensitive A different rule with queueStarts set to 0 and queueEnds set to 1 will populate only queues with names ending in Customer Filtering by agents requires to properly set the keys agentStarts agentEnds agentToken
Download Pdf Manuals
Related Search