Home

GSA HTTP Sniffer - User Guide - 1.3

1. lt parameter name password1 value gt lt parameter name password2 value gt lt file name importFileName value gsaConfig xml gt lt request gt This configuration would treat passwordiIn as a variable and perform a file upload from the file at importFileName 4 HTTP authentication schemes The HTTP Sniffer supports all standard authentication schemes HTTP Basic HTTP Digest SPNEGO Kerberos and NTLM Upon successful HTTP authentication i e HTTP challenge an Authorization request header is set and can be reused by all subsequent requests The value of the Authorization header can be read in the sniffer 1log file passed consequently as a variable to all following requests As an example the following Web process automates the secure search on a Google Search Appliance where content is protected with HTTP BASIC authentication The first request attempts to perform the search and passes the users credentials via variables Provided that the end user is authorized to search an Authorization header is set upon successful authentication challenge recorded in the log file 2006 02 07 10 58 18 942 Thread 0 INFO HTTP response header WWW Authenticate Basic realm Google Authentication When prompted for the authorization variable the value of the header from the first request can copy the Authorization header value and enter the keywords for the second query HTTP Sniffer Version 1 2 January 2008 lt webPro
2. Google Search gt lt request gt As a direct consequence variables may be set within a Web process descriptor to intercept a value of a request parameter assigned dynamically by some JavaScript code on page A and required for subsequent request B to assign an Authorization header returned by a successful authentication request and or to avoid sensitive information password login to be entered in clear File Input Used to point to a file contain the contents to be used for the variable See the XML configuration AddCrawlUrl s for an example usage of the file input where a list of Start Urls is read from a file The sample below makes use of a file to source the contents of a variable in this example the list of start URL s are listed in a file start txt http www google com enterpris http support google com Sample XML snippet using file variable type lt request gt lt type gt GET lt type gt lt URL gt http GSA_HOSTNAME 8000 lt URL gt lt request gt lt file name startUrlsFile src start txt gt lt file name goodUrlsFile src include txt gt lt file name badUrlsFile src exclude txt gt Downloading files Even though all communication between the Sniffer and the web server is logged there is sometimes a need to save the results of a HTTP request into a separate file This is done by using the saveAs parameter and providing a valid file name Here is an example where we have ad
3. 535 Thread 0 INFO HTTP cookie AUTH_26335425 3hahkf9sho6216zu path search domain gsa domain secure These cookies will be valid for a certain period of time until they expire The following example illustrates how Search authentication cookies may be set manually from the Web process descriptor and how the authorization process may be tested separately Since there is no need for authenticating again the HTTP request is declared as public Until expiration of the cookies some search results will be returned successfully lt xml version 1 0 gt lt webProcess name moma xmlns http www google com enterprise gsa gt lt cookie domain name COOKIETEST value 1 secure true gt lt cookie domain gsa domain name AUTH_26335425 value 3hahkf9sho6216zu path search secure true gt lt header name User Agent value Mozilla 5 0 Windows U Windows NT 5al n US ry 1 8 Geeko 20051111 Fir fox 1 5 gt lt variable name keywords default type a z gt lt request gt lt type gt GET lt type gt lt URL gt http gsa domain search lt URL gt lt parameter name q value S keywords gt lt parameter name site value internal gt lt parameter name client value internal gt lt parameter name output value xml_no_dtd gt lt parameter name ie value UTF 8 gt lt parameter name oe value UTF 8 gt lt parameter name proxystyle
4. HTTP parameter value would be set by some JavaScript code in an HTML page and required in the subsequent HTTP request s a copy of the latest HTML page accessed by the HTTP Sniffer e g request SnifferX html where Xis the sequence number of the request in the process description is also saved in the Sniffer_root_dir tmp directory 3 Aninteractive HTTP client In order to provide user interactivity along with the execution of a Web process the concept of variables is available Variables are simple placeholders Two types of variables are currently support user input and file HTTP Sniffer Version 1 2 January 2008 User Input They are set dynamically by the end user from a prompt shell Their format can be controlled by a regular expression and a default value may be assigned Any header cookie and parameter name and or value can be assigned dynamically via a variable The URL of an HTTP request can also be set dynamically As illustrated below a keywords variable with no default value has been defined and allows the end user to enter from his console a list of keywords The regular expression assigned to the variable controls the user input format lt variable name keywords default type a z gt lt request gt lt t ype gt GET lt type gt lt URL gt http www google com search lt URL gt lt parameter name h1 value en gt lt parameter name q value Skeywords gt lt parameter name btnG value
5. assist in identifying any User Agent filtering or any interfering JavaScript snippet that are usually almost impossible to isolate with standard utilities Firefox liveHttpHeaders and the like The HTTP Sniffer will not reduce the complexity of the on going HTTP communication i e programmatic and HTTP redirects etc between several Web enabled systems but will certainly contribute in providing more visibility on the underlying infrastructure HTTP Sniffer Version 1 2 January 2008 2 A configurable HTTP client A Web process is described in a well formed and valid XML file for example web_process xml and executed by the following command sniffer bat sh lt path to web process xml gt The sniffer xsd file that comes with the utility controls the grammar of the Web process descriptor Each HTTP request targeting a certain URL may be of three kinds GET POST and HEAD and may include a set of HTTP headers and or parameters As a convenience the HTTP Sniffer grammar provides the ability to set global headers shared by all requests of a Web process This may particularly be useful to set a specific User Agent etc The example below illustrates a Web process called google that accesses the Google homepage and sends a simple query by entering the google search appliance keywords It makes use of a global header that sets the User Agent value to be the one of the Firefox browser and a few HTTP parameters that define the Google query
6. context lt xml version 1 0 gt lt webProcess name google xmlns http www google com enterprise gsa gt lt header name User Agent value Mozilla 5 0 Windows U Windows NT 5 1 en US ry 1 8 Geeko 20051111 Fir fox 1 5 gt lt request gt lt type gt GET lt type gt lt URL gt http www google com lt URL gt lt request gt lt request gt lt type gt GET lt type gt lt URL gt http www google com search lt URL gt lt parameter name hl value en gt lt parameter name q value google search appliance gt lt parameter name btnG value Google Search gt lt request gt lt webProcess gt The logs sniffer log file contains all information about the HTTP request headers passed by the end user the HTTP response headers returned by the Web server the HTTP cookies set by the Web server and the corresponding HTTP request status message and code Upon completion of a Web process a zipped file is created in the directory tmp Its name is of the following pattern process_name date zip where process_name is the name set in the XML descriptor and date is of type HHmmss This zipped file contains a copy of the sniffer 1log file and of all HTML pages accessed by the HTTP Sniffer e g requestSnifferX html where Xis the sequence number of the HTTP request in the process description along the execution of the Web process In the case where a Web process would contain several HTTP calls and an
7. Go O le Version 1 3 Date January 07 Authors Olivier Colinet Paul Thompson C sar L zaro v1 3 update GSA HTTP Sniffer utility Change History Version 1 0 e initial release Version 1 1 e Support MULTIPART method to upload files or parameters Can be used with the Google Search Appliance to script the import of configuration or to create collections which use multipart methods Sample Sniffer configuration file included gsalmportConfig xm and AddCollection xm e Support for SAVEAS to save the output of a specific request to a file Sample Sniffer configuration file included gsaExportConfig xm e Reading variable input from a file ability to define a variable s values s to be read from a file Can be used to added a list of start URL s from a file and add them to the GSA Admin Start e URL Sample Sniffer configuration file included AddCrawlURL xm e bug fixes Fixed problem with spaces in path in sniffer xsd file Version 1 2 e Added support for HTTP sites that do not have a valid certificate Use Dssl cert any to disable any certificate validation Useful for sites that user self certification option should be used with caution e Removed the need to specify the location of sniffer xsd it is assumed to be in conf directory Version 1 3 e SPNEGO Kerberos authentication mechanism has been added to the other ones already supported in previous versions New Kerberos credentials classes have been included e bug fixes Fixed issue when s
8. cess name moma xmlns http www google com enterprise gsa gt lt header name User Agent value Mozilla 5 0 Windows U Windows NT 5 1 en US rvil 8 Gecko 20051111 Fireftox 1 5 gt lt variable name keywords default type a z gt lt variable name username default userID type gt lt variable name password default _type gt lt request gt lt authentication gt lt Basic gt lt username gt Susername lt username gt lt password gt password lt password gt lt Basic gt lt authentication gt lt type gt GET lt type gt lt URL gt http gsa domain search lt URL gt lt parameter name q value S keywords gt lt parameter name site value internal gt lt parameter name client value internal gt lt parameter name output value xml_no_dtd gt lt parameter name ie value UTF 8 gt lt parameter name oe value UTF 8 gt lt parameter name proxystylesheet value internal gt lt parameter name access value a gt lt request gt lt variable name authorization default _ type gt lt variable name keywords default type a z gt lt request gt lt type gt GET lt type gt lt URL gt http gsa domain search lt URL gt lt header name Authorization value Sauthorization gt lt parameter name q value Skeywords gt lt parameter name site value internal gt lt parameter name clien
9. ded this option to the above web process export gt HTTP Sniffer Version 1 2 January 2008 lt request gt lt type gt GET lt type gt lt saveAs gt gsaExportConfig xml lt saveAs gt lt URL gt http GSA_HOSTNAME 8000 EnterpriseController lt URL gt lt parameter name actionType value importExport gt xl SxpOrte paraphrase gt lt parameter name passwordl value exportpassword gt lt parameter name password2 value exportpassword gt lt parameter name export value Export Configuration gt lt request gt This feature is particularly useful when using the sniffer for automating administrative tasks e g it can be used to download the configuration file from the Google Search Appliance and thus creating a configuration backup routine Uploading files Uploading files is done with by using the MULTIPART request type in conjunction with the file parameter The MULTIPART request type work as a normal POST but will add an extra part to the HTTP post by reading embedding and sending the file specified by the file parameter It could look like this Kaf Imo t gt lt request gt lt t ype gt MULTIPART lt type gt lt URL gt http GSA_HOSTNAME 8000 EnterpriseController lt URL gt lt parameter name actionType value importExport gt lt parameter name passwordiIn value SpasswordIn gt lt parameter name import value Import Configuration gt
10. ending cookies over to sendRequest method Lack of scalability solved using configurable connection pooling parameters Fixed problem with query strings 1 Introduction In today s environment the Enterprise system Web front ends often make use of JavaScript event handlers and browser specific customizations Their integration with commercial and proprietary Single Sign On systems may bring its own additional set of Enterprise specific customizations that are sometimes very difficult to identify for a non security expert The HTTP Sniffer was designed to bridge the gap and diagnose any HTTP communication from the inside The HTTP Sniffer is first and foremost a configurable HTTP client that can automate the execution of a Web process i e a set of HTTP or HTTPs requests It provides tracing capabilities for any HTTP headers parameters and cookies and handles in standard most redirects certificates and authentication schemes More importantly any HTML content accessed successfully by the HTTP Sniffer is cached locally so that the end user can isolate easily any interfering JavaScript snippet and reproduce accordingly its actions By incrementally configuring a set of HTTP requests the end user will achieve successfully the automation of a secure or public Web process The HTTP Sniffer utility will help system administrators security experts Webmasters developers to diagnose and understand a specific set of HTTP or HTTPs requests It will
11. s service name ticket has to be registered in the KDC Key Distribution Center that in Windows environments coincide with the Domain Controller The keytab file is created by the Kerberos ktpass utility Check how to create them as it really depends on your platform The Kerberos v5 configuration file also has to be present This file is usually name as krb5 ini in Windows whereas in Linux this file is usually located at etc krb5 conf You can find some configuration examples at the Sniffer conf directory 5 HTTP cookies The values of the HTTP cookies set by the Web servers are recorded in the sniffer 1log file see the Set Cookie response headers Since some authentication cookies are set by all Single Sign On systems it may be important in certain cases to test the authentication process and the authorization process separately Different HTTP connections may handle the authentication process and the authorization process which in our case would consist in writing two different XML process descriptors the cookies are forwarded from one HTTP connection to another as long as they are initiated in the same domain In order to do so the sniffer xsd grammar provides the ability to set the cookies manually By executing the Web process described in the section above HTTP Authentication Schemes two cookies were set by the Web server 2006 02 07 10 17 16 535 Thread 0 INFO HTTP cookie COOKIETEST 1 path secure 2006 02 07 10 17 16
12. sheet value internal gt lt parameter name access value a gt lt request gt lt webProcess gt HTTP Sniffer Version 1 2 January 2008 6 System requirements The HTTP Sniffer utility is a Java program and as such requires a local JRE 1 5 0 and upwards It depends upon the following libraries commons collections 3 2 jar e http jjakarta apache org commons collections commons httpclient 3 0 1 jar e http jakarta apache org commons httpclient commons logging 1 1 jar e http jakarta apache org commons logging commons codec 1 3 jar e http jakarta apache org commons codec log4j 1 2 14 jar e http Aogging apache org log4j docs index html jdom jar e http Avww jdom org xerces mpl jar e http xerces apache org xerces j HTTP Sniffer Version 1 2 January 2008
13. t value internal gt lt parameter name output value xml_no_dtd gt lt parameter name ie value UTF 8 gt lt parameter name oe value UTF 8 gt lt parameter name proxystylesheet value internal gt lt parameter name access value a gt lt request gt lt webProcess gt When using SPNEGO authentication mechanism it requires some special variables that configure the Kerberos environment so that the configuration file looks like different The sniffer manages Kerberos tickets in a couple of ways e They are created on the fly using username and password credentials This configuration is pretty similar to the authentication tag shown in the example above changing Basic to Kerberos e They are obtained from a special Kerberos file keytab that contains the ticket The configuration information is stored in a special Java configuration file An example spnego conf of this file looks like com sun security jgss initiate com sun security auth module Krb5LoginModule required principal HTTP server google com ENTERPRISE GOOGLE COM keytab C keytabFileLocation krb5 keytab useKeyTab true storeKey true doNotPrompt true debug t rue HTTP Sniffer Version 1 2 January 2008 This file points to the already created keytab file and also specifies its principal service name that in this case is HTTP4Server google com in the ENTERPRISE GOOGLE COM Kerberos domain The Kerbero

GSA HTTP Sniffer - User Guide - 1.3

Contents

Download Pdf Manuals

Related Search

Related Contents