How to do log analysis in Perl?

[Atsushi]

I have heard quite a problem between the operators of Web sites is the analysis of access statistics, the Internet equivalent of Auditel. I am having some basic ideas about log file. All web server access log in a log file in a standard format that the program presented to analyze. I am also knowing that the Perl is proving the ideal tool for this job. The flexibility and power in the management of data in text format with little effort allow Perl to write a program for analyzing web server access logs full and satisfying enough for everyday use. So I am looking for some more knowledge about the log files to analyze in Perl. That's why I need information about how to do log analysis in Perl? I am waiting for the replies from your side.

[GracieSingh]

Analog analyzes logs that are normally stored in a standard format, called common log format. This format is the one adopted by the first two Web servers in history. Our program provides a count of access for each URL and a count of contacts for each site from which the requests were made, you can limit the analysis to some sites, and to some URL with an interval of time. We see the format specification with an example:

saturn.insinet.en - - [02/Oct/1995: 11:00:55 +0100] \
   "GET / cgi-bin/reg-new-user-auto.cgi HTTP/1.0" 200 2295

The fields are in order:

Host the host from which the request
 Rfc931 user information provided by identd, 
		 "-" If it was not possible to identify the user
 Username username for password-protected documents
		 "-" If any (the document is not password)
 Date / Time Date and time of 
 Request the exact URL request from the browser
 Was provided by the server response code (200 is OK)
 N.Byte number of bytes provided in response from the server
		 "-" If it makes no sense (for example, there was a mistake)

[Galbraith]

Analog can be invoked as a CGI or from a form that command line and it always produces results in HTML. It can then be used to perform interactive analysis, and to automatically produce periodic reports. You should have knowledge about:

• Analog in the program itself, which despite its simplicity is of real income (in fact was born as a substitute for more complex programs, which provided a myriad of options that are not necessary);
• a technique for writing scripts that can be used either as a CGI retrieved directly from the command line;
• Perl's regular expressions, which are used intensively and will be investigated.

[FlayoFish]

For a more flexible program, you can limit the analysis for both site of origin for URL requests: for example we can consider all requests this year, excluding those that come from our network, refer to a particular directory and excluding any images. That is:

analog 01/gen/1996 start = \ 

site = site =*.insinet.en *. \ 

url = / jazz / * url =! *. gif

As you can see the parameters are supplied on the command line using <option> = <parameter>. The options that you can specify are:

file = <file> considers the log file <file>
 url = <url> considers only the specified <url>
 url =! <url> excludes specified <url>
 site = <site> considers only the specified <site>
 site =! <site> excludes specified <site>
 begin = <date> considers access occurred after <date>
 end = <date> considers access occurred before <date>

All parameters are optional. Unless stated otherwise, considering all url from all sites during the period covered by the log file and use the log file / var / log / httpd / access_log.

[Suzane]

All parameters are optional. Unless stated otherwise, considering all url from all sites during the period covered by the log file and use the log file / var / log / httpd / access_log.

The site url and parameters can be specified multiple times. The parameters file, begin, end can be specified only once. If you have specified more than once, only considering the last. The arguments <url> <site> and may contain wildcard characters '?' and '*' to be shortened to any character and any sequence of characters. If <url> <site> or are preceded by '!' the subject is excluded, otherwise it is included. The <date> must be in the format dd / mm / yyyy, where the day and year should be numerical, while the month is abbreviated to the first three letters.

[Rover]

For more flexible options for selecting, specifying sites and URLs using the "pattern", which they attribute to the characters * and? A special meaning. These in fact are used in a manner similar to the way in which you specify the file in the DOS world: the? Replaces any character while the * replace any sequence of characters. The! Beginning of a topic can exclude that topic (or website URL) from the analysis. Warning: the point. Did not mean particular, is a character like any other: therefore, the analogy is not perfect with the DOS, the equivalent of *.* in our case is simply *. For security reasons, the program does not accept the parameter "file" when it is invoked by a form, otherwise an attacker could (theoretically) use the program to steal information about your system: if the form can be analyzed only a fixed file, to default (which must be manually changed in the program if it is / var / log / httpd / access_log). If you install the program as a CGI, must also decide if you want to allow external users (and potential competitors) to analyze the access log: If you want to protect the privacy of the log of your site but also be able to analyze them interactively, you should think about protecting the script with a password.