Web Traffic Calculation

Last modified: 27 Dec 2007

WARNING: This documentation covers Parallels H-Sphere versions up to 3.1. For the latest up-to-date Parallels H-Sphere documentation, please proceed to the official Parallels site.

There are two types of traffic calculation in H-Sphere:

• Traffic calculation by third-party log analyzers - H-Sphere writes log files for each customer's domain into respective directories to make them available for third-party log analyzers included into default H-sphere installation: Webalizer, Modlogan, and AWStats.
• H-Sphere built-in traffic calculation - H-Sphere provides its own mechanism of traffic calculation used in billing. H-Sphere traffic reports are available in admin CP as Transfer Traffic Report in the Reports menu.

Note: H-Sphere 2.5 and up introduces a totally different approach to Web traffic calculation and log rotation.

Traffic Calculation By Third-Party Log Analyzers

HTTP logs for each domain are located in the /hsphere/local/home/{user}/logs/{domain.name}/ directory, in the following files, provided respective resources are enabled:

• Transfer log (access Log): the {domain.name}.{timestamp} file;
• Agent log: agent_log.{timestamp}
• Referrer log: referrer_log.{timestamp}
• Error log: error_log.{timestamp}

Here, {user} is account name, and {domain.name} is user domain name, {timestamp} is Unix timestamp for the time the log has been rotated.

Log Rotation

H-Sphere runs daily cron script /hsphere/shared/scripts/cron/cron_rotate.pl:

0 2 * * * nice -15 /hsphere/shared/scripts/cron/cron_rotate.pl

Log rotation data is taken from the /hsphere/local/config/httpd/rotatelog_cron.cfg config file.

The cron_rotate.pl script:

1. rotates current logs in the /hsphere/local/home/{user}/logs/{domain.name}/ directory into the {LOG}.{timestamp}.gz files, for example, sample.com.1125702004.gz for transfer log and agent_log.1125702004.gz for agent log.
2. runs Webalizer's, Modlogan's and AWStat's command-line utilities that parse current logs and store them into respective directories for each domain in format readable for these analyzers:
   
   • Webalizer: /hsphere/local/home/{user}/{domain.name}/webalizer/
   • Modlogan: /hsphere/local/home/{user}/{domain.name}/modlogan/
   • AWStats: /hsphere/local/home/{user}/{domain.name}/awstats/data/
   Webalizer and Modlogan take the current day access log (the /hsphere/local/home/{user}/logs/{domain.name}/{domain.name} file) for today's statistics. AWStats shows yesterday's statistics from the rotated access log ({domain.name}.{timestamp}.gz). Currently used access log file for AWStats is specified in the awstats.{domain.name}.txt file. cron_rotate.pl uses the /hsphere/shared/scripts/awstats_getlogs.pl script to update the latest log file name specified for AWStats.

H-Sphere Built-In Traffic Calculation

Traffic Log

Web server Apache runs the rotatelogspsoft and rotatelogspsoft2 utilities from the /hsphere/shared/apache/bin/ directory to write detailed traffic log into the /hsphere/local/var/httpd/logs/access_log file. rotatelogspsoft and rotatelogspsoft2 are modified logrotate Apache utility.

Apache directives in the /hsphere/local/config/httpd/httpd.conf Apache conf file look like:

CustomLog "| /hsphere/shared/apache/bin/rotatelogspsoft /hsphere/local/var/httpd/logs/access_log 86400" "%v|%f|%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""
CustomLog "| /hsphere/shared/apache/bin/rotatelogspsoft2 /hsphere/local/config/httpd/rotatelog_referer.cfg referrer_log 86400" "%v|%{Referer}i -> %U"
CustomLog "| /hsphere/shared/apache/bin/rotatelogspsoft2 /hsphere/local/config/httpd/rotatelog_agent.cfg agent_log 86400" "%v|%{User-Agent}i"

Analyzing Logs

H-Sphere runs daily the /hsphere/shared/scripts/cron/cron_analyze.pl cron script:

0 0 * * * nice -15 /hsphere/shared/scripts/cron/cron_analyze.pl

The script parses access_log and writes specially formatted dd.mm.YYYY.txt http log files in the /hsphere/local/var/statistic directory (dd.mm.YYYY is date timestamp).

Log format:

|name|xFer(kB)|Hits_All|Hits_HTML|

where name is the domain name, xFer is the total traffic in kilobytes.

TrafficLoader

TrafficLoader H-Sphere Java class is in charge of parsing server traffic. It is launched daily by cron:

30 5 * * * su -l cpanel -c 'java psoft.hsphere.TrafficLoader'

TrafficLoader processes Web, mail, FTP and virtual FTP traffic in the formatted statistics files located in the /hsphere/local/var/statistic directory and inserts these lines into the translog table of the H-Sphere system database.

TrafficLoader also calls the /hsphere/shared/scripts/xfer_cat.pl script to rotate the already loaded statistics files to the /hsphere/local/var/statistic/loaded directory as dd.mm.YYYY.txt.gz archives.