Web Traffic Calculation

Last modified: 27 Dec 2007

IMPORTANT:
H-Sphere 2.5 Beta 1 and up introduces a completely different approach for traffic calculation and log rotation. Now it takes into account both incoming and outgoing traffic. Therefore, after you upgrade from H-Sphere version earlier than 2.5, your clients may find their traffic relatively increased.

There are two types of traffic calculation in H-Sphere:

• Traffic calculation by third-party log analyzers - H-Sphere writes log files for each customer's domain into respective directories to make them available for third-party log analyzers included into default H-sphere installation: Webalizer, Modlogan, and AWStats.
• H-Sphere built-in traffic calculation - H-Sphere provides its own mechanism of traffic calculation used in billing. H-Sphere traffic reports are available in admin CP as Transfer Traffic Report in the Reports menu.

Traffic Calculation By Third-Party Log Analyzers

HTTP Logs for each domain are located in the /hsphere/local/home/{user}/logs/{domain.name}/ directory, in the following files, provided respective resources are enabled:

• Transfer log: the {domain.name} file;
• Agent log: agent_log
• Referrer log: referrer_log
• Error log: error_log

Here, {user} is account name, and {domain.name} is user domain name.

Log Rotation

H-Sphere runs daily cron script /hsphere/shared/scripts/cron/cron_rotate.pl:

0 2 * * * nice -15 /hsphere/shared/scripts/cron/cron_rotate.pl

Log rotation data is taken from the /hsphere/local/config/httpd/logrotate_confs/ directory. The files there look like:

• Transfer log: {domain.name}.transferlog.conf
• Agent log: {domain.name}.agentlog.conf
• Referrer log: {domain.name}.referrerlog.conf
• Error log: {domain.name}.errorlog.conf

The cron_rotate.pl script:

1. ( in 2.5 Beta 5) rotates current log files in the /hsphere/local/home/{user}/logs/{domain.name}/ directory into the {domain.name}.1, {domain.name}.2.gz, {domain.name}.3.gz etc. files (the first one NOT being gziped). For example:
   -rw------- 1 user29 user29 0 Jan 9 20:24 domain29.test
   -rw------- 1 user29 user29 392000 Jan 9 20:24 domain29.test.1
   -rw------- 1 user29 user29 1495 Jan 9 20:24 domain29.test.2.gz
   -rw------- 1 user29 user29 1496 Jan 9 20:11 domain29.test.3.gz
   ...
   Before HS 2.5 Beta 5, logfiles had different title format ({domain.name}.{timestamp}.gz). The converter /hsphere/shared/converters/oldlogfileformat-converter.pl renames all old logfiles to the new format.
2. runs Webalizer's, Modlogan's and AWStat's command-line utilities that parse current logs and store them into respective directories for each domain in format readable for these analyzers:
   
3. Webalizer, ModLogAn and AWStats take statistics from the access log files already rotated. Currently used access log files for Webalizer/Modlogan/AWStats are specified in the respective {LOG}.{domain.name}.txt files:
   
   • Webalizer: webalizer.{domain.name}.txt
   • Modlogan: modlogan.{domain.name}.txt
   • AWStats: awstats.{domain.name}.txt
   cron_rotate.pl uses the /hsphere/shared/scripts/getlogs.pl script to update the latest log file name specified in these files. Also, it calls Webalizer's, Modlogan's and AWStat's command-line utilities that parse current logs and store them into respective directories for each domain in format readable for these analyzers:
   
   • Webalizer: /hsphere/local/home/{user}/{domain.name}/webalizer/
   • Modlogan: /hsphere/local/home/{user}/{domain.name}/modlogan/
   • AWStats: /hsphere/local/home/{user}/{domain.name}/awstats/data/

H-Sphere Built-In Traffic Calculation

Traffic Log

Since version 2.5, H-Sphere introduces the mod_psoft_traffic module to write a more informative and convenient traffic log into the /hsphere/local/var/httpd/logs/traffic_log file. Traffic log has the following format:

<unix_timestamp> <domain_name> <incoming_traffic> <outgoing_traffic>

For example:

1102091887 domain.com 623 623
1102091888 domain.com 65 132

Analyzing Logs

H-Sphere runs daily the /hsphere/shared/scripts/cron/analyze.pl cron script:

0 0 * * * nice -15 /hsphere/shared/scripts/cron/analyze.pl /hsphere/local/var/httpd/logs/traffic_log

The script parses traffic_log and writes specially formatted dd.mm.YYYY.txt http log files in the /hsphere/local/var/statistic directory (dd.mm.YYYY is date timestamp).

Log format:

|{domain.name}|incoming_traffic|outgoing_traffic|

TrafficLoader

TrafficLoader H-Sphere Java class is in charge of parsing server traffic. It is launched daily by cron:

30 5 * * * su -l cpanel -c 'java psoft.hsphere.TrafficLoader'

TrafficLoader processes Web, mail, FTP and virtual FTP traffic in the formatted statistics files located in the /hsphere/local/var/statistic directory and inserts these lines into the translog table of the H-Sphere system database.

TrafficLoader also calls the /hsphere/shared/scripts/xfer_cat.pl script to rotate the already loaded statistics files to the /hsphere/local/var/statistic/loaded directory as dd.mm.YYYY.txt.gz archives.