Parallels H-Sphere Documentation System Administrator Guide

 

Adding Load Balanced NetApp NASs To H-Sphere

(H-Sphere 2.4.x)
 
 

Related Docs:   NetApp Configuration (HS 2.5) Updating H-Sphere On NetApp Filer

Last modified: 27 Dec 2007

 

WARNING: This documentation covers Parallels H-Sphere versions up to 3.1. For the latest up-to-date Parallels H-Sphere documentation, please proceed to the official Parallels site.

In H-Sphere 2.3 and up, it is possible to add load balanced (LB) Web and mail clusters to H-Sphere on the basis of NetApp NAS. NetApp NAS servers are building blocks for open storage networks, allowing companies to simplify, share and scale their storage networking and content delivery infrastructures.

NetApp load balanced solution requires 3 or more physical servers:

  • Load Balancer: any solution like Cytrix® NetScaler for load balancing across the web/mail servers. Load Balancer directs traffic to another server if the first one is currently overloaded.
  • NAS: Network Attached Storage like NetApp Primary Storage (NetApp Filer) for storage of web/mail content. NAS may be installed on the same server with load balancer or on a separate server. Also, Web and mail servers can jointly use one NAS or have their own NAS one for Web and one for mail.
  • At least two boxes (master and slave) for web/mail servers further referred to as Web1 (Web master), Web2(Web slave); Mail1 (mail master), Mail2 (first mail slave), Mail3 (second mail slave), ...

To create Web/mail load balanced clusters integrated into H-Sphere:

  1. Install and Configure Load Balancer and Filer
  2. Prepare the System to Work With NetApp NAS
  3. Install H-Sphere on Web/Mail Servers
  4. Configure LB Web Servers
    - Dedicated IPs On LB Web Servers
  5. Configure LB Mail Servers
  6. Configure CP Server To Implement NetApp Support
  7. Add Web/Mail Servers to Load Balanced H-Sphere Clusters

See also NetApp Configuration in H-Sphere 2.5.


 

Step 1. Install and Configure Load Balancer and Filer

Buy NetApp NAS directly from www.netapp.com. The instructions about installing and configuring the filer itself can be found in the NetApp Documentation.

Do the following steps to configure your NetApp NAS (read the NetApp Manual for commands):

  1. Telnet to the NetApp NAS:

    telnet <NAS_IP>

    Here, <NAS_IP> is the NetApp NAS IP.
  2. Get the list of NetApp partitions with the qtree command:

    qtree

  3. To enable disk quota management, export the /etc directory on the NetApp NAS and allow to mount it only from the CP box:

    exportfs -o access=<CP_IP>,root=<CP_IP>,rw=<CP_IP> /etc

    Here, <CP_IP> is the CP server IP.
  4. To enable user disk space management on the web/mail servers, export the user storage directory on the NetApp NAS allow to mount it from the physical web/mail boxes:

    exportfs -o access=<Web/Mail1_IP>:<Web/Mail2_IP>,root=<Web/Mail1_IP>:<Web/Mail2_IP>,rw=<Web/Mail1_IP>:<Web/Mail2_IP> /vol/hsphere/home

    Here, <Web/Mail1_IP> (master server), <Web/Mail2_IP> (slave server) are web/mail server IPs, /vol/hspere/home is the user storage directory.
  5. Exit telnet session on the NetApp NAS.
 

Step 2. Prepare the System to Work With NetApp NAS

  1. Grant rsh access to the NetApp NAS from the CP box to root and cpanel user.
  2. Grant nfs access to the /etc directory for the CP box in rw mode.
  3. Grant nfs access to the home directory on the storage partition (/vol/vol0/home) for the CP box in rw mode with root privileges (-access=192.168.0.9:192.168.0.10, root=192.168.0.8:192.168.0.9:192.168.0.10).
 

Step 3. Install H-Sphere on Web/Mail Servers

1) Make sure to install on these servers the same version of H-Sphere as is running on your cp server.

  1. Download current installer
  2. Collect your current H-Sphere configuration:
    # ./import
  3. Launch the configuration wizard to enter installation data:
    # ./configure

    In the configuration menu:

    1. Add a master/slave server with the 'a' command. You will be prompted to add the server's IP and mask. After that, it returns to the main configuration menu and you will see the server's IP in the list of physical servers;
    2. Choose the newly added server by typing in its number in the list and pressing Enter. You will see the list of services that can be enabled/disabled for this server.
    3. To enable a service for the newly added server, type in the number that corresponds to WEB/MAIL service.
    4. When you type '-' and return to the main menu, you will see the service enabled in front of the new server's IP in the list of your servers, for example:

      |-[2] 1.2.3.4 [Web]

    5. Repeat steps 1-4 above to add a slave web server in the same manner.
    6. Save configuration and exit with the 's' and 'q' commands from the main menu.
  4. Upload install scripts to the new servers and configure SSH keys to access them:
    make add-server-<ID>
    where <ID> is the number of the new server you have saved.
  5. Login to the new server as root, then install and configure the new server:
    cd /hsphere/install
    make install

2) Please make sure to place public root SSH keys of master server to slave servers and vice versa, so that load balanced synchronization scripts may work without passwords.

3) Stop Web/mail H-Sphere related services on these boxes:

  • Web services: httpd, proftpd
  • Mail services: qmaild, courier-imapd, courier-imapd-ssl

Web on Linux:

# /etc/rc.d/init.d/httpd stop
# /etc/rc.d/init.d/proftd stop

Web on FreeBSD:

# /usr/local/etc/rc.d/apache.sh stop
# /usr/local/etc/rc.d/proftd.sh stop

Mail on Linux:

# /etc/rc.d/init.d/qmaild stop
# /etc/rc.d/init.d/courier-imapd stop
# /etc/rc.d/init.d/courier-imapd-ssl stop

Mail on FreeBSD:

# /usr/local/etc/rc.d/qmaild.sh stop
# /usr/local/etc/rc.d/courier-imapd.sh stop
# /usr/local/etc/rc.d/courier-imapd-ssl.sh stop

 

Step 4. Configure Master and Slave Web Servers

  1. On each web box, mount the NetApp storage partition to the /mnt/NAS directory:

    mkdir /mnt/NAS
    mount -t nfs <NAS_IP>:/vol/hsphere/home /mnt/NAS

  2. Copy the following directories to the mountpoint directory on the NetApp NAS:

    cp -prf /usr/local/frontpage /mnt/NAS/linuxweb1-001/
    cp -prf /hsphere/local/config/httpd/ssl.shared /mnt/NAS/linuxweb1-001/

    On the master (Web1) web box:

    cp -prf /hsphere/* /mnt/NAS/linuxweb1-001/

    On the the slave (Web2) web box:

    cp -prf /hsphere/* /mnt/NAS/linuxweb2-001/

  3. On both Web servers, create the /hsphere and /hsphere2 directories if you don't have them:

    mkdir /hsphere
    mkdir /hsphere2

  4. On the master web server, /hsphere directory should point to the previously copied linuxweb1-001 master web server directory on the NetApp NAS /hsphere2 to the slave server directory (linuxweb2-001). Similarly, on the slave web server, /hsphere should point to the slave web server directory, /hsphere2 to the master web server directory.

    Thus, the two corresponding mountpoints for the /hsphere and /hsphere2 directories should be added in the /etc/fstab file on both web servers.

    For the master web server there should be the following lines:

    <NAS_IP>:/vol/hsphere/home/linuxweb1-001 /hsphere  nfs  defaults,nfsvers=3  0 0
    <NAS_IP>:/vol/hsphere/home/linuxweb2-001 /hsphere2 nfs  defaults,nfsvers=3  0 0
    <NAS_IP>:/vol/hsphere/home/linuxweb1-001/frontpage /usr/local/frontpage nfs  defaults,nfsvers=3  0 0
    <NAS_IP>:/vol/hsphere/home/linuxweb1-001/ssl.shared /hsphere/local/config/httpd/ssl.shared nfs  defaults,nfsvers=3  0 0
  5. For the slave server, the /etc/fstab file should also contain mountpoints to other directories previously copied to the NetApp NAS.

    <NAS_IP>:/vol/hsphere/home/linuxweb2-001 /hsphere nfs defaults,nfsvers=3  0 0
    <NAS_IP>:/vol/hsphere/home/linuxweb1-001 /hsphere2 nfs defaults,nfsvers=3  0 0
    <NAS_IP>:/vol/hsphere/home/linuxweb1-001/local/home /hsphere/local/home nfs defaults,nfsvers=3  0 0
    <NAS_IP>:/vol/hsphere/home/linuxweb1-001/local/var/statistic /hsphere/local/var/statistic nfs defaults,nfsvers=3  0 0
    <NAS_IP>:/vol/hsphere/home/linuxweb1-001/local/var/httpd/logs /hsphere/local/var/httpd/logs nfs defaults,nfsvers=3  0 0
    <NAS_IP>:/vol/hsphere/home/linuxweb1-001/frontpage /usr/local/frontpage nfs defaults,nfsvers=3  0 0
    <NAS_IP>:/vol/hsphere/home/linuxweb1-001/ssl.shared /hsphere/local/config/httpd/ssl.shared nfs defaults,nfsvers=3  0 0
  6. Check if the crontab contains scripts for synchronizing information between the master server Web1 and the slave server Web2.

    Important: Starting from H-Sphere 2.4.1, synchronization scripts are installed to the /hsphere/shared/scripts/load-ballancing directory on both the master and slave servers. You should copy them to the /hsphere/shared/scripts/cron directory where your Web server cron scripts are stored.

    a) Master web server: Crontab on the master web server should contain the following lines:

    1 */4 * * * /usr/sbin/ntpdate ntps1-{0,1,2}.uni-erlangen.de # time syncronization script
    0 1 * * * nice -15 /hsphere/shared/scripts/cron/cron_analyze.pl
    0 2 * * * nice -15 /hsphere/shared/scripts/cron/cron_rotate.pl
    0 3 * * * nice -15 /hsphere/shared/scripts/cron/ftp_anlz.pl
    0 4 * * * nice -15 /hsphere/shared/scripts/cron/ftp_anlz_user.pl
    */2 * * * * nice -15 /hsphere/shared/scripts/cron/apache-confsynch.pl
    */2 * * * * nice -15 /hsphere/shared/scripts/cron/apache-restart.pl
    */2 * * * * nice -15 /hsphere/shared/scripts/cron/ftp-confsynch.pl
    */3 * * * * nice -15 /hsphere/shared/scripts/cron/ftp-restart.pl
    0 5 * * * nice -15 /hsphere/shared/scripts/cron/db_usage.pl

    • /hsphere/shared/scripts/cron/apache-confsynch.pl - synchronize master and slave load balanced mail servers
    • /hsphere/shared/scripts/cron/ftp-confsynch.pl - synchronize master and slave NetApp FTP servers
  7. Important: For correct synchronization between servers, it is REQUIRED to have time sychronization script in crontab on both servers!

    Crontab on the master server should also contain the following scripts:

    On Linux:

    */1 * * * * rsync -e ssh /etc/passwd Web_Slave_IP:/etc/passwd
    */1 * * * * rsync -e ssh /etc/shadow Web_Slave_IP:/etc/shadow
    */1 * * * * rsync -e ssh /etc/group Web_Slave_IP:/etc/group

    On FreeBSD:

    */1 * * * * rsync -e ssh /etc/passwd Web_Slave_IP:/etc/passwd
    */1 * * * * rsync -e ssh /etc/group Web_Slave_IP:/etc/group
    */1 * * * * rsync -e ssh /etc/master.passwd Web_Slave_IP:/etc/master.passwd
    */1 * * * * rsync -e ssh /etc/pwd.db Web_Slave_IP:/etc/pwd.db
    */1 * * * * rsync -e ssh /etc/spwd.db Web_Slave_IP:/etc/spwd.db

    b) Slave web server: Crontab on the master and the slave server should contain the following lines:

    1 */4 * * * /usr/sbin/ntpdate ntps1-{0,1,2}.uni-erlangen.de # time syncronization script
    */2 * * * * nice -15 /hsphere/shared/scripts/cron/apache-need-restart.pl
    */2 * * * * nice -15 /hsphere/shared/scripts/cron/apache-restart.pl
    */2 * * * * nice -15 /hsphere/shared/scripts/cron/ftp-need-restart.pl
    */3 * * * * nice -15 /hsphere/shared/scripts/cron/ftp-restart.pl
    0 5 * * * nice -15 /hsphere/shared/scripts/cron/db_usage.pl

    • /hsphere/shared/scripts/cron/apache-need-restart.pl - control Web server restart after synchronizing master and slave NetApp Web servers
    • /hsphere/shared/scripts/cron/ftp-need-restart.pl - control FTP server restart after synchronizing master and slave NetApp FTP servers
    • On the master Web server, create IP map file:

      vi /hsphere2/local/config/httpd/sites/map_table.txt

      Insert one line of the following format:

      <Master_IP>|<Slave_IP>

    • To synchronize the load balanced mail servers on dedicated IPs, after you have configured the IP map file in the previous step, add the corresponding H-Sphere scripts to the crontab on the master Web server and the slave Web server.

      For the master Web server:

      */4 * * * * /hsphere/shared/scripts/cron/master-ipsynch.pl

      For the slave Web server:

      */4 * * * * /hsphere/shared/scripts/cron/slave-ipupdate.pl

    • On the master Web server, configure separate pid and log files for the master and the slave Web servers.

      1. Pid files (httpd.pid) for the master and the slave will be located in separate directories, logs1 for the master, and logs2 for the slave. Create these directories on the master server:

         mkdir /hsphere/local/var/httpd/logs/logs1
         mkdir /hsphere/local/var/httpd/logs/logs2
        file:///home/vlad/shiva/psoft/www/sysadmin/NAS_configuration_24.html.in
      2. Run the following commands to reconfigure Apache to write to separate log and pid files for the master and slave servers.

        a) On the master Web server:

        For Linux:

        perl -pi -e
        's:/hsphere/local/var/httpd/logs/httpd.pid:/hsphere/local/var/httpd/logs/logs1/httpd.pid:g' /etc/rc.d/init.d/httpd
        /hsphere/shared/apache/bin/apachectl /hsphere/local/config/httpd/httpd.conf
        perl -pi -e 's:/hsphere/local/var/httpd/logs/access_log
        \s:/hsphere/local/var/httpd/logs/access_log.1:g' /hsphere/local/config/httpd/httpd.conf
        perl -pi -e 's:/hsphere/local/var/httpd/logs/error_log
        \s:/hsphere/local/var/httpd/logs/error_log.1:g' /hsphere/local/config/httpd/httpd.conf /etc/logrotate.d/hsphere-apache
        perl -pi -e 's:/hsphere/local/var/httpd/logs/ssl_engine_log
        \s:/hsphere/local/var/httpd/logs/ssl_engine_log.1:g' /hsphere/local/config/httpd/httpd.conf /etc/logrotate.d/hsphere-apache
        

        For FreeBSD:

        perl -pi -e
        's:/hsphere/local/var/httpd/logs/httpd.pid:/hsphere/local/var/httpd/logs/logs1/httpd.pid:g' /hsphere/shared/apache/bin/apachectl
        /hsphere/local/config/httpd/httpd.conf
        perl -pi -e 's:/hsphere/local/var/httpd/logs/access_log
        \s:/hsphere/local/var/httpd/logs/access_log.1:
        g'/hsphere/local/config/httpd/httpd.conf
        perl -pi -e 's:/hsphere/local/var/httpd/logs/error_log
        \s:/hsphere/local/var/httpd/logs/error_log.1:g' /hsphere/local/config/httpd/httpd.conf /etc/logrotate.d/hsphere-apache
        perl -pi -e 's:/hsphere/local/var/httpd/logs/ssl_engine_log
        \s:/hsphere/local/var/httpd/logs/ssl_engine_log.1:g' /hsphere/local/config/httpd/httpd.conf /etc/logrotate.d/hsphere-apache

        b) On the slave Web server:

        For Linux:

        perl -pi -e
        's:/hsphere/local/var/httpd/logs/httpd.pid:/hsphere/local/var/httpd/logs/logs2/httpd.pid:g' /etc/rc.d/init.d/httpd
        /hsphere/shared/apache/bin/apachectl /hsphere/local/config/httpd/httpd.conf
        perl -pi -e 's:/hsphere/local/var/httpd/logs/access_log
        \s:/hsphere/local/var/httpd/logs/access_log.2:g' /hsphere/local/config/httpd/httpd.conf
        perl -pi -e 's:/hsphere/local/var/httpd/logs/error_log
        \s:/hsphere/local/var/httpd/logs/error_log.2:g' /hsphere/local/config/httpd/httpd.conf /etc/logrotate.d/hsphere-apache
        perl -pi -e 's:/hsphere/local/var/httpd/logs/ssl_engine_log
        \s:/hsphere/local/var/httpd/logs/ssl_engine_log.2:g' /hsphere/local/config/httpd/httpd.conf /etc/logrotate.d/hsphere-apache
        

        For FreeBSD:

        perl -pi -e
        's:/hsphere/local/var/httpd/logs/httpd.pid:/hsphere/local/var/httpd/logs/logs2/httpd.pid:g' /hsphere/shared/apache/bin/apachectl
        /hsphere/local/config/httpd/httpd.conf
        perl -pi -e 's:/hsphere/local/var/httpd/logs/access_log
        \s:/hsphere/local/var/httpd/logs/access_log.2:g' /hsphere/local/config/httpd/httpd.conf
        perl -pi -e 's:/hsphere/local/var/httpd/logs/error_log
        \s:/hsphere/local/var/httpd/logs/error_log.2:g' /hsphere/local/config/httpd/httpd.conf /etc/logrotate.d/hsphere-apache
        perl -pi -e 's:/hsphere/local/var/httpd/logs/ssl_engine_log
        \s:/hsphere/local/var/httpd/logs/ssl_engine_log.2:g' /hsphere/local/config/httpd/httpd.conf /etc/logrotate.d/hsphere-apache
      3. Restart Apache on both boxes and check if pid and log files exist on the master server:

        /hsphere/local/var/httpd/logs/logs1/httpd.pid - master pid file
        /hsphere/local/var/httpd/logs/logs2/httpd.pid - slave pid file
        /hsphere/local/var/httpd/logs/access_log.1 - master access log file
        /hsphere/local/var/httpd/logs/access_log.2 - slave access log file
        /hsphere/local/var/httpd/logs/error_log.1 - master error log file
        /hsphere/local/var/httpd/logs/error_log.2 - slave error log file
        /hsphere/local/var/httpd/logs/ssl_engine_log.1 - master ssl engine log file
        /hsphere/local/var/httpd/logs/ssl_engine_log.2 - slave ssl engine log file

 

Step 5. Configure Master and Slave Mail Servers

  1. On each mail box, mount the mail storage partition to the /mnt/NAS directory:

    mkdir /mnt/NAS
    mount -t nfs <NAS_IP>:/vol/hsphere/mail /mnt/NAS

  2. Copy the following directories to the mountpoint directory on the NetApp NAS:

    cp -prv /hsphere/local/var/vpopmail /mnt/NAS/
    cp -prv /var/qmail/control /mnt/NAS/
    cp -prv /var/qmail/users /mnt/NAS/

  3. Configure /etc/fstab for mail servers:

    vi /etc/fstab

    On both the master and the slave mail servers /etc/fstab should contain the following lines:
    <NAS_IP>:/vol/hsphere/mail/vpopmail   /hsphere/local/var/vpopmail  nfs  rw  0 0
    <NAS_IP>:/vol/hsphere/mail/control   /var/qmail/control  nfs  rw  0 0
    <NAS_IP>:/vol/hsphere/mail/users   /var/qmail/users  nfs  rw  0 0
    
  4. On the master mail server, crontab contains all necessary H-Sphere scripts.
  5. On the slave mail server, crontab SHOULD NOT contain any H-Sphere scripts!
 

Step 6. Configure CP Server To Implement NetApp Support

On the H-Sphere CP server:

  1. Install suidperl
    - for Linux, it could be installed, for example, like this:

    rpm -ivh perl-suidperl-5.6.1-34.99.6.i386.rpm

    - for FreeBSD, it is already installed into the system.
  2. Set permissions for the script fileserver-quota.pl:

    chmod +s /hsphere/shared/scripts/fileserver-quota.pl

  3. Set SUPPORT_NET_APP property in the hsphere.properties file:

    SUPPORT_NET_APP=TRUE

  4. Mount /etc/ directory on the netapp fileserver to the /hsphere/NAS_IP/etc directory on CP server, where NAS_IP is NetApp NAS IP.

    mkdir /hsphere/NAS_IP/etc
    mount NAS_IP:/etc /hsphere/NAS_IP/etc

  5. Check the /hsphere/<NAS_IP>/etc/quotas. There should be a line like this:

    * user@/vol/hsphere/home 20000M 160K

  6. Check that rsh and quota are enabled:

    rsh <NAS_IP> quota report

  7. Add physical servers (master and slave) with their IPs to the system.
  8. Add master logical servers to the newly added physical servers (there's no need to add slave logical servers).
 

Step 7. Add Web/Mail Servers to LoadBalanced H-Sphere Clusters

For web clusters, each master server can have only one slave server. This means that when adding web servers to H-Sphere LB cluster, you have to set them up in pairs. Multiple web servers are possible only in H-Sphere 2.5 and up.

For mail clusters, multiple slave servers are possible.

To add web/mail servers to load balanced H-Sphere clusters:

  1. Install H-Sphere on Web/Mail Servers. Make sure to install on these servers the same version of H-Sphere as is running on your cp server.
  2. Configure master and slave web servers.
  3. Configure master and slave mail servers.
  4. Add physical servers (master and slave) with their IPs to the system via the interface.
  5. Add master logical servers to the newly added physical servers (there's no need to add slave logical servers). All management is run on masters only but there are scripts to replicate changes for slaves.
  6. In admin CP, logical Web server properties (E.Manager -> L.Servers), set File Server and File Path parameters for master Web server, where:
    - File Server is NetApp NAS name or IP-address, and qtree-name;
    - File Path is a path to the mounted filer storage directory.
  7. For example:

    File Server: NAS_IP:/QTREE
    File Path: NAS_PATH

Finally, start H-Sphere related Web/mail services (those you have stopped on Step 3) on master and slave servers to run H-Sphere with NetApp load balanced clusters.


Related Docs:   NetApp Configuration (HS 2.5) Updating H-Sphere On NetApp Filer



© Copyright 2017. Parallels Holdings. All rights reserved.