LoadBalanced Web Slave Recovery

Last modified: 18 Oct 2006

To restore crashed web slave server:

1. Install H-Sphere and install web service on this box.
2. Place the public OpenSSH keys into the /root/.ssh folder on slave box. You can find them on your master web box in the /root/.ssh/identity.pub and the /root/.ssh/id_dsa.pub files. Insert the keys from these files into the /root/.ssh/authorized_keys and the /root/.ssh/authorized_keys2 files correspondingly on slave web box. Check ssh connection from master to slave running the following command from master web servers root account
   #ssh IP
   Where IP is an IP of slave web server.
   Answer yes if prompted. You should be able to connect to slave without password prompt.
   Log to this server under root and create hsphere and hsphere2 directories:

    mkdir /hsphere
    mkdir /hsphere2

3. Let's assume we are going to restore linuxweb2-001 slave web server. H-Sphere content is located on your Filer under /vol/hsphere/home/linuxweb2-001 path. Add the following lines to /etc/fstab:

          
    <NetApp_IP>:/vol/hsphere/home/linuxweb2-001 /hsphere nfs defaults,nfsvers=3  0 0
    <NetApp_IP>:/vol/home/linuxweb1-001 /hsphere2 nfs defaults,nfsvers=3  0 0
    <NetApp_IP>:/vol/hsphere/home/linuxweb1-001/local/home /hsphere/local/home nfs defaults,nfsvers=3  0 0
    <NetApp_IP>:/vol/hsphere/home/linuxweb1-001/local/var/statistic /hsphere/local/var/statistic nfs defaults,nfsvers=3  0 0 
    <NetApp_IP>:/vol/hsphere/home/linuxweb1-001/local/var/httpd/logs /hsphere/local/var/httpd/logs nfs defaults,nfsvers=3  0 0
    <NetApp_IP>:/vol/hsphere/home/linuxweb1-001/frontpage /usr/local/frontpage nfs defaults,nfsvers=3  0 0
    <NetApp_IP>:/vol/hsphere/home/linuxweb1-001/ssl.shared /hsphere/local/config/httpd/ssl.shared nfs defaults,nfsvers=3  0 0 

   Where <NetApp_IP> is the IP of your Filer.
4. Mount all the mount points from /etc/fstab:
   mount -a
5. Execute:

   
    perl -pi -e's:/hsphere/local/var/httpd/logs/httpd.pid:/hsphere/local/var/httpd/logs/logs2/httpd.pid:g'/etc/rc.d/init.d/httpd 
   

6. Kill all httpd processes on the box:
   killall -9 httpd
7. Start httpd on the box:
   /etc/rc.d/init.d/httpd start
8. Put the following lines to root's cron:

           1 */4 * * * /usr/sbin/ntpdate ntps1-{0,1,2}.uni-erlangen.de
           0 1 * * * nice -15 /hsphere/shared/scripts/cron/cron_analyze.pl
           0 2 * * * nice -15 /hsphere/shared/scripts/cron/cron_rotate.pl
           0 3 * * * nice -15 /hsphere/shared/scripts/cron/ftp_anlz.pl
           0 4 * * * nice -15 /hsphere/shared/scripts/cron/ftp_anlz_user.pl
           */2 * * * * nice
           -15 /hsphere/shared/scripts/cron/apache-need-restart.pl
           */2 * * * * nice
           -15 /hsphere/shared/scripts/cron/apache-restart.pl
           */2 * * * * nice
           -15 /hsphere/shared/scripts/cron/ftp-need-restart.pl
           */3 * * * * nice -15 /hsphere/shared/scripts/cron/ftp-restart.pl
           0 5 * * * nice -15 /hsphere/shared/scripts/cron/db_usage.pl
           */4 * * * * /hsphere/shared/scripts/cron/slave-ipupdate.pl

   Where IP is an IP of slave web server.