Renaming Veritas Cluster Server Nodes

I recently found myself with the need to rename both nodes in my two node VCS deployment because they’d been built using the wrong naming convention. The OS on each node is RHEL 5.2 running VCS 5.0MP3.

The system hostname is unimportant to VCS. VCS uses the file /etc/VRTSvcs/conf/syname, which is created during the software installation, for use within the cluster configuration. To make the change at the OS level as well as the cluster level this is the procedure I followed. All commands were run as root. If any of the VCS related configuration files don’t have matching nodenames or the main.cf file becomes broken, you can be left with a completely broken cluster. Proceed with caution and check your changes before you save them.

Dump a known good copy of the cluster config and make the config read-only (From node 1)

/opt/VRTS/bin/haconf -dump -makero

Stop VCS on both nodes

/opt/VRTS/bin/hastop -all -force

Stop remaining Veritas Fencing Driver (Run on Both Nodes)

/etc/init.d/vxfen stop
/etc/init.d/llt stop
/etc/init.d/gab stop
/etc/init.d/vxpal.actionagent stop
/etc/init.d/vxpal.gridnode stop
/etc/init.d/vxpal.StorageAgent stop

After issuing the hastop command above, the stop on llt and gab aren’t required, but those modules didn’t unload after stopping the fencing driver and after I’d stopped them again. I began to stop other VRTS related processes and after stopping the three vxpal services all the drivers I was concerned about were gone. In this case I think the culprit may have been the gridnode driver. I may have been impatient, but this procedure won’t cause any harm.

Confirm that the vxfen, llt and gab kernel modules are no longer loaded

/sbin/lsmod | grep -e vxfen grep -e llt -e gab

Edit the following files on both nodes with the editor of choice. Change each instance of node1 and node2 with their respective new names

/etc/llttab
/etc/llthosts
/etc/VRTSvcs/conf/sysname
/etc/sysconfig/network
On node 1 ONLY
/etc/VRTSvcs/conf/config/main.cf

Verify config

hacf -verify /etc/VRTSvcs/conf/config

A reboot is required for the OS level re-naming

/sbin/init 6

After the reboot you’ll want to verify that VCS is running and that all resources configured in the cluster are operating as expected. For me a quick check is

/sbin/gabconfig -a
/sbin/lltstat
/opt/VRTS/bin/hastatus

Creating an Automated Reverse Proxy with Apache

Overview

In this, the inaugural post of my little slice of the blog-o-sphere, I’m going to describe a project that I just launched at work. At a high level what was accomplished was allowing members of our various development groups to expose applications to the public for a short period of time in a self-service manner. The high level architectural diagram is shown below.

The Details

The goals for this project were straight forward and well defined.

Allow outside access to internal applications

Access to the applications must be authenticated

Self-Service provisioning of the external access configuration

Let’s start from the top down and describe the configuration at each level and what role each component is playing in the overall project.

DNS

Externally, a wildcard A record was put in the zone that the applications live in. Having this setup in place makes it so that as soon as access for a new application is configured, the DNS works. We run Bind on our external name servers so the single entry looks like this.

* IN A 192.168.42.42

Networking

The networking in place is fairly simple. There is a single static NAT in place between the access layer routers and DMZ load balancers on our network. The NAT terminates on the load balancers, sitting on RFC 1918 space, which send the traffic to a pool containing the proxy servers.

Apache

The apache web server was chosen to act as the proxy server for this project. The apache servers are where the reverse proxying and authentication happen. We’re using the stock apache 2.0.xx that comes with RHEL 4. The load balancers round robin requests (with source address stickiness) to these servers. Each application that is configured to be proxyed lives in a unique configuration file. The benefit to doing this is two fold. One is that it allows for adding new configurations and only require a reload of the httpd service. Two is that it allows us to disable and delete configurations simply by changing the file extention when the time comes. In the main /etc/httpd/conf/httpd.conf we have an Include statement to catch all our proxy configurations.

Include /etc/httpd/conf.d/proxy/*.conf

The configuration files are generated via a form based web application and then moved over to the proxy servers via SFTP. THe files are moved in batches via a cronjob which also reloads each apache instance. An example configuration file looks like this:

  <VirtualHost *:80>
  ServerName appname.dev.example.com
  LoadModule ldap_module modules/mod_ldap.so
  LoadModule auth_ldap_module modules/mod_auth_ldap.so
  LoadModule proxy_module modules/mod_proxy.so
  LoadModule proxy_http_module modules/mod_proxy_http.so
  LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
  LogFormat "%h %l %u %t \"%r\" %>s %b" common
  LogFormat "%{Referer}i -> %U" referer
  LogFormat "%{User-agent}i" agent
  LogLevel warn
  ErrorLog /etc/httpd/logs/proxy/appname_error.log
  CustomLog /etc/httpd/logs/proxy/appname_access.log combined
  
  ProxyRequests Off
  <Proxy *>
    Order deny,allow
    Allow from all
  </Proxy>
  
  ProxyPass / http://appserver.dev.example.com:8042/
  ProxyPassReverse / http://appserver.dev.example.com:8042/
  
  <Location />
  AuthLDAPEnabled on
  AuthLDAPAuthoritative on
  AuthName "Application Proxy Access"
  AuthType basic
  AuthLDAPBindDN "CN=xxxxxxx,OU=proxy,DC=example,DC=com"
  AuthLDAPBindPassword "xxxxxxxx"
  AuthLDAPURL "ldap://ad.example.com:3268/DC=example,DC=com?sAMAccountName?sub?(objectClass=user)"
  Require user extproxytest
  </Location>
  </VirtualHost>
  

Most of the configuration is basic and standard if you’ve worked with Apache before. However, there are three bits of the configuration that are worth expanding upon.

1. ServerName – The ServerName directive is important because we’re using name based virtual hosting. The HTTP headers in the GET request will contain the ServerName allowing the proxies to send the traffic to the proper host on the backend.
2. Proxy Configuration – The proxy configuration is done via mod_proxy. Since the servers are setup to be reverse proxies, we turn ProxyRequests off. We then follow a standard configuration for a reverse proxy and send all requests for this vhost to the specified backend server.
3. Authentication – The authentication is done against an active directory domain via mod_ldap. Specifically the authentication requests are made against a global catalog for improved performance. When the access for a new application is configurated a unique user for that application is created in the specified proxy OU.

There you have it a simple yet robust, self-service reverse proxy system. The result is that any developer can fill out a simple web form and 15 minutes later have a new application exposed to the outside world in a controlled, reliable and secure manner. I’d be happy to hear any comments or questions about this.