webOS vs. iOS

I recently got rid of the iPhone 3G I had carried for almost two years and got a Palm Pre Plus on Verizon Wireless. Why the hell would I do such a thing you ask? Because despite the iPhone being incapable of doing wrong and being the best mobile platform ever (did I forget to open the sarcasm tag there?), I was sick of it. The primary driving factor was AT&T’s network being a complete pile of crap and my complete frustration with their inability to do things any modern mobile carrier should be able to do (tethering anyone?). I was also pretty damn sick of how Apple handles their mobile platform (changes to the now infamous section 3.3.1 of the developer agreement aside).

What do I like about the palm pre and webos? So far, almost everything. First on the list is real multi-tasking. As in, I can run the mobile hotspot app and provide my laptop with internet connectivity, stream music from pandora, read my email, sync data via the missing sync and have a text message open while installing a new game from the app catalog. ALL AT THE SAME TIME. I can even, evidence from the non-suicidal Wilson brother to the contrary, browse the web while talking on the phone. I can copy and paste text from any one app into any other app. Not just the apps installed by default from Palm.

The fact that iPhone OS didn’t have these features until the “revolutionary” 4.0 release is bullshit on a level I have a hard time describing without screaming “WHAT THE FUCK”. Embedded operating systems have been multitasking capable for at least a decade and even a 2G iPhone is at least 10 times more powerful than my first computer, which by the way ran a multi-tasking operating system without much fuss. The arguments from rabid, apple fellating fanboys that there isn’t enough memory and that multitasking is hard and that apple wants to give them the best “experience” makes me want to kick them where they have the least padding, their wallets (or maybe the crotch, you can pick). These arguments are anywhere from misguided to so wrong I don’t know how to address people that believe them without wanting to get all fist-puncher on them.

HP’s purchase of Palm is a good thing. A really good thing. The webOS operating system now has the power of a multi-billion dollar company behind it, just like the inferior iPhone OS. HP has already annouced they are dropping Windows 7 in favor of webos on their Slate tablet project. If HP really puts the effort they say they will into webOS development, they will become a dominant force in the mobile space. You heard it here first (well maybe not even close to first, but mark those words). HP is going to make awesome mobile devices with webOS. Phones, PDAs and tablets. In fact the first webOS based tablet will be a difficult purchase for me to not make.

Is google listening?

I’ve had an iphone for almost two years. The device itself is okay and does everything I need but not everything I want. No multi-tasking, copy and paste only between apple applications and horrendous battery life are amazingly frustrating. I would love an Android based device, even though I don’t trust Google at all. So, why haven’t I run out an bought myself a Nexus One or a Motorola Droid? Because the Android platform doesn’t support active sync and that’s a requirement for pretty much anyone using a mobile device in a corporate setting. The day that Android supports the active sync protocol, I will be buying myself an Android based phone. I’ve been watching this thread on google code about the issue. Google recently announced that their mobile apps would support many active sync features (though they didn’t specifically mention active sync). The support for these features isn’t available in the mobile apps on android yet, but this is promising on that front as well as in the Android platform. Hopefully Google is listening and this support will be announced before summer.

Renaming Veritas Cluster Server Nodes

I recently found myself with the need to rename both nodes in my two node VCS deployment because they’d been built using the wrong naming convention. The OS on each node is RHEL 5.2 running VCS 5.0MP3.

The system hostname is unimportant to VCS. VCS uses the file /etc/VRTSvcs/conf/syname, which is created during the software installation, for use within the cluster configuration. To make the change at the OS level as well as the cluster level this is the procedure I followed. All commands were run as root. If any of the VCS related configuration files don’t have matching nodenames or the main.cf file becomes broken, you can be left with a completely broken cluster. Proceed with caution and check your changes before you save them.

Dump a known good copy of the cluster config and make the config read-only (From node 1)

/opt/VRTS/bin/haconf -dump -makero

Stop VCS on both nodes

/opt/VRTS/bin/hastop -all -force

Stop remaining Veritas Fencing Driver (Run on Both Nodes)

/etc/init.d/vxfen stop
/etc/init.d/llt stop
/etc/init.d/gab stop
/etc/init.d/vxpal.actionagent stop
/etc/init.d/vxpal.gridnode stop
/etc/init.d/vxpal.StorageAgent stop

After issuing the hastop command above, the stop on llt and gab aren’t required, but those modules didn’t unload after stopping the fencing driver and after I’d stopped them again. I began to stop other VRTS related processes and after stopping the three vxpal services all the drivers I was concerned about were gone. In this case I think the culprit may have been the gridnode driver. I may have been impatient, but this procedure won’t cause any harm.

Confirm that the vxfen, llt and gab kernel modules are no longer loaded

/sbin/lsmod | grep -e vxfen grep -e llt -e gab

Edit the following files on both nodes with the editor of choice. Change each instance of node1 and node2 with their respective new names

/etc/llttab
/etc/llthosts
/etc/VRTSvcs/conf/sysname
/etc/sysconfig/network
On node 1 ONLY
/etc/VRTSvcs/conf/config/main.cf

Verify config

hacf -verify /etc/VRTSvcs/conf/config

A reboot is required for the OS level re-naming

/sbin/init 6

After the reboot you’ll want to verify that VCS is running and that all resources configured in the cluster are operating as expected. For me a quick check is

/sbin/gabconfig -a
/sbin/lltstat
/opt/VRTS/bin/hastatus

Threaded SSH executer in Python

In addition to my full time job I also do some consulting on the side. Recently, I’ve been working on auditing one of my clients’ infrastructure. The results of the audit will be in the form of recommendations for updates/upgrades of their operating systems, packages, patches and software. This involves gathering the same data for every host in their environment and then analyzing it to produce the results. To make the gathering easier and to try to increase my Python fu I wrote a script using the threading and paramiko python modules. The threading module is part of the python core and paramiko is a pure python SSH2 implementation. The script reads a list of hosts stored in a file and forks a thread for each. In each thread the script logs into a host and executes any arbitrary command on the host printing the results to STDOUT. Because the start() method in the threading module is non-blocking the for loop is appropriate, but it also means that the main script could exit before all the threads return. That is why each thread is appended to the threads list object and then joined. I think the script itself is fairly simple and I’m sure there is room for improvement, but this was my first foray into threaded programming and I’m still learning python. Comments are welcome.

#!/usr/bin/env python
"""
The MIT License

Copyright (c) 2009 seano 

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
"""

import paramiko, threading

# setup the threading class and routine
class SSHThread (threading.Thread):
    def run (self):
        # SSH connection settings
        hostname = self.hostname
        port = 22
        username = 'username'
        ssh_pkey = '/home/username/.ssh/id_dsa'

        # make the connection and run the commands
        key = paramiko.DSSKey.from_private_key_file(ssh_pkey)
        conn = paramiko.SSHClient()
        conn.load_system_host_keys()
        conn.connect(hostname, port, username=username, pkey=key)
        stdin, stdout, stderr = conn.exec_command("/bin/ls")
        print stdout.read().strip()
        conn.close()

    def __init__(self,hostname):
        threading.Thread.__init__(self)
        self.hostname = hostname

def main():
    hostlist = '/home/username/.machines'

    threads = []
    for host in open(hostlist, 'r').readlines():
        hostname = host.strip()
        current_thread = SSHThread(hostname)
        threads.append(current_thread)
        current.start()

    for t in threads:
        t.join()

if __name__ == "__main__":
    main()

Creating an Automated Reverse Proxy with Apache

Overview

In this, the inaugural post of my little slice of the blog-o-sphere, I’m going to describe a project that I just launched at work. At a high level what was accomplished was allowing members of our various development groups to expose applications to the public for a short period of time in a self-service manner. The high level architectural diagram is shown below.

The Details

The goals for this project were straight forward and well defined.

Allow outside access to internal applications

Access to the applications must be authenticated

Self-Service provisioning of the external access configuration

Let’s start from the top down and describe the configuration at each level and what role each component is playing in the overall project.

DNS

Externally, a wildcard A record was put in the zone that the applications live in. Having this setup in place makes it so that as soon as access for a new application is configured, the DNS works. We run Bind on our external name servers so the single entry looks like this.

* IN A 192.168.42.42

Networking

The networking in place is fairly simple. There is a single static NAT in place between the access layer routers and DMZ load balancers on our network. The NAT terminates on the load balancers, sitting on RFC 1918 space, which send the traffic to a pool containing the proxy servers.

Apache

The apache web server was chosen to act as the proxy server for this project. The apache servers are where the reverse proxying and authentication happen. We’re using the stock apache 2.0.xx that comes with RHEL 4. The load balancers round robin requests (with source address stickiness) to these servers. Each application that is configured to be proxyed lives in a unique configuration file. The benefit to doing this is two fold. One is that it allows for adding new configurations and only require a reload of the httpd service. Two is that it allows us to disable and delete configurations simply by changing the file extention when the time comes. In the main /etc/httpd/conf/httpd.conf we have an Include statement to catch all our proxy configurations.

Include /etc/httpd/conf.d/proxy/*.conf

The configuration files are generated via a form based web application and then moved over to the proxy servers via SFTP. THe files are moved in batches via a cronjob which also reloads each apache instance. An example configuration file looks like this:

  <VirtualHost *:80>
  ServerName appname.dev.example.com
  LoadModule ldap_module modules/mod_ldap.so
  LoadModule auth_ldap_module modules/mod_auth_ldap.so
  LoadModule proxy_module modules/mod_proxy.so
  LoadModule proxy_http_module modules/mod_proxy_http.so
  LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
  LogFormat "%h %l %u %t \"%r\" %>s %b" common
  LogFormat "%{Referer}i -> %U" referer
  LogFormat "%{User-agent}i" agent
  LogLevel warn
  ErrorLog /etc/httpd/logs/proxy/appname_error.log
  CustomLog /etc/httpd/logs/proxy/appname_access.log combined
  
  ProxyRequests Off
  <Proxy *>
    Order deny,allow
    Allow from all
  </Proxy>
  
  ProxyPass / http://appserver.dev.example.com:8042/
  ProxyPassReverse / http://appserver.dev.example.com:8042/
  
  <Location />
  AuthLDAPEnabled on
  AuthLDAPAuthoritative on
  AuthName "Application Proxy Access"
  AuthType basic
  AuthLDAPBindDN "CN=xxxxxxx,OU=proxy,DC=example,DC=com"
  AuthLDAPBindPassword "xxxxxxxx"
  AuthLDAPURL "ldap://ad.example.com:3268/DC=example,DC=com?sAMAccountName?sub?(objectClass=user)"
  Require user extproxytest
  </Location>
  </VirtualHost>
  

Most of the configuration is basic and standard if you’ve worked with Apache before. However, there are three bits of the configuration that are worth expanding upon.

1. ServerName – The ServerName directive is important because we’re using name based virtual hosting. The HTTP headers in the GET request will contain the ServerName allowing the proxies to send the traffic to the proper host on the backend.
2. Proxy Configuration – The proxy configuration is done via mod_proxy. Since the servers are setup to be reverse proxies, we turn ProxyRequests off. We then follow a standard configuration for a reverse proxy and send all requests for this vhost to the specified backend server.
3. Authentication – The authentication is done against an active directory domain via mod_ldap. Specifically the authentication requests are made against a global catalog for improved performance. When the access for a new application is configurated a unique user for that application is created in the specified proxy OU.

There you have it a simple yet robust, self-service reverse proxy system. The result is that any developer can fill out a simple web form and 15 minutes later have a new application exposed to the outside world in a controlled, reliable and secure manner. I’d be happy to hear any comments or questions about this.