Proper System Properties

In sandboxed programming platforms like .Net and Java, the developer is insulated from the details of the underlying machine upon which the code is running. At times, however, it is often useful to have more knowledge about the environment. One way to provide such information without introducing any machine-dependent code into the sandbox is via system properties. They can be implemented in a number of ways, but the most common is a set of name/value pairs that can be accessed via a core class in the language runtime. In java, one can get a list of all the system properties using System.getProperties().

The properties are not limited to those set by the language virtual machine; they can be extended to include any property the developer wants to set. Most languages provide a way to do this at virtual machine initialization or programmatically within the application code. Having system properties is a powerful tool and can enable code to handle certain conditions based on the capabilities of the underlying machine in an elegant way. With power, though, comes responsibility.

Some developers have been abusing this construct and using it as if were a container for global variables: instead of passing parameters into methods, a method will simply look up a value from the system properties. This is especially egregious when the property is not universally true from the system perspective. One such example is specifying a HTTP proxy. In Java, the default HttpConneciton class uses system properties to specify a proxy. In most cases, this is fine, but in some enterprise environments, the proxy server one must use depends on the address being accessed. If your program needs to access two addresses that use different proxies, then changing proxies involves updating the values in the system properties. Again, this is usually not a big problem, but if you have a multi-threaded program, this could cause a race condition unless one enforces the atomicity of the property update with the HTTP request. Depending on the application requirements, this may or may not be an acceptable answer.

While there is not much one can do about the proxy issue (outside of using some other library to make HTTP requests), developers can prevent imposing similar limitations on their own code. Since it is impossible to anticipate every way in which code may be used and it is equally impossible to know every possible system configuration, the use of system properties that do not actually describe the system upon which the code is executing should be avoided.

Threads & Exceptions

In this age of multi-core processors, being able to write efficient and correct multi-threaded programs is becoming increasingly important. Unfortunately, however, writing such programs is not a trivial task. Deadlocks, race conditions, and thread starvation are among the myriad pitfalls that could break one's programs. Complicating matters further are “legacy” (for lack of a better term) third-party libraries that may or may not be thread-safe.Errors due to threading issues are among the worst class of bugs: the elusive Heisenbug. The use of a good profiler tool is essential in tracking down and eradicating bugs like this.

Just as important as knowing how to avoid writing code subject to deadlocks and race conditions is knowing how your programming environment responds to bugs that occur in child threads. In the Java programming language, for example, one's program could find itself in an unexpected state due to an uncaught exception in a child thread. When one is running a single-threaded program and an uncaught exception occurs the system will exit and print the exception's stack trace to standard error. Not graceful, but also not a very subtle error condition. When an uncaught exception occurs in a child thread, however, the flow of control is as follows:

1. The JVM checks the UncaughtExceptionHandler of the Thread object. This is null by default. If you wish to specify your own handler, you must call the setUncaughtExceptionHandler method on the thread instance (usually before starting its execution). Setting this handler only applies to the thread instance upon which you installed the handler (not descendant threads).
2. If the thread does not have an UncaughtExceptionHandler, then the ThreadGroup is checed to see if it has a handler of its own.


3. If the thread's ThreadGroup does not provide a handler, then the default handler is invoked. Unless the programmer changes it (by calling the static Thread.setDefaultUncaughtExceptionHandler method) the default handler simply prints the exception's stack trace to standard error.

The Java API documentation contains a very important warning for those programmers who want to provide their own default UncaughtExceptionHandler:

The default uncaught exception handler should not usually defer to the thread's ThreadGroup object, as that could cause infinite recursion.

When developing code that has any potential for reuse in other applications, one has a responsibility to take threading considerations into account. If it is decided that a method or class will not be thread-safe, then it is the developer's responsibility to clearly document this limitation in both the in-code documentation (i.e. JavaDoc) and any other API/Development guides.

Oracle RAC Load Balancing

Relational databases play a central role in many software systems. In many enterprises, Oracle is the RDBMS of choice. Among those, many mission-critical systems make use of Oracle's Real Application Cluster (RAC), a load-balanced multi-node database that can handle failures of individual nodes without causing an outage. This article provides a decent summary of RAC.

While RAC has many features and advantages (as well as trade-offs), that isn't why this post is here. It is here because the way in which the Java ODBC driver establishes a connection to the RAC is worth understanding in the event that one suspects a connectivity issue between the client code and the database server.

For a Java developer, using RAC is very similar to any other Oracle database. Usually, the only indication that you're using a RAC is the contents of your connect string. It typically lists a number of database server addresses in an ADDRESS_LIST element. For the purposes of illustrating the discussion, an example may be helpful:

jdbc:oracle:thin:@(DESCRIPTION =(ADDRESS_LIST =(ADDRESS = (PROTOCOL = TCP)(HOST = dbserver1.somedomain.com)(PORT = 12345))(ADDRESS = (PROTOCOL = TCP)(HOST = dbserver2.somedomain.com)(PORT = 12345))(LOAD_BALANCE = yes)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = dbServiceName.somedomain.com) ) )

There are two types of load balancing offered by the RAC: client-based load balancing and server-based load balancing. The system can be configured to use one, both or neither forms of load balancing.

The LOAD_BALANCE element in the connect string above specifies that the client code should perform its own load balancing. Essentially, this means that the client will randomly choose one of the servers from the ADDRESS_LIST whenever a connection is requested.

If server-based load balancing is enabled, a listener service provides automatic load distribution across all nodes in the RAC. The listener will use the query optimizer to determine which node in the RAC should service the request (based on their current workload, machine profiles and, in newer versions of Oracle, admin-specified rules).

If one were to use the connect string above and server-based load balancing were enabled, the sequence of events that would occur when a call to DriverManager.getConnection(...) is invoked are:

• The driver selects dbserver2.somedomain.com (this choice was made at random by the driver) and issues a connection request.
  
• The listener on dbserver2.somedomain.com determines that the most under taxed node in the RAC is dbserver3.somedomain.com and returns it to the client. NOTE: this assumes that the RAC has 3 nodes. It is important to note that all the nodes do NOT need to be included in the connect string... the nodes in the ADDRESS_LIST simply indicate which nodes the client will balance its initial "getConnection" requests against
  
• The client will attempt to establish a connection to dbserver3.somedomain.com.
  

Sudo Voodoo

The Unix sudo command can be quite useful when one needs to give another user access to a select group of commands that need to be run as a certain user without having to give up the password for that user account. The easiest way to do this is to edit the sudoers file (usually in /etc/sudoers). The file allows an administrator to configure a list of commands that a user can execute as another user without a password. The policy file follows a fairly simple context-free grammer (defined quite concisely in the man page for sudoers). There are a few things to keep in mind when editing the sudoers file:

• You must edit the file using the visudo command as root
• When defining a command, you must use the fully qualified path to the command. The system will report a syntax error if you attempt to save the sudoers file with a command that does not start with a /character. This makes sense when one condiders that the whole point of the command is to run something as another user. That being the case, the system can make no guarantees as to what the working directory will be when running the command so requiring the absolute path is a reasonable constraint.
• The command(s) defined in the sudoers file must exactly match the command to be run.

Here is an example of a simple sudoers configuration:

Cmnd_Alias SOME_COMMANDS = /home/somedir/command1.sh,/home/somedir/command2.sh

user1 ALL=(ALL) NOPASSWD: SOME_COMMANDS

The first line sets up a "command alias" or a list of commands specified using their absolute path with each command separated by a comma. If you needed to pass a comma into a command as an argument, for example, it would need to be escaped using the backslash character.

The next line tells the system that the user user1 is allowed to run any of the commands listed in SOME_COMMANDS as any other user without having to specify a password.

Once you've installed these lines in the sudoers policy file, the user1 user can log in and execute sudo -u user2/home/somedir/command1.sh to run the command1.sh script as user2 without ever having to provide user2's credentials.

For a full treatment of how to configure sudo policies, the man pages are the best resource.

A Dearth of Debugging

There seem to be two schools of thought for getting to the bottom of a piece of buggy software: those who log everything vs. those that use the debugger. I've been surprised to learn how large the former camp is, at least among those developing for web-based applications. I'm not saying that one should use one approach exclusively, just that the debugger is underused in web programming.

I suppose a reason for this is the perceived difficulty in debugging server-side code from a client machine, but this is more a case of lack of knowledge than an actual technical issue. Most application servers support some facility for remote debugging. Similarly, in Java, at least, the virtual machine itself can be configured to accept connections from remote debugger clients (if, for example, you wanted to configure the JVM to listen for debugger connections on port 8000, you'd include this in the command-line arguments: -Xdebug -Xnoagent -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8000). Once started in this way, the debugger can connect to the remote JVM and do everything it could do if it were running on the same machine.... you can set breakpoints, inspect the contents of memory, evaluate statements and see the full stack trace for all the running threads... all of which are quite powerful when looking for an elusive bug.

The heuristic I use for deciding whether to use the debugger or a logger to find the root of an issue is as follows:

If the bug occurs infrequently and it is not clear how to duplicate it, I use a logger to see if I can detect a pattern. If I can, then I'll use the debugger to look deeper into things. Conversely, if it's clear how to duplicate the bug, I'll usually jump right in with the debugger.


Here are some resources for remote debugging:

• JBoss: http://www.onjava.com/pub/a/onjava/2005/08/31/eclipse-jboss-remote-debug.html


• Tomcat: http://wiki.apache.org/tomcat/FAQ/Developing#Q1


• IIS: http://support.microsoft.com/kb/910448


• PHP: http://www.php-debugger.com/dbg/

Volatility

In the last few months, I've found myself writing a lot more multi-threaded programs than I usually do. This made me reflect upon some of the constructs Java provides for multi-threading. Two such constructs are the volatile and synchronized keywords. In my years working with a wide range of developers, I've found that most know (at least at a cursory level) what synchronized means, but very few know what volatile means or when it should be used.

The semantics of the volatile keyword are similar to those of the synchronized keyword, but they're not equivalent. When one thinks of multi-threaded programs, they have 3 primary properties: visibility - the ability of one thread to see the work of another thread, atomicity - defining sets of indivisible actions, and ordering - enforcing that some operations in one thread happen before others in another. Using synchronized, a developer can enforce all three properties. When one uses volatile, however, one cannot enforce atomicity.

Why would one want to use volatile over synchronized since it seems to be a less powerful construct? Well, for one thing, it incurs slightly less overhead than obtaining locks with synchronized. Additionally, volatile can be used on primitives and null values (you can't directly synchronize on a primitive). One common use case is for a flag value that is updated by one thread but read by another. Declaring it volatile ensures that the other thread is reading the current value and not its own cached copy.

Java is an interesting programming language for many reasons, not the least of which is the way it insulates the developer from so many concerns of the low-level machine. The difference between volatile and synchronized, however, illustrates that, despite this abstraction, developers are obligated to understand the inner workings of the JVM if they are to write correct and efficient code.