Are you familiar with the Availability Equation? In a nutshell, this equation shows how the total time needed to restore an application to usability is equal to the time required to detect that an application is experiencing a problem plus the time required to perform a recovery action:
TRESTORE = TDETECT + TRECOVER
The equation introduces the key concepts of high availability (HA): clustering, problem detection, and subsequent recovery. HA solutions monitor the health of business application components; when problems are detected, these solutions act to restore them to service. The objective of deploying an HA solution is to minimize downtime.
Reducing detection and recovery times are two important tasks of any HA solution that you choose to deploy. Today’s applications are combinations of technologies: servers, storage, network infrastructure, and so on. When reviewing your HA options, be certain that you understand the technologies that each solution uses to detect and recover from all outage types. Each technology has a direct impact on service restoration times.
One technology that is crucial to providing the fastest possible restoration time is known as local detection and recovery (aka service-level problem detection and recovery). In a basic clustering solution, servers are connected and configured such that one or more servers can take over the operations of another in the event of a server failure. The server nodes in the cluster continuously send small data packets, often called heartbeat signals, to each other to indicate that they are “alive”.
In simple clustered environments, when one server stops generating heartbeats, other cluster members assume that this server is down and begin the process of taking over responsibility for that server’s domain of operation. This approach is adequate for detecting failure at the server level. But unless problems cause the interruption or cessation of heartbeat signals, server-level detection is inadequate. More than that, it can actually magnify the extent and impact of an outage.
For example, if Apache processes hang, the server may still send heartbeats — even though the Web server subsystem has ceased to perform its primary function. Rather than restart the Apache subsystem on the same or a different server, a basic server-level clustering solution would restart the entire software stack of the failed server on a backup server, thereby causing interruption to users and extending recovery time.
Using local detection and recovery, advanced clustering solutions deploy health-monitoring agents within individual cluster servers, to monitor individual system components such as a file system, a database, user-level application, IP address, and so on. These agents use heuristics that are specific to the monitored component. Therefore, the agents can predict and detect operational issues and then take the most appropriate recovery action. Often, the most efficient recovery method is to stop and restart the problem subsystem on the same server.
By detecting failures at a more granular level than simply by observing server-level heartbeats, and by enabling recovery within the same physical server, the time to restore an application to user availability can be greatly reduced. Solutions such as the SteelEye Protection Suite for Linux from SIOS provides this level of detection and recovery for your environment. Make certain that whichever HA solution you deploy can also support local detection and recovery.