Section 7: Service Recovery

The Big Picture: What If the Worker Collapses?

If a hotel cleaner faints, the manager does not close the hotel. The manager has a plan: first call the cleaner back to work. If that fails, call a replacement. If it keeps happening, call the owner.

Windows has a plan too. For each service you can set recovery actions: what the SCM should do when the service fails. This is why a crashed service often comes back by itself and you never notice.

What counts as a "failure"? The service process ends unexpectedly (it crashes), or the service reports that it stopped with an error. A service stopped normally on purpose (by you pressing Stop) is not a failure.

7.1 First/Second/Subsequent Failure Actions

What it is: Windows lets you choose a different action for each of the first three failures:

The four choices for each slot:

Other settings in the same tab:

Where it is stored. The registry value FailureActions in the service's key (Section 2.2). It holds the reset period, an optional reboot message, an optional command, and the list of actions with delays. You do not edit it by hand.

How to see it (read-only):

• In services.msc: double-click a service, open the Recovery tab.
• In a command prompt:

sc qfailure Spooler

You will see lines like RESET_PERIOD, FAILURE_ACTIONS, and entries such as RESTART -- Delay = 60000 milliseconds. (Delays are in milliseconds: 60000 is 1 minute.)

To change it (only on a service you own, as administrator):

sc failure <ServiceName> reset= 86400 actions= restart/60000/restart/60000//

That says: reset the counter after 86400 seconds (1 day); restart after 60,000 ms on the first failure; restart after 60,000 ms on the second; do nothing afterwards. Notice the reset= and actions= words again need the space after the equal sign.

A typical safe plan:

 Failure 1 ──► Restart the service (after 1 minute)
 Failure 2 ──► Restart the service (after 1 minute)
 Failure 3+ ─► Take no action  (and send an alert so a human looks)

Why not restart forever? A service that fails again and again within seconds may have a real fault. Endless restarts waste CPU, fill logs and can hide the problem. The three-slot plan stops that.

7.2 Restart Service Option

What it is: The most common action. The SCM starts the service again, as if you had asked yourself.

What really happens:

1. The service process crashes.
2. The SCM notices (it is watching the process).
3. It writes a message to the System log (Event ID 7031 when a recovery action is taken, 7034 when the service stopped unexpectedly with no action).
4. It waits for the delay you set.
5. It starts the service again, and first starts any of its dependencies that are down (Section 4).

 RUNNING ──► crash ──► SCM sees it ──► waits (delay) ──► START_PENDING ──► RUNNING
                         │
                         └─► logs "terminated unexpectedly" in the System log

Good to know:

Real-world analogy: the manager walks to the cleaner, helps them up, and tells them to go back to work.

7.3 Run a Program on Failure

What it is: The SCM starts a program or script of your choice when the service fails. You supply the file path and any command-line options.

What people use it for:

Where to set it. Recovery tab, choose Run a Program, then fill in Program and Command line parameters. There is also a tick box Append fail count to end of command line (/fail=%1%) so the script knows which failure number this is.

Safety points:

• The program runs with the service's account and in session 0. That is usually LocalSystem, so a careless script is powerful.
• It cannot show windows.
• Keep it short and quick. Use full paths to the program (never a bare name).
• Protect the script file so ordinary users cannot edit it. If a normal user can change a file that LocalSystem runs, they can take over the PC.

Warning: Do not point this at a program you do not trust. Treat it as an administrator-level action.

Real-world analogy: the manager calls the owner by phone and writes down what happened.

7.4 Reboot Computer on Failure

What it is: The SCM restarts the whole PC. It is the strongest action and is meant for services without which the computer is useless.

Extras you can set: a delay before the reboot and an optional message that is sent to users ("Restarting in 1 minute because service X failed").

When it makes sense:

When it does not make sense:

Safe order. Put reboot only on the last slot (third failure and later), after restarts have been tried. Pair it with a reasonable reset period.

Warning: A reboot action on a service that fails at boot can trap a PC in a loop. If that happens, boot into Safe Mode (Chapter 6 introduced boot options) and fix the recovery settings.

Real-world analogy: the owner closes the whole hotel for an hour to reset everything. It works, but everyone is disturbed.

Summary