How to continue a task when Fabric receives an error

When I define a task to run on several remote servers, if the task runs on server one and exits with an error, Fabric will stop and abort the task. But I want to make fabric ignore the error and run the task on the next server. How can I make it do this?

For example:

$ fab site1_service_gw
[site1rpt1] Executing task 'site1_service_gw'

[site1fep1] run: echo 'Nm123!@#' | sudo -S route
[site1fep1] err:
[site1fep1] err: We trust you have received the usual lecture from the local System
[site1fep1] err: Administrator. It usually boils down to these three things:
[site1fep1] err:
[site1fep1] err:     #1) Respect the privacy of others.
[site1fep1] err:     #2) Think before you type.
[site1fep1] err:     #3) With great power comes great responsibility.
[site1fep1] err: root's password:
[site1fep1] err: sudo: route: command not found

Fatal error: run() encountered an error (return code 1) while executing 'echo 'Nm123!@#' | sudo -S route '

Aborting.

From the docs:

> ... Fabric defaults to a “fail-fast” behavior pattern: if anything goes wrong, such as a remote program returning a nonzero return value or your fabfile’s Python code encountering an exception, execution will halt immediately. > > This is typically the desired behavior, but there are many exceptions to the rule, so Fabric provides env.warn_only, a Boolean setting. It defaults to False, meaning an error condition will result in the program aborting immediately. However, if env.warn_only is set to True at the time of failure – with, say, the settings context manager – Fabric will emit a warning message but continue executing.

Looks like you can exercise fine-grained control over where errors are ignored by using the settings context manager, something like so:

from fabric.api import settings

sudo('mkdir tmp') # can't fail
with settings(warn_only=True):
    sudo('touch tmp/test') # can fail
sudo('rm tmp') # can't fail

As of Fabric 1.5, there is a ContextManager that makes this easier:

from fabric.api import sudo, warn_only

with warn_only():
    sudo('mkdir foo')

Update: I re-confirmed that this works in ipython using the following code.

from fabric.api import local, warn_only

#aborted with SystemExit after 'bad command'
local('bad command'); local('bad command 2')

#executes both commands, printing errors for each
with warn_only():
    local('bad command'); local('bad command 2')

You can also set the entire script's warn_only setting to be true with

def local():
    env.warn_only = True

You should set the abort_exception environment variable and catch the exception.

For example:

from fabric.api        import env
from fabric.operations import sudo

class FabricException(Exception):
    pass

env.abort_exception = FabricException
# ... set up the rest of the environment...

try:
    sudo('reboot')
except FabricException:
    pass  # This is expected, we can continue.

You can also set it in a with block. See the documentation here.

In Fabric 2.x you can just use invoke's run with the warn=True argument. Anyway, invoke is a dependency of Fabric 2.x:

from invoke import run
run('bad command', warn=True)

From within a task:

from invoke import task

@task
def my_task(c):
    c.run('bad command', warn=True)

In Fabric 1.3.2 at least, you can recover the exception by catching the SystemExit exception. That's helpful if you have more than one command to run in a batch (like a deploy) and want to cleanup if one of them fails.

In my case, on Fabric >= 1.4 this answer was the correct one.

You can skip bad hosts by adding this:

env.skip_bad_hosts = True

Or passing the --skip-bad-hosts flag/