How to set target hosts in Fabric file
PythonHostFabricPython Problem Overview
I want to use Fabric to deploy my web app code to development, staging and production servers. My fabfile:
def deploy_2_dev():
deploy('dev')
def deploy_2_staging():
deploy('staging')
def deploy_2_prod():
deploy('prod')
def deploy(server):
print 'env.hosts:', env.hosts
env.hosts = [server]
print 'env.hosts:', env.hosts
Sample output:
host:folder user$ fab deploy_2_dev
env.hosts: []
env.hosts: ['dev']
No hosts found. Please specify (single) host string for connection:
When I create a set_hosts()
task as shown in the Fabric docs, env.hosts is set properly. However, this is not a viable option, neither is a decorator. Passing hosts on the command line would ultimately result in some kind of shell script that calls the fabfile, I would prefer having one single tool do the job properly.
It says in the Fabric docs that 'env.hosts is simply a Python list object'. From my observations, this is simply not true.
Can anyone explain what is going on here ? How can I set the host to deploy to ?
Python Solutions
Solution 1 - Python
I do this by declaring an actual function for each environment. For example:
def test():
env.user = 'testuser'
env.hosts = ['test.server.com']
def prod():
env.user = 'produser'
env.hosts = ['prod.server.com']
def deploy():
...
Using the above functions, I would type the following to deploy to my test environment:
fab test deploy
...and the following to deploy to production:
fab prod deploy
The nice thing about doing it this way is that the test
and prod
functions can be used before any fab function, not just deploy. It is incredibly useful.
Solution 2 - Python
Use roledefs
from fabric.api import env, run
env.roledefs = {
'test': ['localhost'],
'dev': ['[email protected]'],
'staging': ['[email protected]'],
'production': ['[email protected]']
}
def deploy():
run('echo test')
Choose role with -R:
$ fab -R test deploy
[localhost] Executing task 'deploy'
...
Solution 3 - Python
Here's a simpler version of serverhorror's answer:
from fabric.api import settings
def mystuff():
with settings(host_string='192.0.2.78'):
run("hostname -f")
Solution 4 - Python
Was stuck on this myself, but finally figured it out. You simply can't set the env.hosts configuration from within a task. Each task is executed N times, once for each Host specified, so the setting is fundamentally outside of task scope.
Looking at your code above, you could simply do this:
@hosts('dev')
def deploy_dev():
deploy()
@hosts('staging')
def deploy_staging():
deploy()
def deploy():
# do stuff...
Which seems like it would do what you're intending.
Or you can write some custom code in the global scope that parses the arguments manually, and sets env.hosts before your task function is defined. For a few reasons, that's actually how I've set mine up.
Solution 5 - Python
Since fab 1.5 this is a documented way to dynamically set hosts.
http://docs.fabfile.org/en/1.7/usage/execution.html#dynamic-hosts
Quote from the doc below.
> Using execute with dynamically-set host lists > > A common intermediate-to-advanced use case for Fabric is to > parameterize lookup of one’s target host list at runtime (when use of > Roles does not suffice). execute can make this extremely simple, like > so:
from fabric.api import run, execute, task
# For example, code talking to an HTTP API, or a database, or ...
from mylib import external_datastore
# This is the actual algorithm involved. It does not care about host
# lists at all.
def do_work():
run("something interesting on a host")
# This is the user-facing task invoked on the command line.
@task
def deploy(lookup_param):
# This is the magic you don't get with @hosts or @roles.
# Even lazy-loading roles require you to declare available roles
# beforehand. Here, the sky is the limit.
host_list = external_datastore.query(lookup_param)
# Put this dynamically generated host list together with the work to be
# done.
execute(do_work, hosts=host_list)
Solution 6 - Python
Contrary to some other answers, it is possible to modify the env
environment variables within a task. However, this env
will only be used for subsequent tasks executed using the fabric.tasks.execute
function.
from fabric.api import task, roles, run, env
from fabric.tasks import execute
# Not a task, plain old Python to dynamically retrieve list of hosts
def get_stressors():
hosts = []
# logic ...
return hosts
@task
def stress_test():
# 1) Dynamically generate hosts/roles
stressors = get_stressors()
env.roledefs['stressors'] = map(lambda x: x.public_ip, stressors)
# 2) Wrap sub-tasks you want to execute on new env in execute(...)
execute(stress)
# 3) Note that sub-tasks not nested in execute(...) will use original env
clean_up()
@roles('stressors')
def stress():
# this function will see any changes to env, as it was wrapped in execute(..)
run('echo "Running stress test..."')
# ...
@task
def clean_up():
# this task will NOT see any dynamic changes to env
Without wrapping sub-tasks in execute(...)
, your module-level env
settings or whatever is passed from the fab
CLI will be used.
Solution 7 - Python
You need to set host_string
an example would be:
from fabric.context_managers import settings as _settings
def _get_hardware_node(virtualized):
return "localhost"
def mystuff(virtualized):
real_host = _get_hardware_node(virtualized)
with _settings(
host_string=real_host):
run("echo I run on the host %s :: `hostname -f`" % (real_host, ))
Solution 8 - Python
To explain why it's even an issue. The command fab is leveraging fabric the library to run the tasks on the host lists. If you try and change the host list inside a task, you're esentially attempting to change a list while iterating over it. Or in the case where you have no hosts defined, loop over an empty list where the code where you set the list to loop over is never executed.
The use of env.host_string is a work around for this behavior only in that it's specifying directly to the functions what hosts to connect with. This causes some issues in that you'll be remaking the execution loop if you want to have a number of hosts to execute on.
The simplest way the people make the ability to set hosts at run time, is to keep the env populatiing as a distinct task, that sets up all the host strings, users, etc. Then they run the deploy task. It looks like this:
fab production deploy
or
fab staging deploy
Where staging and production are like the tasks you have given, but they do not call the next task themselves. The reason it has to work like this, is that the task has to finish, and break out of the loop (of hosts, in the env case None, but it's a loop of one at that point), and then have the loop over the hosts (now defined by the preceding task) anew.
Solution 9 - Python
You need to modify env.hosts at the module level, not within a task function. I made the same mistake.
from fabric.api import *
def _get_hosts():
hosts = []
... populate 'hosts' list ...
return hosts
env.hosts = _get_hosts()
def your_task():
... your task ...
Solution 10 - Python
It's very simple. Just initialize the env.host_string variable and all of the following commands will be executed on this host.
from fabric.api import env, run
env.host_string = '[email protected]'
def foo:
run("hostname -f")
Solution 11 - Python
I'm totally new to fabric, but to get fabric to run the same commands on multiple hosts (e.g. to deploy to multiple servers, in one command) you can run:
fab -H staging-server,production-server deploy
where staging-server and production-server are 2 servers you want to run the deploy action against. Here's a simple fabfile.py that will display the OS name. Note that the fabfile.py should be in the same directory as where you run the fab command.
from fabric.api import *
def deploy():
run('uname -s')
This works with fabric 1.8.1 at least.
Solution 12 - Python
So, in order to set the hosts, and have the commands run across all the hosts, you have to start with:
def PROD():
env.hosts = ['10.0.0.1', '10.0.0.2']
def deploy(version='0.0'):
sudo('deploy %s' % version)
Once those are defined, then run the command on the command line:
fab PROD deploy:1.5
What will run the deploy task across all of the servers listed in the PROD function, as it sets the env.hosts before running the task.
Solution 13 - Python
You can assign to env.hoststring
before executing a subtask. Assign to this global variable in a loop if you want to iterate over multiple hosts.
Unfortunately for you and me, fabric is not designed for this use case. Check out the main
function at http://github.com/bitprophet/fabric/blob/master/fabric/main.py to see how it works.
Solution 14 - Python
Here's another "summersault" pattern that enables the fab my_env_1 my_command
usage:
With this pattern, we only have to define environments one time using a dictionary. env_factory
creates functions based on the keynames of ENVS
. I put ENVS
in its own directory and file secrets.config.py
to separate config from the fabric code.
The drawback is that, as written, adding the @task
decorator will break it.
Notes: We use def func(k=k):
instead of def func():
in the factory because of late binding. We get the running module with this solution and patch it to define the function.
secrets.config.py
ENVS = {
'my_env_1': {
'HOSTS': [
'host_1',
'host_2',
],
'MY_OTHER_SETTING': 'value_1',
},
'my_env_2': {
'HOSTS': ['host_3'],
'MY_OTHER_SETTING': 'value_2'
}
}
fabfile.py
import sys
from fabric.api import env
from secrets import config
def _set_env(env_name):
# can easily customize for various use cases
selected_config = config.ENVS[env_name]
for k, v in selected_config.items():
setattr(env, k, v)
def _env_factory(env_dict):
for k in env_dict:
def func(k=k):
_set_env(k)
setattr(sys.modules[__name__], k, func)
_env_factory(config.ENVS)
def my_command():
# do work
Solution 15 - Python
Using roles is currently considered to be the "proper" and "correct" way of doing this and is what you "should" do it.
That said, if you are like most of what you "would like" or "desire" is the ability to perform a "twisted syster" or switching target systems on the fly.
So for entertainment purposes only (!) the following example illustrates what many might consider to a risky, and yet somehow thoroughly satisfying, manoeuvre that goes something like this:
env.remote_hosts = env.hosts = ['10.0.1.6']
env.remote_user = env.user = 'bob'
env.remote_password = env.password = 'password1'
env.remote_host_string = env.host_string
env.local_hosts = ['127.0.0.1']
env.local_user = 'mark'
env.local_password = 'password2'
def perform_sumersault():
env_local_host_string = env.host_string = env.local_user + '@' + env.local_hosts[0]
env.password = env.local_password
run("hostname -f")
env.host_string = env.remote_host_string
env.remote_password = env.password
run("hostname -f")
Then running:
fab perform_sumersault