How can I run Tensorboard on a remote server?
TensorflowMachine LearningData VisualizationRemote AccessTensorboardTensorflow Problem Overview
I'm new to Tensorflow and would greatly benefit from some visualizations of what I'm doing. I understand that Tensorboard is a useful visualization tool, but how do I run it on my remote Ubuntu machine?
Tensorflow Solutions
Solution 1 - Tensorflow
Here is what I do to avoid the issues of making the remote server accept your local external IP:
- when I ssh into the machine, I use the option
-L
to transfer the port6006
of the remote server into the port16006
of my machine (for instance):
ssh -L 16006:127.0.0.1:6006 olivier@my_server_ip
What it does is that everything on the port 6006
of the server (in 127.0.0.1:6006
) will be forwarded to my machine on the port 16006
.
- You can then launch tensorboard on the remote machine using a standard
tensorboard --logdir log
with the default6006
port - On your local machine, go to http://127.0.0.1:16006 and enjoy your remote TensorBoard.
Solution 2 - Tensorflow
You can port-forward with another ssh
command that need not be tied to how you are connecting to the server (as an alternative to the other answer). Thus, the ordering of the below steps is arbitrary.
-
from your local machine, run
ssh -N -f -L localhost:16006:localhost:6006 <user@remote>
-
on the remote machine, run:
tensorboard --logdir <path> --port 6006
-
Then, navigate to (in this example) http://localhost:16006 on your local machine.
(explanation of ssh command:
-N
: no remote commands
-f
: put ssh in the background
-L <machine1>:<portA>:<machine2>:<portB>
:
forward <machine1>:<portA>
(local scope) to <machine2>:<portB>
(remote scope)
Solution 3 - Tensorflow
You don't need to do anything fancy. Just run:
tensorboard --host 0.0.0.0 <other args here>
and connect with your server url and port. The --host 0.0.0.0
tells tensorflow to listen from connections on all IPv4 addresses on the local machine.
Solution 4 - Tensorflow
Another option if you can't get it working for some reason is to simply mount a logdir directory on your filesystem with sshfs:
sshfs user@host:/home/user/project/summary_logs ~/summary_logs
and then run Tensorboard locally.
Solution 5 - Tensorflow
--bind_all
option is useful.
$ tensorboard --logdir runs --bind_all
The port will be automatically selected from 6006 incrementally.(6006, 6007, 6008... )
Solution 6 - Tensorflow
You can directly run the following command on terminal of your remote server to run tensorboard:
tensorboard --logdir {tf_log directory path} --host "0.0.0.0" --port 6006
Or you can also start the tensorboard within your ipython notebook:
%load_ext tensorboard
%tensorboard --logdir {tf_log directory path} --host "0.0.0.0" --port 6006
Solution 7 - Tensorflow
- Find your local external IP by googling
"whats my ip"
or entering this command:wget http://ipinfo.io/ip -qO -
- Determine your remote external IP. This is probably what comes after your username when ssh-ing into the remote server. You can also
wget http://ipinfo.io/ip -qO -
again from there too. - Secure your remote server traffic to just accept your local external IP address
- Run Tensorboard. Note the port it defaults to:
6006
- Enter your remote external IP address into your browser, followed by the port:
123.123.12.32:6006
If your remote server is open to traffic from your local IP address, you should be able to see your remote Tensorboard.
Warning: if all internet traffic can access your system (if you haven't specified a single IP address that can access it), anyone may be able to view your TensorBoard results and runaway with creating SkyNet themselves.
Solution 8 - Tensorflow
You have to create a ssh connection using port forwarding:
ssh -L 16006:127.0.0.1:6006 user@host
Then you run the tensorboard
command:
tensorboard --logdir=/path/to/logs
Then you can easily access the tensorboard
in your browser under:
localhost:16006/
Solution 9 - Tensorflow
This is not a proper answer but a troubleshooter, hopefully helps other less seasoned networkers like me.
In my case (firefox+ubuntu16) the browser was connecting, but showing a blank page (with the tensorboard logo on the tab), and no log activity at all was shown. I still don't know what could be the reason for that (didn't look much into it but if anybody knows please let know!), but I solved it switching to ubuntu's default browser. Here the exact steps, pretty much the same as in @Olivier Moindrot's answer:
- On the server, start tensorboard:
tensorboard --logdir=. --host=localhost --port=6006
- On the client, open the ssh tunnel
ssh -p 23 <USER>@<SERVER> -N -f -L localhost:16006:localhost:6006
- Open ubuntu's
Browser
and visitlocalhost:16006
. The tensorboard page should load without much delay.
To check that the SSH tunnel is effectively working, a simple echo server like this python script can help:
- Put the script into an
<ECHO>.py
file in the server and run it withpython <ECHO>.py
. Now the server will have the echo script listening on 0.0.0.0:5555. - On the client, open the ssh tunnel
ssh -p <SSH_PORT> <USER>@<SERVER> -N -f -L localhost:12345:localhost:5555
- On the client, in the same terminal used to open the tunnel (step 2.), issuing
telnet localhost 12345
will connect to the echo script running in the server. Typinghello
and pressing enter should printhello
back. If that is the case, your SSH tunnel is working. This was my case, and lead me to the conclusion that the problem involved the browser. Trying to connect from a different terminal caused the terminal to freeze.
As I said, hope it helps!
Cheers,
Andres
Solution 10 - Tensorflow
Another approach is to use a reverse proxy, which allows you to view Tensorboard from any internet connected device without SSHing. This approach can make it far easier / tractable to view Tensorboard on mobile devices, for example.
Steps:
-
Download reverse proxy Ngrok on your remote machine hosting Tensorboard. See https://ngrok.com/download for instructions (~5 minute setup).
-
Run
ngrok http 6006
(assuming you're hosting Tensorboard on port 6006) -
Save the URL that ngrok outputs:
- Enter that into any browser to view TensorBoard:
Special thanks to Sam Kirkiles
Solution 11 - Tensorflow
For anyone who must use the ssh keys (for a corporate server).
Just add -i /.ssh/id_rsa
at the end.
$ ssh -N -f -L localhost:8211:localhost:6007 myname@servername -i /.ssh/id_rsa
Solution 12 - Tensorflow
While running the tensorboard give one more option --host=ip of your system and then you can access it from other system using http://ip of your host system:6006