ssh script returns 255 error

In my code I have the following to run a remote script.

ssh root@host.domain.com "sh /home/user/backup_mysql.sh"

For some reason it keeps 255'ing on me. Any ideas?

I can SSH into the box just fine (passless keys setup)

REMOTE SCRIPT:

MUSER='root' 
MPASS='123123'
MHOST="127.0.0.1"
VERBOSE=0

### Set bins path ###
GZIP=/bin/gzip
MYSQL=/usr/bin/mysql
MYSQLDUMP=/usr/bin/mysqldump
RM=/bin/rm
MKDIR=/bin/mkdir
MYSQLADMIN=/usr/bin/mysqladmin
GREP=/bin/grep

### Setup dump directory ###
BAKRSNROOT=/.snapshots/tmp

#####################################
### ----[ No Editing below ]------###
#####################################
### Default time format ###
TIME_FORMAT='%H_%M_%S%P'

### Make a backup ###
backup_mysql_rsnapshot(){
        local DBS="$($MYSQL -u $MUSER -h $MHOST -p$MPASS -Bse 'show databases')"
        local db="";
        [ ! -d $BAKRSNROOT ] && ${MKDIR} -p $BAKRSNROOT
        ${RM} -f $BAKRSNROOT/* >/dev/null 2>&1
#       [ $VERBOSE -eq 1 ] && echo "*** Dumping MySQL Database ***"
#       [ $VERBOSE -eq 1 ] && echo -n "Database> "
        for db in $DBS
        do
                local tTime=$(date +"${TIME_FORMAT}")
                local FILE="${BAKRSNROOT}/${db}.${tTime}.gz"
#               [ $VERBOSE -eq 1 ] && echo -n "$db.."
                ${MYSQLDUMP} --single-transaction -u ${MUSER} -h ${MHOST} -p${MPASS} $db | ${GZIP} -9 > $FILE
        done
#               [ $VERBOSE -eq 1 ] && echo ""
#               [ $VERBOSE -eq 1 ] && echo "*** Backup done [ files wrote to $BAKRSNROOT] ***"
}

### Die on demand with message ###
die(){
        echo "$@"
        exit 999
}

### Make sure bins exists.. else die
verify_bins(){
        [ ! -x $GZIP ] && die "File $GZIP does not exists. Make sure correct path is set in $0."
        [ ! -x $MYSQL ] && die "File $MYSQL does not exists. Make sure correct path is set in $0."
        [ ! -x $MYSQLDUMP ] && die "File $MYSQLDUMP does not exists. Make sure correct path is set in $0."
        [ ! -x $RM ] && die "File $RM does not exists. Make sure correct path is set in $0."
        [ ! -x $MKDIR ] && die "File $MKDIR does not exists. Make sure correct path is set in $0."
        [ ! -x $MYSQLADMIN ] && die "File $MYSQLADMIN does not exists. Make sure correct path is set in $0."
        [ ! -x $GREP ] && die "File $GREP does not exists. Make sure correct path is set in $0."
}

### Make sure we can connect to server ... else die
verify_mysql_connection(){
        $MYSQLADMIN  -u $MUSER -h $MHOST -p$MPASS ping | $GREP 'alive'>/dev/null
        [ $? -eq 0 ] || die "Error: Cannot connect to MySQL Server. Make sure username and password are set correctly in $0"
}

### main ####
verify_bins
verify_mysql_connection
backup_mysql_rsnapshot

This is usually happens when the remote is down/unavailable; or the remote machine doesn't have ssh installed; or a firewall doesn't allow a connection to be established to the remote host.

ssh returns 255 when an error occurred or 255 is returned by the remote script:

 EXIT STATUS

     ssh exits with the exit status of the remote command or
     with 255 if an error occurred.

Usually you would an error message something similar to:

ssh: connect to host host.domain.com port 22: No route to host

Or

ssh: connect to host HOSTNAME port 22: Connection refused

Check-list:

• What happens if you run the ssh command directly from the command line?

• Are you able to ping that machine?

• Does the remote has ssh installed?

• If installed, then is the ssh service running?

This error will also occur when using pdsh to hosts which are not contained in your "known_hosts" file.

I was able to correct this by SSH'ing into each host manually and accepting the question "Do you want to add this to known hosts".

If there's a problem with authentication or connection, such as not being able to read a password from the terminal, ssh will exit with 255 without being able to run your actual script. Verify to make sure you can run 'true' instead, to see if the ssh connection is established successfully.

I was stumped by this. Once I got passed the 255 problem... I ended up with a mysterious error code 1. This is the foo to get that resolved:

 pssh -x '-tt' -h HOSTFILELIST -P "sudo yum -y install glibc"

-P means write the output out as you go and is optional. But the -x '-tt' trick is what forces a psuedo tty to be allocated.

You can get a clue what the error code 1 means this if you try:

ssh AHOST "sudo yum -y install glibc"

You may see:

[slc@bastion-ci ~]$ ssh MYHOST "sudo yum -y install glibc"
sudo: sorry, you must have a tty to run sudo
[slc@bastion-ci ~]$ echo $?
1

Notice the return code for this is 1, which is what pssh is reporting to you.

I found this -x -tt trick here. Also note that turning on verbose mode (pssh --verbose) for these cases does nothing to help you.

Isn't the problem in the lines:

### Die on demand with message ###
die(){
  echo "$@"
  exit 999
}

Correct me if I'm wrong but I believe exit 999 is out of range for an exit code and results in a exit status of 255.

It can very much be an ssh-agent issue. Check whether there is an ssh-agent PID currently running with eval "$(ssh-agent -s)"

Check whether your identity is added with ssh-add -l and if not, add it with ssh-add <pathToYourRSAKey>.

Then try again your ssh command (or any other command that spawns ssh daemons, like autossh for example) that returned 255.

If above didn't help: check if locale is valid on client and server:
https://www.linuxbabe.com/linux-server/fix-ssh-locale-environment-variable-error
https://stackoverflow.com/questions/29609371/how-do-not-pass-locale-through-ssh

### Die on demand with message ###
die(){
        echo "$@"
        exit 999
}

I don't have the rep to comment on Alex's answer but the exit 999 line returns code 231 on my WSL Ubuntu 20.04.4 box. Not quite sure why that is returned but I understand that it's out of range.