Retry a Bash command with timeout

Bash Problem Overview

How to retry a bash command until its status is ok or until a timeout is reached?

My best shot (I'm looking for something simpler):

NEXT_WAIT_TIME=0
COMMAND_STATUS=1
until [ $COMMAND_STATUS -eq 0 || $NEXT_WAIT_TIME -eq 4 ]; do
  command
  COMMAND_STATUS=$?
  sleep $NEXT_WAIT_TIME
  let NEXT_WAIT_TIME=NEXT_WAIT_TIME+1
done

Bash Solutions

Solution 1 - Bash

You can simplify things a bit by putting command right in the test and doing increments a bit differently. Otherwise the script looks fine:

NEXT_WAIT_TIME=0
until [ $NEXT_WAIT_TIME -eq 5 ] || command; do
    sleep $(( NEXT_WAIT_TIME++ ))
done
[ $NEXT_WAIT_TIME -lt 5 ]

Solution 2 - Bash

One line and shortest, and maybe the best approach:

timeout 12h bash -c 'until ssh root@mynewvm; do sleep 10; done'

Credited by http://jeromebelleman.gitlab.io/posts/devops/until/

Solution 3 - Bash

retry fuction is from:

http://fahdshariff.blogspot.com/2014/02/retrying-commands-in-shell-scripts.html

#!/bin/bash
 
# Retries a command on failure.
# $1 - the max number of attempts
# $2... - the command to run
retry() {
    local -r -i max_attempts="$1"; shift
    local -r cmd="$@"
    local -i attempt_num=1
 
    until $cmd
    do
        if (( attempt_num == max_attempts ))
        then
            echo "Attempt $attempt_num failed and there are no more attempts left!"
            return 1
        else
            echo "Attempt $attempt_num failed! Trying again in $attempt_num seconds..."
            sleep $(( attempt_num++ ))
        fi
    done
}
 
# example usage:
retry 5 ls -ltr foo

if you want to retry an function in your script, you should do like this:

# example usage:
foo()
{
   #whatever you want do.
}

declare -fxr foo
retry 3 timeout 60 bash -ce 'foo'

Solution 4 - Bash

Put together some tools.

retry: https://github.com/kadwanev/retry

timeout: http://manpages.courier-mta.org/htmlman1/timeout.1.html

Then see the magic

retry timeout 3 ping google.com

PING google.com (173.194.123.97): 56 data bytes
64 bytes from 173.194.123.97: icmp_seq=0 ttl=55 time=13.982 ms
64 bytes from 173.194.123.97: icmp_seq=1 ttl=55 time=44.857 ms
64 bytes from 173.194.123.97: icmp_seq=2 ttl=55 time=64.187 ms
Before retry #1: sleeping 0.3 seconds
PING google.com (173.194.123.103): 56 data bytes
64 bytes from 173.194.123.103: icmp_seq=0 ttl=55 time=56.549 ms
64 bytes from 173.194.123.103: icmp_seq=1 ttl=55 time=60.220 ms
64 bytes from 173.194.123.103: icmp_seq=2 ttl=55 time=8.872 ms
Before retry #2: sleeping 0.6 seconds
PING google.com (173.194.123.103): 56 data bytes
64 bytes from 173.194.123.103: icmp_seq=0 ttl=55 time=25.819 ms
64 bytes from 173.194.123.103: icmp_seq=1 ttl=55 time=16.382 ms
64 bytes from 173.194.123.103: icmp_seq=2 ttl=55 time=3.224 ms
Before retry #3: sleeping 1.2 seconds
PING google.com (173.194.123.103): 56 data bytes
64 bytes from 173.194.123.103: icmp_seq=0 ttl=55 time=58.438 ms
64 bytes from 173.194.123.103: icmp_seq=1 ttl=55 time=94.828 ms
64 bytes from 173.194.123.103: icmp_seq=2 ttl=55 time=61.075 ms
Before retry #4: sleeping 2.4 seconds
PING google.com (173.194.123.103): 56 data bytes
64 bytes from 173.194.123.103: icmp_seq=0 ttl=55 time=43.361 ms
64 bytes from 173.194.123.103: icmp_seq=1 ttl=55 time=32.171 ms
...

Check exit status for ultimate pass/fail.

Solution 5 - Bash

For anyone wanting to actually wait until some time passed, taking into account the time of your command might be significant:

TIMEOUT_SEC=180
start_time="$(date -u +%s)"
while [ condition_or_just_true ]; do
  current_time="$(date -u +%s)"
  elapsed_seconds=$(($current_time-$start_time))
  if [ $elapsed_seconds -gt $TIMEOUT_SEC ]; then
    echo "timeout of $TIMEOUT_SEC sec"
    exit 1
  fi
  echo "another attempt (elapsed $elapsed_seconds sec)"
  some_command_and_maybe_sleep
done

Solution 6 - Bash

I made some tweaks to this answer which let you switch on whether the timeout was reached, or whether the command succeed. Also, in this version there is a retry every second:

ELAPSED=0
started=$(mktemp)
echo "False" > $started
until the_command_here && echo "True" > $started || [ $ELAPSED -eq 30 ]
do
   sleep 1
   (( ELAPSED++ ))
done
                                                                                                                        
if [[ $(cat $started) == "True" ]]                                                                                                                                                                                                                            
then                                                                                                                    
    echo "the command completed after $ELAPSED seconds"                                                                                              
else                                                                                                                    
    echo "timed out after $ELAPSED seconds"                                                                               
    exit 111                                                                                                            
fi

Content Type	Original Author	Original Content on Stackoverflow
Question	Philippe Blayo	View Question on Stackoverflow
Solution 1 - Bash	Grisha Levit	View Answer on Stackoverflow
Solution 2 - Bash	petertc	View Answer on Stackoverflow
Solution 3 - Bash	rhinoceros.xn	View Answer on Stackoverflow
Solution 4 - Bash	nkadwa	View Answer on Stackoverflow
Solution 5 - Bash	Mugen	View Answer on Stackoverflow
Solution 6 - Bash	MatrixManAtYrService	View Answer on Stackoverflow