Waiting for background processes to finish before exiting script

How do I make sure that all my background processes have finished execution before I exit my script (TCL/Bash).

I was thinking of writing all my background process pids to a pidfile. And then at the end pgrep the pidfile to see if any processes are still running before I exit.

Is there some simpler way to do this? And is there a TCL specific way to do this?

If you want to wait for jobs to finish, use wait. This will make the shell wait until all background jobs complete. However, if any of your jobs daemonize themselves, they are no longer children of the shell and wait will have no effect (as far as the shell is concerned, the child is already done. Indeed, when a process daemonizes itself, it does so by terminating and spawning a new process that inherits its role).

#!/bin/sh
{ sleep 5; echo waking up after 5 seconds; } &
{ sleep 1; echo waking up after 1 second; } &
wait
echo all jobs are done!

You can use kill -0 for checking whether a particular pid is running or not.

Assuming, you have list of pid numbers in a file called pid in pwd

while true;
do 
    if [ -s pid ] ; then
        for pid in `cat pid`
        do  
            echo "Checking the $pid"
            kill -0 "$pid" 2>/dev/null || sed -i "/^$pid$/d" pid
        done
    else
        echo "All your process completed" ## Do what you want here... here all your pids are in finished stated
        break
    fi
done

WARNING: Long script ahead.

A while ago, I faced a similar problem: from a Tcl script, launch a number of processes, then wait for all of them to finish. Here is a demo script I wrote to solve this problem.

main.tcl

#!/usr/bin/env tclsh

# Launches many processes and wait for them to finish.
# This script will works on systems that has the ps command such as
# BSD, Linux, and OS X

package require Tclx; # For process-management utilities

proc updatePidList {stat} {
    global pidList
    global allFinished

    # Parse the process ID of the just-finished process
    lassign $stat processId howProcessEnded exitCode

    # Remove this process ID from the list of process IDs
    set pidList [lindex [intersect3 $pidList $processId] 0]
    set processCount [llength $pidList]

    # Occasionally, a child process quits but the signal was lost. This
    # block of code will go through the list of remaining process IDs
    # and remove those that has finished
    set updatedPidList {}
    foreach pid $pidList {
        if {![catch {exec ps $pid} errmsg]} {
            lappend updatedPidList $pid
        }
    }

    set pidList $updatedPidList

    # Show the remaining processes
    if {$processCount > 0} {
        puts "Waiting for [llength $pidList] processes"
    } else {
        set allFinished 1
        puts "All finished"
    }
}

# A signal handler that gets called when a child process finished.
# This handler needs to exit quickly, so it delegates the real works to
# the proc updatePidList
proc childTerminated {} {
    # Restart the handler
    signal -restart trap SIGCHLD childTerminated

    # Update the list of process IDs
    while {![catch {wait -nohang} stat] && $stat ne {}} {
        after idle [list updatePidList $stat]
    }
}

#
# Main starts here
#

puts "Main begins"
set NUMBER_OF_PROCESSES_TO_LAUNCH 10
set pidList {}
set allFinished 0

# When a child process exits, call proc childTerminated
signal -restart trap SIGCHLD childTerminated

# Spawn many processes
for {set i 0} {$i < $NUMBER_OF_PROCESSES_TO_LAUNCH} {incr i} {
    set childId [exec tclsh child.tcl $i &]
    puts "child #$i, pid=$childId"
    lappend pidList $childId
    after 1000
}

# Do some processing
puts "list of processes: $pidList"
puts "Waiting for child processes to finish"
# Do some more processing if required

# After all done, wait for all to finish before exiting
vwait allFinished

puts "Main ends"

child.tcl

#!/usr/bin/env tclsh
# child script: simulate some lengthy operations

proc randomInteger {min max} {
    return [expr int(rand() * ($max - $min + 1) * 1000 + $min)]
}

set duration [randomInteger 10 30]
puts "  child #$argv runs for $duration miliseconds"
after $duration
puts "  child #$argv ends"

Sample output for running main.tcl

Main begins
child #0, pid=64525
  child #0 runs for 17466 miliseconds
child #1, pid=64526
  child #1 runs for 14181 miliseconds
child #2, pid=64527
  child #2 runs for 10856 miliseconds
child #3, pid=64528
  child #3 runs for 7464 miliseconds
child #4, pid=64529
  child #4 runs for 4034 miliseconds
child #5, pid=64531
  child #5 runs for 1068 miliseconds
child #6, pid=64532
  child #6 runs for 18571 miliseconds
  child #5 ends
child #7, pid=64534
  child #7 runs for 15374 miliseconds
child #8, pid=64535
  child #8 runs for 11996 miliseconds
  child #4 ends
child #9, pid=64536
  child #9 runs for 8694 miliseconds
list of processes: 64525 64526 64527 64528 64529 64531 64532 64534 64535 64536
Waiting for child processes to finish
Waiting for 8 processes
Waiting for 8 processes
  child #3 ends
Waiting for 7 processes
  child #2 ends
Waiting for 6 processes
  child #1 ends
Waiting for 5 processes
  child #0 ends
Waiting for 4 processes
  child #9 ends
Waiting for 3 processes
  child #8 ends
Waiting for 2 processes
  child #7 ends
Waiting for 1 processes
  child #6 ends
All finished
Main ends

GNU parallel and xargs

These two tools that can make scripts simpler, and also control the maximum number of threads (thread pool). E.g.:

seq 10 | xargs -P4 -I'{}' echo '{}'

or:

seq 10 | parallel -j4  echo '{}'

See also: https://stackoverflow.com/questions/6441509/how-to-write-a-process-pool-bash-shell

Even if you do not have the pid, you can trigger 'wait;' after triggering all background processes. For. eg. in commandfile.sh-

bteq < input_file1.sql > output_file1.sql &
bteq < input_file2.sql > output_file2.sql &
bteq < input_file3.sql > output_file3.sql &
wait

Then when this is triggered, as -

subprocess.call(['sh', 'commandfile.sh'])
print('all background processes done.')

This will be printed only after all the background processes are done.