How to loop through dates using Bash?
BashLoopsDateBash Problem Overview
I have such bash script:
array=( '2015-01-01', '2015-01-02' )
for i in "${array[@]}"
do
python /home/user/executeJobs.py {i} &> /home/user/${i}.log
done
Now I want to loop through a range of dates, e.g. 2015-01-01 until 2015-01-31.
How to achieve in Bash?
Update:
Nice-to-have: No job should be started before a previous run has completed. In this case, when executeJobs.py is completed bash prompt $ will return.
e.g. could I incorporate wait%1 in my loop?
Bash Solutions
Solution 1 - Bash
Using GNU date:
d=2015-01-01
while [ "$d" != 2015-02-20 ]; do
echo $d
d=$(date -I -d "$d + 1 day")
# mac option for d decl (the +1d is equivalent to + 1 day)
# d=$(date -j -v +1d -f "%Y-%m-%d" "2020-12-12" +%Y-%m-%d)
done
Note that because this uses string comparison, it requires full ISO 8601 notation of the edge dates (do not remove leading zeros). To check for valid input data and coerce it to a valid form if possible, you can use date as well:
# slightly malformed input data
input_start=2015-1-1
input_end=2015-2-23
# After this, startdate and enddate will be valid ISO 8601 dates,
# or the script will have aborted when it encountered unparseable data
# such as input_end=abcd
startdate=$(date -I -d "$input_start") || exit -1
enddate=$(date -I -d "$input_end") || exit -1
d="$startdate"
while [ "$d" != "$enddate" ]; do
echo $d
d=$(date -I -d "$d + 1 day")
done
One final addition: To check that $startdate is before $enddate, if you only expect dates between the years 1000 and 9999, you can simply use string comparison like this:
while [[ "$d" < "$enddate" ]]; do
To be on the very safe side beyond the year 10000, when lexicographical comparison breaks down, use
while [ "$(date -d "$d" +%Y%m%d)" -lt "$(date -d "$enddate" +%Y%m%d)" ]; do
The expression $(date -d "$d" +%Y%m%d) converts $d to a numerical form, i.e., 2015-02-23 becomes 20150223, and the idea is that dates in this form can be compared numerically.
Solution 2 - Bash
for i in 2015-01-{01..31} …
More:
for i in 2015-02-{01..28} 2015-{04,06,09,11}-{01..30} 2015-{01,03,05,07,08,10,12}-{01..31} …
Proof:
$ echo 2015-02-{01..28} 2015-{04,06,09,11}-{01..30} 2015-{01,03,05,07,08,10,12}-{01..31} | wc -w
365
Compact/nested:
$ echo 2015-{02-{01..28},{04,06,09,11}-{01..30},{01,03,05,07,08,10,12}-{01..31}} | wc -w
365
Ordered, if it matters:
$ x=( $(printf '%s\n' 2015-{02-{01..28},{04,06,09,11}-{01..30},{01,03,05,07,08,10,12}-{01..31}} | sort) )
$ echo "${#x[@]}"
365
Since it's unordered, you can just tack leap years on:
$ echo {2015..2030}-{02-{01..28},{04,06,09,11}-{01..30},{01,03,05,07,08,10,12}-{01..31}} {2016..2028..4}-02-29 | wc -w
5844
Solution 3 - Bash
start='2019-01-01'
end='2019-02-01'
start=$(date -d $start +%Y%m%d)
end=$(date -d $end +%Y%m%d)
while [[ $start -le $end ]]
do
echo $(date -d $start +%Y-%m-%d)
start=$(date -d"$start + 1 day" +"%Y%m%d")
done
Solution 4 - Bash
The previous solution by @Gilli is pretty clever, because it plays with the fact, that you can simple format two dates make them look like integers. Then you can use -le / less-equal - which usually works with numeric data only.
Problem is, that this binds you to the date format YMD, like 20210201. If you need something different, like 2021-02-01 (that's what OP implicated as a requirement), the script will not work:
start='2021-02-01'
end='2021-02-05'
start=$(date -d $start +%Y-%m-%d)
end=$(date -d $end +%-Y%m-%d)
while [[ $start -le $end ]]
do
echo $start
start=$(date -d"$start + 1 day" +"%Y-%m-%d")
done
The output will look like this:
2021-02-01
2021-02-02
2021-02-03
2021-02-04
2021-02-05
2021-02-06
2021-02-07
./loop.sh: line 16: [[: 2021-02-08: value too great for base (error token is "08")
To fix that and use this loop for custom date formats, you need to work with one additional variable, let's call it "d_start":
d_start='2021-02-01'
end='2021-02-05'
start=$(date -d $d_start +%Y%m%d)
end=$(date -d $end +%Y%m%d)
while [[ $start -le $end ]]
do
echo $d_start
start=$(date -d"$start + 1 day" +"%Y%m%d")
d_start=$(date -d"$d_start + 1 day" +"%Y-%m-%d")
done
This will lead to this output:
2021-02-01
2021-02-02
2021-02-03
2021-02-04
2021-02-05
Solution 5 - Bash
I needed to loop through dates on AIX, BSDs, Linux, OS X and Solaris. The date command is one of the least portable and most miserable commands to use across platforms I have encountered. I found it easier to write a my_date command that just worked everywhere.
The C program below takes a starting date, and adds or subtracts days from it. If no date is supplied, it adds or subtracts days from the current date.
The my_date command allows you to perform the following everywhere:
start="2015-01-01"
stop="2015-01-31"
echo "Iterating dates from ${start} to ${stop}."
while [[ "${start}" != "${stop}" ]]
do
python /home/user/executeJobs.py {i} &> "/home/user/${start}.log"
start=$(my_date -s "${start}" -n +1)
done
And the C code:
#include <stdio.h>
#include <stdlib.h>
#include <stdbool.h>
#include <time.h>
int show_help();
int main(int argc, char* argv[])
{
int eol = 0, help = 0, n_days = 0;
int ret = EXIT_FAILURE;
time_t startDate = time(NULL);
const time_t ONE_DAY = 24 * 60 * 60;
for (int i=0; i<argc; i++)
{
if (strcmp(argv[i], "-l") == 0)
{
eol = 1;
}
else if (strcmp(argv[i], "-n") == 0)
{
if (++i == argc)
{
show_help();
ret = EXIT_FAILURE;
goto finish;
}
n_days = strtoll(argv[i], NULL, 0);
}
else if (strcmp(argv[i], "-s") == 0)
{
if (++i == argc)
{
show_help();
ret = EXIT_FAILURE;
goto finish;
}
struct tm dateTime;
memset (&dateTime, 0x00, sizeof(dateTime));
const char* start = argv[i];
const char* end = strptime (start, "%Y-%m-%d", &dateTime);
/* Ensure all characters are consumed */
if (end - start != 10)
{
show_help();
ret = EXIT_FAILURE;
goto finish;
}
startDate = mktime (&dateTime);
}
}
if (help == 1)
{
show_help();
ret = EXIT_SUCCESS;
goto finish;
}
char buff[32];
const time_t next = startDate + ONE_DAY * n_days;
strftime(buff, sizeof(buff), "%Y-%m-%d", localtime(&next));
/* Paydirt */
if (eol)
fprintf(stdout, "%s\n", buff);
else
fprintf(stdout, "%s", buff);
ret = EXIT_SUCCESS;
finish:
return ret;
}
int show_help()
{
fprintf(stderr, "Usage:\n");
fprintf(stderr, " my_date [-s date] [-n [+|-]days] [-l]\n");
fprintf(stderr, " -s date: optional, starting date in YYYY-MM-DD format\n");
fprintf(stderr, " -n days: optional, number of days to add or subtract\n");
fprintf(stderr, " -l: optional, add new-line to output\n");
fprintf(stderr, "\n");
fprintf(stderr, " If no options are supplied, then today is printed.\n");
fprintf(stderr, "\n");
return 0;
}
Solution 6 - Bash
I had the same issue and I tried some of the above answers, maybe they are ok, but none of those answers fixed on what I was trying to do, using macOS.
I was trying to iterate over dates in the past, and the following is what worked for me:
#!/bin/bash
# Get the machine date
newDate=$(date '+%m-%d-%y')
# Set a counter variable
counter=1
# Increase the counter to get back in time
while [ "$newDate" != 06-01-18 ]; do
echo $newDate
newDate=$(date -v -${counter}d '+%m-%d-%y')
counter=$((counter + 1))
done
Hope it helps.
Solution 7 - Bash
Bash is best written by leveraging pipes(|). This should result in memory efficient and concurrent(faster) processing. I would write the following:
seq 0 100 | xargs printf "20 Aug 2020 - %sdays\n" \
| xargs -d '\n' -l date -d
The following will print the date of 20 aug 2020 and print the dates of the 100 days before it.
This oneliner can be made into a utility.
#!/usr/bin/env bash
# date-range template <template>
template="${1:--%sdays}"
export LANG;
xargs printf "$template\n" | xargs -d '\n' -l date -d
By default we choose to iterate into the past 1 day at a time.
$ seq 10 | date-range
Mon Mar 2 17:42:43 CET 2020
Sun Mar 1 17:42:43 CET 2020
Sat Feb 29 17:42:43 CET 2020
Fri Feb 28 17:42:43 CET 2020
Thu Feb 27 17:42:43 CET 2020
Wed Feb 26 17:42:43 CET 2020
Tue Feb 25 17:42:43 CET 2020
Mon Feb 24 17:42:43 CET 2020
Sun Feb 23 17:42:43 CET 2020
Sat Feb 22 17:42:43 CET 2020
Let's say we want to generate dates up to a certain date. We don't know yet how many iterations we need to get there. Let's say Tom was born 1 Jan 2001. We want to generate each date till a certain one. We can achieve this by using sed.
seq 0 $((2**63-1)) | date-range | sed '/.. Jan 2001 /q'
> The $((2**63-1)) trick is used to create a big integer.
Once sed exits it will also exit the date-range utility.
One can also iterate using a 3 month interval:
$ seq 0 3 12 | date-range '+%smonths'
Tue Mar 3 18:17:17 CET 2020
Wed Jun 3 19:17:17 CEST 2020
Thu Sep 3 19:17:17 CEST 2020
Thu Dec 3 18:17:17 CET 2020
Wed Mar 3 18:17:17 CET 2021
Solution 8 - Bash
If you're stuck with busybox date, I've found working with timestamps to be the most reliable approach:
STARTDATE="2019-12-30"
ENDDATE="2020-01-04"
start=$(date -d $STARTDATE +%s)
end=$(date -d $ENDDATE +%s)
d="$start"
while [[ $d -le $end ]]
do
date -d @$d +%Y-%m-%d
d=$(( $d + 86400 ))
done
This will output:
2019-12-30
2019-12-31
2020-01-01
2020-01-02
2020-01-03
2020-01-04
Unix timestamps don't include leap seconds, so 1 day equals always exactly 86400 seconds.
Solution 9 - Bash
If one wants to loop from input date to any range below can be used, also it will print output in format of yyyyMMdd...
#!/bin/bash
in=2018-01-15
while [ "$in" != 2018-01-25 ]; do
in=$(date -I -d "$in + 1 day")
x=$(date -d "$in" +%Y%m%d)
echo $x
done
Solution 10 - Bash
This might also help. Based on Gilli answer, but a different solution of the issue with an integer conversion.
Basically, while verifying the input, LoopEachDay stores the "end" date in seconds and compares with it firstly converting the current day into seconds(date -d "$dateIteration" '+%s'), too.
#/bin/bash
RegexVerify()
{
regex="$1";
shift;
if [[ "$@" =~ $regex ]];
then
return 0;
fi
return 1;
}
VerifyDateISO8601()
{
if RegexVerify '^[0-9]{4}-(0?[1-9]|10|11|12)-(0?[1-9]|[12][0-9]|3[01])$' "$1";
then
return 0;
fi
return 1;
}
# Iterate each day
#
# * The *first* argument is an ISO8601 start date.
# * The *second* argument is an ISO8601 end date or an empty string which assumes
# the current date.
LoopEachDay()
{
if ! VerifyDateISO8601 "$1";
then
return 1;
fi
if ! VerifyDateISO8601 "$2" && [ "$2" != '' ];
then
return 2;
fi
dateIteration="$(date -d "$1" '+%Y-%m-%d')";
dateIterationEndSeconds="$(date -d "$2" '+%s')";
while (("$(date -d "$dateIteration" '+%s')" <= dateIterationEndSeconds))
do
printf $'%s\n' "$dateIteration"; # A work with "$dateIteration"
dateIteration="$(date -d "$dateIteration + 1 day" '+%Y-%m-%d')";
done
}
LoopEachDay '2021-13-01' '';
printf $'Exit code: %s\n\n' "$?";
# Exit code: 1
LoopEachDay '2021-04-01' '';
# 2021-04-01
# 2021-04-02
# 2021-04-03
# 2021-04-04
# 2021-04-05
# 2021-04-06
# 2021-04-07
# 2021-04-08
printf $'\n';
LoopEachDay '2021-04-03' '2021-04-06';
# 2021-04-03
# 2021-04-04
# 2021-04-05
# 2021-04-06