Iterating through JSON array in Shell script

I have a JSON data as follows in data.json file

[  {"original_name":"pdf_convert","changed_name":"pdf_convert_1"},  {"original_name":"video_encode","changed_name":"video_encode_1"},  {"original_name":"video_transcode","changed_name":"video_transcode_1"}]

I want to iterate through the array and extract the value for each element in a loop. I saw jq. I find it difficult to use it to iterate. How can I do that?

Just use a filter that would return each item in the array. Then loop over the results, just make sure you use the compact output option (-c) so each result is put on a single line and is treated as one item in the loop.

jq -c '.[]' input.json | while read i; do
    # do stuff with $i
done

jq has a shell formatting option: @sh.

You can use the following to format your json data as shell parameters:

cat data.json | jq '. | map([.original_name, .changed_name])' | jq @sh

The output will look like:

"'pdf_convert' 'pdf_convert_1'"
"'video_encode' 'video_encode_1'",
"'video_transcode' 'video_transcode_1'"

To process each row, we need to do a couple of things:

• Set the bash for-loop to read the entire row, rather than stopping at the first space (default behavior).
• Strip the enclosing double-quotes off of each row, so each value can be passed as a parameter to the function which processes each row.

To read the entire row on each iteration of the bash for-loop, set the IFS variable, as described in this answer.

To strip off the double-quotes, we'll run it through the bash shell interpreter using xargs:

stripped=$(echo $original | xargs echo)

---

Putting it all together, we have:

#!/bin/bash

function processRow() {
  original_name=$1
  changed_name=$2

  # TODO
}

IFS=$'\n' # Each iteration of the for loop should read until we find an end-of-line
for row in $(cat data.json | jq '. | map([.original_name, .changed_name])' | jq @sh)
do
  # Run the row through the shell interpreter to remove enclosing double-quotes
  stripped=$(echo $row | xargs echo)

  # Call our function to process the row
  # eval must be used to interpret the spaces in $stripped as separating arguments
  eval processRow $stripped
done
unset IFS # Return IFS to its original value

By leveraging the power of Bash arrays, you can do something like:

# read each item in the JSON array to an item in the Bash array
readarray -t my_array < <(jq -c '.[]' input.json)

# iterate through the Bash array
for item in "${my_array[@]}"; do
  original_name=$(jq '.original_name' <<< "$item")
  changed_name=$(jq '.changed_name' <<< "$item")
  # do your stuff
done

From https://stackoverflow.com/questions/58635875/iterate-over-json-array-of-dates-in-bash-has-whitespace

items=$(echo "$JSON_Content" | jq -c -r '.[]')
for item in ${items[@]}; do
    echo $item
    # whatever you are trying to do ...
done

Try Build it around this example. (Source: Original Site)

Example:

jq '[foreach .[] as $item ([[],[]]; if $item == null then [[],.[0]]     else [(.[0] + [$item]),[]] end; if $item == null then .[1] else empty end)]'

Input [1,2,3,4,null,"a","b",null]

Output [[1,2,3,4],["a","b"]]

An earlier answer in this thread suggested using jq's foreach, but that may be much more complicated than needed, especially given the stated task. Specifically, foreach (and reduce) are intended for certain cases where you need to accumulate results.

In many cases (including some cases where eventually a reduction step is necessary), it's better to use .[] or map(_). The latter is just another way of writing [.[] | _] so if you are going to use jq, it's really useful to understand that .[] simply creates a stream of values. For example, [1,2,3] | .[] produces a stream of the three values.

To take a simple map-reduce example, suppose you want to find the maximum length of an array of strings. One solution would be [ .[] | length] | max.

I stopped using jq and started using jp, since JMESpath is the same language as used by the --query argument of my cloud service and I find it difficult to juggle both languages at once. You can quickly learn the basics of JMESpath expressions here: https://jmespath.org/tutorial.html

Since you didn't specifically ask for a jq answer but instead, an approach to iterating JSON in bash, I think it's an appropriate answer.

Style points:

1. I use backticks and those have fallen out of fashion. You can substitute with another command substitution operator.
2. I use cat to pipe the input contents into the command. Yes, you can also specify the filename as a parameter, but I find this distracting because it breaks my left-to-right reading of the sequence of operations. Of course you can update this from my style to yours.
3. set -u has no function in this solution, but is important if you are fiddling with bash to get something to work. The command forces you to declare variables and therefore doesn't allow you to misspell a variable name.

Here's how I do it:

#!/bin/bash
set -u

# exploit the JMESpath length() function to get a count of list elements to iterate
export COUNT=`cat data.json | jp "length( [*] )"`

# The `seq` command produces the sequence `0 1 2` for our indexes
# The $(( )) operator in bash produces an arithmetic result ($COUNT minus one)
for i in `seq 0 $((COUNT - 1))` ; do

     # The list elements in JMESpath are zero-indexed
     echo "Here is element $i:"
     cat data.json | jp "[$i]"

     # Add or replace whatever operation you like here.

done

Now, it would also be a common use case to pull the original JSON data from an online API and not from a local file. In that case, I use a slightly modified technique of caching the full result in a variable:

#!/bin/bash
set -u

# cache the JSON content in a stack variable, downloading it only once
export DATA=`api --profile foo compute instance list --query "bar"`

export COUNT=`echo "$DATA" | jp "length( [*] )"`
for i in `seq 0 $((COUNT - 1))` ; do
     echo "Here is element $i:"
     echo "$DATA" | jp "[$i]"
done

This second example has the added benefit that if the data is changing rapidly, you are guaranteed to have a consistent count between the elements you are iterating through, and the elements in the iterated data.

This is what I have done so far

 arr=$(echo "$array" | jq -c -r '.[]')
            for item in ${arr[@]}; do
               original_name=$(echo $item | jq -r '.original_name')
               changed_name=$(echo $item | jq -r '.changed_name')
              echo $original_name $changed_name
            done