JQ what I learned today

#jq #json #bash #todayilearned

What's jq ?

A command line parser for json that can be downloaded here or you can use the playground. I will use this local json file as an beginner example hello-world.json

{
  "hello": "world",
  "Water":{
    "Atlantic":"Ocean",
    "Pacific":"Ocean",
    "Mediterranean":{
      "Ocean": false,
      "Sea": true
    }
  },
  "Land":{
    "Europe":[
      "Ireland",
      "France",
      "Germany"
    ],
    "Asia":[
      "India",
      "China"
    ]
  }
}

Using jq

Simplest use case is to render the whole json file and pretty print it

# You can pass filters and json files as arguments
jq '.' hello-world.json
# You can cat the file and pipe it to jq
cat hello-world.json | jq
# You can curl remote json and pipe it to jq e.g. from https://stedolan.github.io/jq/tutorial/
curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' | jq '.'

Filtering with jq

The manual is full of options for filtering and can be a lot to take in. We have seen already the most basic filter '.' which can be used with either method of invoking jq to return all data in the json.

# We can filter to specific keys
$ jq '.hello' hello-world.json                                               
"world"
#  you can create a new object and create a new json files
$ cat hello-world.json | jq .Water > water.json
{
  "Atlantic": "Ocean",
  "Pacific": "Ocean",
  "Mediterranean": {
    "Ocean": false,
    "Sea": true
  }
}
# We can drill down
$ cat hello-world.json | jq '.Water.Atlantic'
"Ocean"
$ cat hello-world.json | jq '.Water.Mediterranean'
{
  "Ocean": false,
  "Sea": true
}
$ cat hello-world.json | jq '.Water.Mediterranean.Ocean'
false
# arrays syntax is a little different
$ cat hello-world.json | jq '.Land.Europe[]'
"Ireland"
"France"
"Germany"
$ cat hello-world.json | jq '.Land.Europe'  
[
  "Ireland",
  "France",
  "Germany"
]
# Arrays can be accessed by indexes or splits [0:2]
$ cat hello-world.json | jq '.Land.Europe[0]'
"Ireland"
$ cat hello-world.json| jq '.Land.Europe[0:2]'
[
  "Ireland",
  "France"
]
# array symbol can be used to iterate over values
$ cat hello-world.json | jq '.Land[]'
[
  "Ireland",
  "France",
  "Germany"
]
[
  "India",
  "China"
]
$ cat hello-world.json | jq '.Water[]'
"Ocean"
"Ocean"
{
  "Ocean": false,
  "Sea": true
}
# Lots of other array slicing methods available in the manual this is only scratching the surface

You can pipe multiple filters together and reassign values to create new json

$ cat hello-world.json | jq '.Land | .Europe | {EU:[{country:.[]}]}'
{
  "EU": [
    {
      "country": "Ireland"
    },
    {
      "country": "France"
    },
    {
      "country": "Germany"
    }
  ]
}

Using jq files to do more advance stuff

You can use jq files as filters with the -f flag to create more complex filters with supporting functions. The file Node-metrics.json has some cpu and memory usages stats from a Kubernetes Openshift cluster. I am looking to generate a report from this json.

I created the following file filterGenerateCsv.jq to filter the data in the json and creates an csv.

# function definition filter the data
def getNodeMetrics:
    [.items[]|
    {
        name:.metadata.name,
        cpu:.usage.cpu, 
        memory:.usage.memory
    }
    ];


# use geNodeUsages to generate csv
.[] |
getNodeMetrics |
group_by(.name) |
map({
  "Node Name": .[].name ,
  "CPU Node - Real": [.[].cpu] | add, 
  "Memory Node - Real": [.[].memory] | add
})| (.[0] | to_entries | map(.key)), (.[] | [.[]]) | @csv

#  (.[0] | to_entries | map(.key)) this filter sets the first array as follows
# [
#  "Node Name",
#  "CPU Node - Real",
#  "Memory Node - Real"
# ]
# (.[] | [.[]]) filter generates the arrays of values and @csv pipes to csv format

I can then call the jq file as my filter argument with the -f flag

# Curl the file and use the -f flag for file to point at filterGenerateCsv.jq
$ curl https://gist.githubusercontent.com/austincunningham/eca562b35651b9a6dd214d0023c19cdf/raw | jq -f filterGenerateCsv.jq 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5594  100  5594    0     0   9497      0 --:--:-- --:--:-- --:--:--  9481
"\"Node Name\",\"CPU Node - Real\",\"Memory Node - Real\""
"\"ip-10-0-130-91.eu-west-1.compute.internal\",\"130m\",\"1596380Ki\""
"\"ip-10-0-131-137.eu-west-1.compute.internal\",\"115m\",\"1257300Ki\""
"\"ip-10-0-133-196.eu-west-1.compute.internal\",\"248m\",\"1787868Ki\""
"\"ip-10-0-134-141.eu-west-1.compute.internal\",\"473m\",\"3330440Ki\""
"\"ip-10-0-137-148.eu-west-1.compute.internal\",\"127m\",\"1485252Ki\""
"\"ip-10-0-139-72.eu-west-1.compute.internal\",\"172m\",\"1524604Ki\""
"\"ip-10-0-140-152.eu-west-1.compute.internal\",\"113m\",\"1276292Ki\""
"\"ip-10-0-140-180.eu-west-1.compute.internal\",\"609m\",\"4152784Ki\""
"\"ip-10-0-142-230.eu-west-1.compute.internal\",\"667m\",\"4110252Ki\""
"\"ip-10-0-142-51.eu-west-1.compute.internal\",\"727m\",\"4080132Ki\""