jq examples
jq is a command-line JSON processor to parse json format data. You can find the detailed documentation here. And you can try to play it online at jqplay.org. I am listing out few jq command line examples that I found quite useful for day-to-day work 😃.
Let’s try jq with AWS resources api.
// to parse and extract json data
$ curl -s \
https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/index.json \
| jq .offers.AmazonEC2
{
"offerCode": "AmazonEC2",
"versionIndexUrl": "/offers/v1.0/aws/AmazonEC2/index.json",
"currentVersionUrl": "/offers/v1.0/aws/AmazonEC2/current/index.json",
"currentRegionIndexUrl": "/offers/v1.0/aws/AmazonEC2/current/region_index.json"
}
To capture the field of “currentVersionUrl”
curl -s \
https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/index.json \
| jq .offers.AmazonEC2 \
| jq .currentVersionUrl
"/offers/v1.0/aws/AmazonEC2/current/index.json"
--raw-output / -r
is the flag used to capture the json string without quotes. It is actually quite useful if you want to use jq in a bash script
curl -s \
https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/index.json \
| jq .offers.AmazonEC2 \
| jq -r .currentVersionUrl
/offers/v1.0/aws/AmazonEC2/current/index.json
// using jq in a script
for i in $(curl -s \
https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/index.json \
| jq -r .offers.AmazonEC2.versionIndexUrl ); \
do echo https://pricing.us-east-1.amazonaws.com$i ; \
done
https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/index.json
for i in $(curl -s \
https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/index.json \
| jq .offers.AmazonEC2.versionIndexUrl ); \
do echo https://pricing.us-east-1.amazonaws.com$i ; \
done
https://pricing.us-east-1.amazonaws.com"/offers/v1.0/aws/AmazonEC2/index.json"
// see sometimes it is better to parse and capture the data without quotes
To parse the json data with arrary[]
.
curl -s \
--request GET \
--url 'http://quotes.rest/qod.json' \
| jq .
// by running jq, you can get a formatted json output which has already been parsed
{
"success": {
"total": 1
},
"contents": {
"quotes": [
{
"quote": "Do not wait to strike till the iron is hot; but make it hot by striking....",
"length": "75",
"author": "William B. Sprague",
"tags": [
"action",
"inspire",
"tod"
],
"category": "inspire",
"date": "2019-08-13",
"permalink": "https://theysaidso.com/quote/william-b-sprague-do-not-wait-to-strike-till-the-iron-is-hot-but-make-it-hot-by",
"title": "Inspiring Quote of the day",
"background": "https://theysaidso.com/img/bgs/man_on_the_mountain.jpg",
"id": "UNe_Znlalg5mFEBaAWzoCweF"
}
],
"copyright": "2017-19 theysaidso.com"
}
}
// To capture the json data within the array[]
curl -s \
--request GET \
--url 'http://quotes.rest/qod.json' \
| jq .contents.quotes[] \
| jq '[.quote, .tags[0], .tags[1] ]'
[
"Do not wait to strike till the iron is hot; but make it hot by striking....",
"action",
"inspire"
]
To filter json data:
curl -s \
--request GET \
--url 'http://quotes.rest/qod.json' \
| jq .contents.quotes[] \
| jq 'select(.tags[] == null)'
// no data returned since everything has been filtered out
curl -s \
--request GET \
--url 'http://quotes.rest/qod.json' \
| jq .contents.quotes[] \
| jq 'select(.tags[] = "td") | [.quote, .tags[1]]'
[
"Do not wait to strike till the iron is hot; but make it hot by striking....",
"inspire"
]
curl -s \
--request GET \
--url 'http://quotes.rest/qod.json' \
| jq .contents.quotes[] \
| jq 'select(.tags[] == "tod") | [.quote, .tags[1]]'
[
"Do not wait to strike till the iron is hot; but make it hot by striking....",
"inspire"
]
// "=" can work as contains instead of equals
To reformat json data:
curl -s \
--request GET \
--url 'http://quotes.rest/qod.json' \
| jq .contents.quotes[] \
| jq '{tag_1: .tags[0], tag_2: .tags[1], tag_3: .tags[2]}'
{
"tag_1": "action",
"tag_2": "inspire",
"tag_3": "tod"
}
curl -s \
--request GET \
--url 'http://quotes.rest/qod.json' \
| jq .contents.quotes[] \
| jq '{tag_1: .tags[0], tag_2: .tags[1], tag_3: .tags[2]}' \
| jq .tag_1
"action"
Also there is some interesing array[] indexing jq is following. It works similar to iterating over slices in golang. Something like this:
curl -s \
--request GET \
--url 'http://quotes.rest/qod.json' \
| jq .contents.quotes[] \
| jq 'select(.tags[] != "tod")| .quote'
"Do not wait to strike till the iron is hot; but make it hot by striking...."
"Do not wait to strike till the iron is hot; but make it hot by striking...."
curl -s \
--request GET \
--url 'http://quotes.rest/qod.json' \
| jq .contents.quotes[] \
| jq 'select(.tags[] != "td")| .quote'
"Do not wait to strike till the iron is hot; but make it hot by striking...."
"Do not wait to strike till the iron is hot; but make it hot by striking...."
"Do not wait to strike till the iron is hot; but make it hot by striking...."
To escape null, you can use //
as the alternative operator:
curl -s \
--request GET \
--url 'http://quotes.rest/qod.json' \
| jq .contents.quotes[] \
| jq '{tag_1: .tags[0], tag_2: .tags[1], tag_3: .tags[2]}' \
| jq '[.tag_1, .tag_2, .tag_4//"no that tag yet"]'
[
"action",
"inspire",
"no that tag yet"
]
Oh well, there are much more jq commands you can use, like a MYSQL style of join()
. Documentaion is quite good and you can find it here. Leave a comment if you have more jq questions 😃