Jq is a very convenient tool to handle JSON from the cli or in scripts and programs.

Yet, many find it complex to use, and not obvious.

Jq relies on a few core concepts. Once understood, they make using jq a lot easier.

In this post, I’ll discuss the 4 most important jq concepts. For each, I’ll give a few examples of usual things that are possible.

## 1. Concepts

Jq allows you do a few simple things, and to mix them together:

1. formatting and conversion;

2. filtering;

3. modifying;

4. building new objects.

Once each concept is clear, you’ll be able to decompose your problem into those 4 axis, and enjoy using jq.

## 2. Formatting and conversion

### 2.1. Definition and usage

Formatting is about converting any JSON into a standard, well-formatted shape.

This is usually the first step when receiving a JSON, both to make it easier to read by people, but also to detect errors from a malformed document.

This is the simplest action, and the shortest:   (yes, just nothing) or ..

fromjson and tojsong allow converting a JSON to a string and vice versa.

### 2.2. Examples

$echo '{"name": "Julia", "age": "unknown"}' | jq { "name": "Julia", "age": "unknown" } #### 2.2.2. Ensuring the JSON is well-formed $ echo '{"name": "Julia" "age": "unknown"}' | jq
parse error: Expected separator between values at line 1, column 22

#### 2.2.3. Piping from curl

curl outputs the body of the http response on the standard input, and the rest on the error output, allowing one to use jq directly after a curl call.

$curl -v "http://worldtimeapi.org/api/timezone/Europe/Berlin" | jq * Connected to worldtimeapi.org (34.253.22.180) port 80 (#0) > GET /api/timezone/Europe/Berlin HTTP/1.1 > Host: worldtimeapi.org > User-Agent: curl/7.70.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < Connection: keep-alive < Access-Control-Allow-Credentials: true < Access-Control-Allow-Origin: * < Access-Control-Expose-Headers: < Cache-Control: max-age=0, private, must-revalidate < Content-Length: 394 < Content-Type: application/json; charset=utf-8 < Cross-Origin-Window-Policy: deny < Date: Thu, 11 Jun 2020 06:58:20 GMT < Server: Cowboy < X-Content-Type-Options: nosniff < X-Download-Options: noopen < X-Frame-Options: SAMEORIGIN < X-Permitted-Cross-Domain-Policies: none < X-Request-Id: 1e775e63-0da6-4a15-9993-98dfc6cdb855 < X-Runtime: 2ms < X-Xss-Protection: 1; mode=block < Via: 1.1 vegur < { [394 bytes data] 100 394 100 394 0 0 4104 0 --:--:-- --:--:-- --:--:-- 4104 * Connection #0 to host worldtimeapi.org left intact { "abbreviation": "CEST", "client_ip": "95.90.197.39", "datetime": "2020-06-11T08:58:20.767744+02:00", "day_of_week": 4, "day_of_year": 163, "dst": true, "dst_from": "2020-03-29T01:00:00+00:00", "dst_offset": 3600, "dst_until": "2020-10-25T01:00:00+00:00", "raw_offset": 3600, "timezone": "Europe/Berlin", "unixtime": 1591858700, "utc_datetime": "2020-06-11T06:58:20.767744+00:00", "utc_offset": "+02:00", "week_number": 24 } ## 3. Filtering ### 3.1. Definition and usage Filtering allows extracting parts of a JSON. You can filter with almost anything you want, from fields names, index in an array, or by searching for a match. ### 3.2. Examples #### 3.2.1. Filtering by selecting a field Syntax: .field.subfield.subsubfield. Input to filter by field { "blue": { "rgb": { "r": 124, "g": 445, "b": 777 }, "hex": "345F28" }, "green": { "rgb": { "r": 845, "g": 234, "b": 099 }, "hex": "FFF445" } } Query: .blue.rgb.b Output filtered by field 777 #### 3.2.2. Filtering arrays by index Syntax: • [index] or nth(index) (or first for [0]). • [begin:end] • [-1] or last for the last one. Input, select First Item of an array [ { "name": "blue", "rgb": { "r": 124, "g": 445, "b": 777 }, "hex": "345F28" }, { "name": "green", "rgb": { "r": 845, "g": 234, "b": 099 }, "hex": "FFF445" } ] Queries (equivalent): • .[0] • . | first • first(.). Output, select First Item of an array { "name": "blue", "rgb": { "r": 124, "g": 445, "b": 777 }, "hex": "345F28" } #### 3.2.3. Filtering by finding a match Syntax: | select(expression). expression can be complex, and use jq functions as well. Input, find the blue colour in the list { "colours":[ { "name": "blue", "rgb": { "r": 124, "g": 445, "b": 777 }, "hex": "345F28" }, { "name": "green", "rgb": { "r": 845, "g": 234, "b": 099 }, "hex": "FFF445" } ] } Queries (equivalent): • .colours[] | select(.name=="blue") • .colours[] | select(.name | test("blu.\*")) (with regexps). Output, blue colour found { "name": "blue", "rgb": { "r": 124, "g": 445, "b": 777 }, "hex": "345F28" } Another example is to find all colours with enough green (same input, and same output): Query: .colours[] | select(.rgb.g > 400) ## 4. Modifying ### 4.1. Definition and usage You can modify part of a JSON, or construct a totally new JSON using sub-pieces. ### 4.2. Examples #### 4.2.1. Modifying values Using the same input as before, we can decide to increase by 10 all the green components in the rgb values. Queries (equivalent): • .colours[].rgb.g |= . + 10 • .colours[].rgb.g += 10. Output, green channels increased by 10 { "colours": [ { "name": "blue", "rgb": { "r": 124, "g": 455, "b": 777 }, "hex": "345F28" }, { "name": "green", "rgb": { "r": 845, "g": 244, "b": 99 }, "hex": "FFF445" } ] } #### 4.2.2. Adding static keys Using the same input as before, we decide that colours now have a new field owner with a fixed value. Query: .colours[].owner = "everyone" Output, new fixed owner key added { "colours": [ { "name": "blue", "rgb": { "r": 124, "g": 445, "b": 777 }, "hex": "345F28", "owner": "everyone" }, { "name": "green", "rgb": { "r": 845, "g": 234, "b": 99 }, "hex": "FFF445", "owner": "everyone" } ] } #### 4.2.3. Modifying a key’s name Using the same input as before, we decide that colours no longer have a name, but an id instead. This is slightly more complicated, because we will have to create new objects, because keys can’t be changed that easily. Query: .colours = [.colours[] | with_entries(if .key == "name" then .key = "id" else . end)] Output, changing key name to id { "colours": [ { "id": "blue", "rgb": { "r": 124, "g": 445, "b": 777 }, "hex": "345F28" }, { "id": "green", "rgb": { "r": 845, "g": 234, "b": 99 }, "hex": "FFF445" } ] } #### 4.2.4. Adding dynamic keys What if we want to add a new key whose value depends on the values of other keys in same object? Let’s build a new key address, from the name of the colour (same input): Queries (equivalent): • reduce .colours[] as$item ([]; . + [$item + {address: ("https://colours.com/" +$item.name)}])

• foreach .colours[] as $item ([]; . + [$item + {address: ("https://colours.com/" + $item.name)}]; if$item.name == "green" then . else empty end)

Output, new key address built from name
[
{
"name": "blue",
"rgb": {
"r": 124,
"g": 445,
"b": 777
},
"hex": "345F28",
},
{
"name": "green",
"rgb": {
"r": 845,
"g": 234,
"b": 99
},
"hex": "FFF445",
]