"Fossies" - the Fresh Open Source Software Archive 
Member "jq-1.6/docs/content/3.manual/manual.yml" (2 Nov 2018, 124370 Bytes) of package /linux/misc/jq-1.6.tar.gz:
As a special service "Fossies" has tried to format the requested source page into HTML format using (guessed) Ansible YAML source code syntax highlighting (style:
standard) with prefixed line numbers.
Alternatively you can here
view or
download the uninterpreted source code file.
1 ---
2 headline: jq Manual (development version)
3
4 history: |
5
6 *For released versions, see [jq 1.6](/jq/manual/v1.6),
7 [jq 1.5](/jq/manual/v1.5), [jq 1.4](/jq/manual/v1.4)
8 or [jq 1.3](/jq/manual/v1.3).*
9
10 body: |
11
12 A jq program is a "filter": it takes an input, and produces an
13 output. There are a lot of builtin filters for extracting a
14 particular field of an object, or converting a number to a string,
15 or various other standard tasks.
16
17 Filters can be combined in various ways - you can pipe the output of
18 one filter into another filter, or collect the output of a filter
19 into an array.
20
21 Some filters produce multiple results, for instance there's one that
22 produces all the elements of its input array. Piping that filter
23 into a second runs the second filter for each element of the
24 array. Generally, things that would be done with loops and iteration
25 in other languages are just done by gluing filters together in jq.
26
27 It's important to remember that every filter has an input and an
28 output. Even literals like "hello" or 42 are filters - they take an
29 input but always produce the same literal as output. Operations that
30 combine two filters, like addition, generally feed the same input to
31 both and combine the results. So, you can implement an averaging
32 filter as `add / length` - feeding the input array both to the `add`
33 filter and the `length` filter and then performing the division.
34
35 But that's getting ahead of ourselves. :) Let's start with something
36 simpler:
37
38 manpage_intro: |
39 jq(1) -- Command-line JSON processor
40 ====================================
41
42 ## SYNOPSIS
43
44 `jq` [<options>...] <filter> [<files>...]
45
46 `jq` can transform JSON in various ways, by selecting, iterating,
47 reducing and otherwise mangling JSON documents. For instance,
48 running the command `jq 'map(.price) | add'` will take an array of
49 JSON objects as input and return the sum of their "price" fields.
50
51 `jq` can accept text input as well, but by default, `jq` reads a
52 stream of JSON entities (including numbers and other literals) from
53 `stdin`. Whitespace is only needed to separate entities such as 1
54 and 2, and true and false. One or more <files> may be specified, in
55 which case `jq` will read input from those instead.
56
57 The <options> are described in the [INVOKING JQ] section; they
58 mostly concern input and output formatting. The <filter> is written
59 in the jq language and specifies how to transform the input
60 file or document.
61
62 ## FILTERS
63
64 manpage_epilogue: |
65 ## BUGS
66
67 Presumably. Report them or discuss them at:
68
69 https://github.com/stedolan/jq/issues
70
71 ## AUTHOR
72
73 Stephen Dolan `<mu@netsoc.tcd.ie>`
74
75 sections:
76 - title: Invoking jq
77 body: |
78
79 jq filters run on a stream of JSON data. The input to jq is
80 parsed as a sequence of whitespace-separated JSON values which
81 are passed through the provided filter one at a time. The
82 output(s) of the filter are written to standard out, again as a
83 sequence of whitespace-separated JSON data.
84
85 Note: it is important to mind the shell's quoting rules. As a
86 general rule it's best to always quote (with single-quote
87 characters) the jq program, as too many characters with special
88 meaning to jq are also shell meta-characters. For example, `jq
89 "foo"` will fail on most Unix shells because that will be the same
90 as `jq foo`, which will generally fail because `foo is not
91 defined`. When using the Windows command shell (cmd.exe) it's
92 best to use double quotes around your jq program when given on the
93 command-line (instead of the `-f program-file` option), but then
94 double-quotes in the jq program need backslash escaping.
95
96 You can affect how jq reads and writes its input and output
97 using some command-line options:
98
99 * `--version`:
100
101 Output the jq version and exit with zero.
102
103 * `--seq`:
104
105 Use the `application/json-seq` MIME type scheme for separating
106 JSON texts in jq's input and output. This means that an ASCII
107 RS (record separator) character is printed before each value on
108 output and an ASCII LF (line feed) is printed after every
109 output. Input JSON texts that fail to parse are ignored (but
110 warned about), discarding all subsequent input until the next
111 RS. This mode also parses the output of jq without the `--seq`
112 option.
113
114 * `--stream`:
115
116 Parse the input in streaming fashion, outputing arrays of path
117 and leaf values (scalars and empty arrays or empty objects).
118 For example, `"a"` becomes `[[],"a"]`, and `[[],"a",["b"]]`
119 becomes `[[0],[]]`, `[[1],"a"]`, and `[[1,0],"b"]`.
120
121 This is useful for processing very large inputs. Use this in
122 conjunction with filtering and the `reduce` and `foreach` syntax
123 to reduce large inputs incrementally.
124
125 * `--slurp`/`-s`:
126
127 Instead of running the filter for each JSON object in the
128 input, read the entire input stream into a large array and run
129 the filter just once.
130
131 * `--raw-input`/`-R`:
132
133 Don't parse the input as JSON. Instead, each line of text is
134 passed to the filter as a string. If combined with `--slurp`,
135 then the entire input is passed to the filter as a single long
136 string.
137
138 * `--null-input`/`-n`:
139
140 Don't read any input at all! Instead, the filter is run once
141 using `null` as the input. This is useful when using jq as a
142 simple calculator or to construct JSON data from scratch.
143
144 * `--compact-output` / `-c`:
145
146 By default, jq pretty-prints JSON output. Using this option
147 will result in more compact output by instead putting each
148 JSON object on a single line.
149
150 * `--tab`:
151
152 Use a tab for each indentation level instead of two spaces.
153
154 * `--indent n`:
155
156 Use the given number of spaces (no more than 8) for indentation.
157
158 * `--color-output` / `-C` and `--monochrome-output` / `-M`:
159
160 By default, jq outputs colored JSON if writing to a
161 terminal. You can force it to produce color even if writing to
162 a pipe or a file using `-C`, and disable color with `-M`.
163
164 Colors can be configured with the `JQ_COLORS` environment
165 variable (see below).
166
167 * `--ascii-output` / `-a`:
168
169 jq usually outputs non-ASCII Unicode codepoints as UTF-8, even
170 if the input specified them as escape sequences (like
171 "\u03bc"). Using this option, you can force jq to produce pure
172 ASCII output with every non-ASCII character replaced with the
173 equivalent escape sequence.
174
175 * `--unbuffered`
176
177 Flush the output after each JSON object is printed (useful if
178 you're piping a slow data source into jq and piping jq's
179 output elsewhere).
180
181 * `--sort-keys` / `-S`:
182
183 Output the fields of each object with the keys in sorted order.
184
185 * `--raw-output` / `-r`:
186
187 With this option, if the filter's result is a string then it
188 will be written directly to standard output rather than being
189 formatted as a JSON string with quotes. This can be useful for
190 making jq filters talk to non-JSON-based systems.
191
192 * `--join-output` / `-j`:
193
194 Like `-r` but jq won't print a newline after each output.
195
196 * `-f filename` / `--from-file filename`:
197
198 Read filter from the file rather than from a command line, like
199 awk's -f option. You can also use '#' to make comments.
200
201 * `-Ldirectory` / `-L directory`:
202
203 Prepend `directory` to the search list for modules. If this
204 option is used then no builtin search list is used. See the
205 section on modules below.
206
207 * `-e` / `--exit-status`:
208
209 Sets the exit status of jq to 0 if the last output values was
210 neither `false` nor `null`, 1 if the last output value was
211 either `false` or `null`, or 4 if no valid result was ever
212 produced. Normally jq exits with 2 if there was any usage
213 problem or system error, 3 if there was a jq program compile
214 error, or 0 if the jq program ran.
215
216 Another way to set the exit status is with the `halt_error`
217 builtin function.
218
219 * `--arg name value`:
220
221 This option passes a value to the jq program as a predefined
222 variable. If you run jq with `--arg foo bar`, then `$foo` is
223 available in the program and has the value `"bar"`. Note that
224 `value` will be treated as a string, so `--arg foo 123` will
225 bind `$foo` to `"123"`.
226
227 Named arguments are also available to the jq program as
228 `$ARGS.named`.
229
230 * `--argjson name JSON-text`:
231
232 This option passes a JSON-encoded value to the jq program as a
233 predefined variable. If you run jq with `--argjson foo 123`, then
234 `$foo` is available in the program and has the value `123`.
235
236 * `--slurpfile variable-name filename`:
237
238 This option reads all the JSON texts in the named file and binds
239 an array of the parsed JSON values to the given global variable.
240 If you run jq with `--slurpfile foo bar`, then `$foo` is available
241 in the program and has an array whose elements correspond to the
242 texts in the file named `bar`.
243
244 * `--rawfile variable-name filename`:
245
246 This option reads in the named file and binds its contents to the given
247 global variable. If you run jq with `--rawfile foo bar`, then `$foo` is
248 available in the program and has a string whose contents are to the texs
249 in the file named `bar`.
250
251 * `--argfile variable-name filename`:
252
253 Do not use. Use `--slurpfile` instead.
254
255 (This option is like `--slurpfile`, but when the file has just
256 one text, then that is used, else an array of texts is used as
257 in `--slurpfile`.)
258
259 * `--args`:
260
261 Remaining arguments are positional string arguments. These are
262 available to the jq program as `$ARGS.positional[]`.
263
264 * `--jsonargs`:
265
266 Remaining arguments are positional JSON text arguments. These
267 are available to the jq program as `$ARGS.positional[]`.
268
269 * `--run-tests [filename]`:
270
271 Runs the tests in the given file or standard input. This must
272 be the last option given and does not honor all preceding
273 options. The input consists of comment lines, empty lines, and
274 program lines followed by one input line, as many lines of
275 output as are expected (one per output), and a terminating empty
276 line. Compilation failure tests start with a line containing
277 only "%%FAIL", then a line containing the program to compile,
278 then a line containing an error message to compare to the
279 actual.
280
281 Be warned that this option can change backwards-incompatibly.
282
283 - title: Basic filters
284 entries:
285 - title: "Identity: `.`"
286 body: |
287
288 The absolute simplest filter is `.` . This is a filter that
289 takes its input and produces it unchanged as output. That is,
290 this is the identity operator.
291
292 Since jq by default pretty-prints all output, this trivial
293 program can be a useful way of formatting JSON output from,
294 say, `curl`.
295
296 examples:
297 - program: '.'
298 input: '"Hello, world!"'
299 output: ['"Hello, world!"']
300
301 - title: "Object Identifier-Index: `.foo`, `.foo.bar`"
302 body: |
303
304 The simplest *useful* filter is `.foo`. When given a
305 JSON object (aka dictionary or hash) as input, it produces
306 the value at the key "foo", or null if there's none present.
307
308 A filter of the form `.foo.bar` is equivalent to `.foo|.bar`.
309
310 This syntax only works for simple, identifier-like keys, that
311 is, keys that are all made of alphanumeric characters and
312 underscore, and which do not start with a digit.
313
314 If the key contains special characters, you need to surround
315 it with double quotes like this: `."foo$"`, or else `.["foo$"]`.
316
317 For example `.["foo::bar"]` and `.["foo.bar"]` work while
318 `.foo::bar` does not, and `.foo.bar` means `.["foo"].["bar"]`.
319
320 examples:
321 - program: '.foo'
322 input: '{"foo": 42, "bar": "less interesting data"}'
323 output: [42]
324 - program: '.foo'
325 input: '{"notfoo": true, "alsonotfoo": false}'
326 output: ['null']
327 - program: '.["foo"]'
328 input: '{"foo": 42}'
329 output: [42]
330
331 - title: "Optional Object Identifier-Index: `.foo?`"
332 body: |
333
334 Just like `.foo`, but does not output even an error when `.`
335 is not an array or an object.
336
337 examples:
338 - program: '.foo?'
339 input: '{"foo": 42, "bar": "less interesting data"}'
340 output: [42]
341 - program: '.foo?'
342 input: '{"notfoo": true, "alsonotfoo": false}'
343 output: ['null']
344 - program: '.["foo"]?'
345 input: '{"foo": 42}'
346 output: [42]
347 - program: '[.foo?]'
348 input: '[1,2]'
349 output: ['[]']
350
351 - title: "Generic Object Index: `.[<string>]`"
352 body: |
353
354 You can also look up fields of an object using syntax like
355 `.["foo"]` (.foo above is a shorthand version of this, but
356 only for identifier-like strings).
357
358 - title: "Array Index: `.[2]`"
359 body: |
360
361 When the index value is an integer, `.[<value>]` can index
362 arrays. Arrays are zero-based, so `.[2]` returns the third
363 element.
364
365 Negative indices are allowed, with -1 referring to the last
366 element, -2 referring to the next to last element, and so on.
367
368 examples:
369 - program: '.[0]'
370 input: '[{"name":"JSON", "good":true}, {"name":"XML", "good":false}]'
371 output: ['{"name":"JSON", "good":true}']
372
373 - program: '.[2]'
374 input: '[{"name":"JSON", "good":true}, {"name":"XML", "good":false}]'
375 output: ['null']
376
377 - program: '.[-2]'
378 input: '[1,2,3]'
379 output: ['2']
380
381 - title: "Array/String Slice: `.[10:15]`"
382 body: |
383
384 The `.[10:15]` syntax can be used to return a subarray of an
385 array or substring of a string. The array returned by
386 `.[10:15]` will be of length 5, containing the elements from
387 index 10 (inclusive) to index 15 (exclusive). Either index may
388 be negative (in which case it counts backwards from the end of
389 the array), or omitted (in which case it refers to the start
390 or end of the array).
391
392 examples:
393 - program: '.[2:4]'
394 input: '["a","b","c","d","e"]'
395 output: ['["c", "d"]']
396
397 - program: '.[2:4]'
398 input: '"abcdefghi"'
399 output: ['"cd"']
400
401 - program: '.[:3]'
402 input: '["a","b","c","d","e"]'
403 output: ['["a", "b", "c"]']
404
405 - program: '.[-2:]'
406 input: '["a","b","c","d","e"]'
407 output: ['["d", "e"]']
408
409 - title: "Array/Object Value Iterator: `.[]`"
410 body: |
411
412 If you use the `.[index]` syntax, but omit the index
413 entirely, it will return *all* of the elements of an
414 array. Running `.[]` with the input `[1,2,3]` will produce the
415 numbers as three separate results, rather than as a single
416 array.
417
418 You can also use this on an object, and it will return all
419 the values of the object.
420
421 examples:
422 - program: '.[]'
423 input: '[{"name":"JSON", "good":true}, {"name":"XML", "good":false}]'
424 output:
425 - '{"name":"JSON", "good":true}'
426 - '{"name":"XML", "good":false}'
427
428 - program: '.[]'
429 input: '[]'
430 output: []
431
432 - program: '.[]'
433 input: '{"a": 1, "b": 1}'
434 output: ['1', '1']
435
436 - title: "`.[]?`"
437 body: |
438
439 Like `.[]`, but no errors will be output if . is not an array
440 or object.
441
442 - title: "Comma: `,`"
443 body: |
444
445 If two filters are separated by a comma, then the
446 same input will be fed into both and the two filters' output
447 value streams will be concatenated in order: first, all of the
448 outputs produced by the left expression, and then all of the
449 outputs produced by the right. For instance, filter `.foo,
450 .bar`, produces both the "foo" fields and "bar" fields as
451 separate outputs.
452
453 examples:
454 - program: '.foo, .bar'
455 input: '{"foo": 42, "bar": "something else", "baz": true}'
456 output: ['42', '"something else"']
457
458 - program: ".user, .projects[]"
459 input: '{"user":"stedolan", "projects": ["jq", "wikiflow"]}'
460 output: ['"stedolan"', '"jq"', '"wikiflow"']
461
462 - program: '.[4,2]'
463 input: '["a","b","c","d","e"]'
464 output: ['"e"', '"c"']
465
466 - title: "Pipe: `|`"
467 body: |
468
469 The | operator combines two filters by feeding the output(s) of
470 the one on the left into the input of the one on the right. It's
471 pretty much the same as the Unix shell's pipe, if you're used to
472 that.
473
474 If the one on the left produces multiple results, the one on
475 the right will be run for each of those results. So, the
476 expression `.[] | .foo` retrieves the "foo" field of each
477 element of the input array.
478
479 Note that `.a.b.c` is the same as `.a | .b | .c`.
480
481 Note too that `.` is the input value at the particular stage
482 in a "pipeline", specifically: where the `.` expression appears.
483 Thus `.a | . | .b` is the same as `.a.b`, as the `.` in the
484 middle refers to whatever value `.a` produced.
485
486 examples:
487 - program: '.[] | .name'
488 input: '[{"name":"JSON", "good":true}, {"name":"XML", "good":false}]'
489 output: ['"JSON"', '"XML"']
490
491 - title: "Parenthesis"
492 body: |
493
494 Parenthesis work as a grouping operator just as in any typical
495 programming language.
496
497 examples:
498 - program: '(. + 2) * 5'
499 input: '1'
500 output: [15]
501
502 - title: Types and Values
503 body: |
504
505 jq supports the same set of datatypes as JSON - numbers,
506 strings, booleans, arrays, objects (which in JSON-speak are
507 hashes with only string keys), and "null".
508
509 Booleans, null, strings and numbers are written the same way as
510 in javascript. Just like everything else in jq, these simple
511 values take an input and produce an output - `42` is a valid jq
512 expression that takes an input, ignores it, and returns 42
513 instead.
514
515 entries:
516 - title: "Array construction: `[]`"
517 body: |
518
519 As in JSON, `[]` is used to construct arrays, as in
520 `[1,2,3]`. The elements of the arrays can be any jq
521 expression, including a pipeline. All of the results produced
522 by all of the expressions are collected into one big array.
523 You can use it to construct an array out of a known quantity
524 of values (as in `[.foo, .bar, .baz]`) or to "collect" all the
525 results of a filter into an array (as in `[.items[].name]`)
526
527 Once you understand the "," operator, you can look at jq's array
528 syntax in a different light: the expression `[1,2,3]` is not using a
529 built-in syntax for comma-separated arrays, but is instead applying
530 the `[]` operator (collect results) to the expression 1,2,3 (which
531 produces three different results).
532
533 If you have a filter `X` that produces four results,
534 then the expression `[X]` will produce a single result, an
535 array of four elements.
536
537 examples:
538 - program: "[.user, .projects[]]"
539 input: '{"user":"stedolan", "projects": ["jq", "wikiflow"]}'
540 output: ['["stedolan", "jq", "wikiflow"]']
541 - program: "[ .[] | . * 2]"
542 input: '[1, 2, 3]'
543 output: ['[2, 4, 6]']
544
545 - title: "Object Construction: `{}`"
546 body: |
547
548 Like JSON, `{}` is for constructing objects (aka
549 dictionaries or hashes), as in: `{"a": 42, "b": 17}`.
550
551 If the keys are "identifier-like", then the quotes can be left
552 off, as in `{a:42, b:17}`. Keys generated by expressions need
553 to be parenthesized, e.g., `{("a"+"b"):59}`.
554
555 The value can be any expression (although you may need to
556 wrap it in parentheses if it's a complicated one), which gets
557 applied to the {} expression's input (remember, all filters
558 have an input and an output).
559
560 {foo: .bar}
561
562 will produce the JSON object `{"foo": 42}` if given the JSON
563 object `{"bar":42, "baz":43}` as its input. You can use this
564 to select particular fields of an object: if the input is an
565 object with "user", "title", "id", and "content" fields and
566 you just want "user" and "title", you can write
567
568 {user: .user, title: .title}
569
570 Because that is so common, there's a shortcut syntax for it:
571 `{user, title}`.
572
573 If one of the expressions produces multiple results,
574 multiple dictionaries will be produced. If the input's
575
576 {"user":"stedolan","titles":["JQ Primer", "More JQ"]}
577
578 then the expression
579
580 {user, title: .titles[]}
581
582 will produce two outputs:
583
584 {"user":"stedolan", "title": "JQ Primer"}
585 {"user":"stedolan", "title": "More JQ"}
586
587 Putting parentheses around the key means it will be evaluated as an
588 expression. With the same input as above,
589
590 {(.user): .titles}
591
592 produces
593
594 {"stedolan": ["JQ Primer", "More JQ"]}
595
596 examples:
597 - program: '{user, title: .titles[]}'
598 input: '{"user":"stedolan","titles":["JQ Primer", "More JQ"]}'
599 output:
600 - '{"user":"stedolan", "title": "JQ Primer"}'
601 - '{"user":"stedolan", "title": "More JQ"}'
602 - program: '{(.user): .titles}'
603 input: '{"user":"stedolan","titles":["JQ Primer", "More JQ"]}'
604 output: ['{"stedolan": ["JQ Primer", "More JQ"]}']
605
606 - title: "Recursive Descent: `..`"
607 body: |
608
609 Recursively descends `.`, producing every value. This is the
610 same as the zero-argument `recurse` builtin (see below). This
611 is intended to resemble the XPath `//` operator. Note that
612 `..a` does not work; use `..|.a` instead. In the example
613 below we use `..|.a?` to find all the values of object keys
614 "a" in any object found "below" `.`.
615
616 This is particularly useful in conjunction with `path(EXP)`
617 (also see below) and the `?` operator.
618
619 examples:
620 - program: '..|.a?'
621 input: '[[{"a":1}]]'
622 output: ['1']
623
624 - title: Builtin operators and functions
625 body: |
626
627 Some jq operator (for instance, `+`) do different things
628 depending on the type of their arguments (arrays, numbers,
629 etc.). However, jq never does implicit type conversions. If you
630 try to add a string to an object you'll get an error message and
631 no result.
632
633 entries:
634 - title: "Addition: `+`"
635 body: |
636
637 The operator `+` takes two filters, applies them both
638 to the same input, and adds the results together. What
639 "adding" means depends on the types involved:
640
641 - **Numbers** are added by normal arithmetic.
642
643 - **Arrays** are added by being concatenated into a larger array.
644
645 - **Strings** are added by being joined into a larger string.
646
647 - **Objects** are added by merging, that is, inserting all
648 the key-value pairs from both objects into a single
649 combined object. If both objects contain a value for the
650 same key, the object on the right of the `+` wins. (For
651 recursive merge use the `*` operator.)
652
653 `null` can be added to any value, and returns the other
654 value unchanged.
655
656 examples:
657 - program: '.a + 1'
658 input: '{"a": 7}'
659 output: ['8']
660 - program: '.a + .b'
661 input: '{"a": [1,2], "b": [3,4]}'
662 output: ['[1,2,3,4]']
663 - program: '.a + null'
664 input: '{"a": 1}'
665 output: ['1']
666 - program: '.a + 1'
667 input: '{}'
668 output: ['1']
669 - program: '{a: 1} + {b: 2} + {c: 3} + {a: 42}'
670 input: 'null'
671 output: ['{"a": 42, "b": 2, "c": 3}']
672
673 - title: "Subtraction: `-`"
674 body: |
675
676 As well as normal arithmetic subtraction on numbers, the `-`
677 operator can be used on arrays to remove all occurrences of
678 the second array's elements from the first array.
679
680 examples:
681 - program: '4 - .a'
682 input: '{"a":3}'
683 output: ['1']
684 - program: . - ["xml", "yaml"]
685 input: '["xml", "yaml", "json"]'
686 output: ['["json"]']
687
688 - title: "Multiplication, division, modulo: `*`, `/`, and `%`"
689 body: |
690
691 These infix operators behave as expected when given two numbers.
692 Division by zero raises an error. `x % y` computes x modulo y.
693
694 Multiplying a string by a number produces the concatenation of
695 that string that many times. `"x" * 0` produces **null**.
696
697 Dividing a string by another splits the first using the second
698 as separators.
699
700 Multiplying two objects will merge them recursively: this works
701 like addition but if both objects contain a value for the
702 same key, and the values are objects, the two are merged with
703 the same strategy.
704
705 examples:
706 - program: '10 / . * 3'
707 input: 5
708 output: [6]
709 - program: '. / ", "'
710 input: '"a, b,c,d, e"'
711 output: ['["a","b,c,d","e"]']
712 - program: '{"k": {"a": 1, "b": 2}} * {"k": {"a": 0,"c": 3}}'
713 input: 'null'
714 output: ['{"k": {"a": 0, "b": 2, "c": 3}}']
715 - program: '.[] | (1 / .)?'
716 input: '[1,0,-1]'
717 output: ['1', '-1']
718
719
720 - title: "`length`"
721 body: |
722
723 The builtin function `length` gets the length of various
724 different types of value:
725
726 - The length of a **string** is the number of Unicode
727 codepoints it contains (which will be the same as its
728 JSON-encoded length in bytes if it's pure ASCII).
729
730 - The length of an **array** is the number of elements.
731
732 - The length of an **object** is the number of key-value pairs.
733
734 - The length of **null** is zero.
735
736 examples:
737 - program: '.[] | length'
738 input: '[[1,2], "string", {"a":2}, null]'
739 output: [2, 6, 1, 0]
740
741
742 - title: "`utf8bytelength`"
743 body: |
744
745 The builtin function `utf8bytelength` outputs the number of
746 bytes used to encode a string in UTF-8.
747
748 examples:
749 - program: 'utf8bytelength'
750 input: '"\u03bc"'
751 output: [2]
752
753 - title: "`keys`, `keys_unsorted`"
754 body: |
755
756 The builtin function `keys`, when given an object, returns
757 its keys in an array.
758
759 The keys are sorted "alphabetically", by unicode codepoint
760 order. This is not an order that makes particular sense in
761 any particular language, but you can count on it being the
762 same for any two objects with the same set of keys,
763 regardless of locale settings.
764
765 When `keys` is given an array, it returns the valid indices
766 for that array: the integers from 0 to length-1.
767
768 The `keys_unsorted` function is just like `keys`, but if
769 the input is an object then the keys will not be sorted,
770 instead the keys will roughly be in insertion order.
771
772 examples:
773 - program: 'keys'
774 input: '{"abc": 1, "abcd": 2, "Foo": 3}'
775 output: ['["Foo", "abc", "abcd"]']
776 - program: 'keys'
777 input: '[42,3,35]'
778 output: ['[0,1,2]']
779
780 - title: "`has(key)`"
781 body: |
782
783 The builtin function `has` returns whether the input object
784 has the given key, or the input array has an element at the
785 given index.
786
787 `has($key)` has the same effect as checking whether `$key`
788 is a member of the array returned by `keys`, although `has`
789 will be faster.
790
791 examples:
792 - program: 'map(has("foo"))'
793 input: '[{"foo": 42}, {}]'
794 output: ['[true, false]']
795 - program: 'map(has(2))'
796 input: '[[0,1], ["a","b","c"]]'
797 output: ['[false, true]']
798
799 - title: "`in`"
800 body: |
801
802 The builtin function `in` returns whether or not the input key is in the
803 given object, or the input index corresponds to an element
804 in the given array. It is, essentially, an inversed version
805 of `has`.
806
807 examples:
808 - program: '.[] | in({"foo": 42})'
809 input: '["foo", "bar"]'
810 output: ['true', 'false']
811 - program: 'map(in([0,1]))'
812 input: '[2, 0]'
813 output: ['[false, true]']
814
815 - title: "`map(x)`, `map_values(x)`"
816 body: |
817
818 For any filter `x`, `map(x)` will run that filter for each
819 element of the input array, and return the outputs in a new
820 array. `map(.+1)` will increment each element of an array of numbers.
821
822 Similarly, `map_values(x)` will run that filter for each element,
823 but it will return an object when an object is passed.
824
825 `map(x)` is equivalent to `[.[] | x]`. In fact, this is how
826 it's defined. Similarly, `map_values(x)` is defined as `.[] |= x`.
827
828 examples:
829 - program: 'map(.+1)'
830 input: '[1,2,3]'
831 output: ['[2,3,4]']
832
833 - program: 'map_values(.+1)'
834 input: '{"a": 1, "b": 2, "c": 3}'
835 output: ['{"a": 2, "b": 3, "c": 4}']
836
837 - title: "`path(path_expression)`"
838 body: |
839
840 Outputs array representations of the given path expression
841 in `.`. The outputs are arrays of strings (object keys)
842 and/or numbers (array indices).
843
844 Path expressions are jq expressions like `.a`, but also `.[]`.
845 There are two types of path expressions: ones that can match
846 exactly, and ones that cannot. For example, `.a.b.c` is an
847 exact match path expression, while `.a[].b` is not.
848
849 `path(exact_path_expression)` will produce the array
850 representation of the path expression even if it does not
851 exist in `.`, if `.` is `null` or an array or an object.
852
853 `path(pattern)` will produce array representations of the
854 paths matching `pattern` if the paths exist in `.`.
855
856 Note that the path expressions are not different from normal
857 expressions. The expression
858 `path(..|select(type=="boolean"))` outputs all the paths to
859 boolean values in `.`, and only those paths.
860
861 examples:
862 - program: 'path(.a[0].b)'
863 input: 'null'
864 output: ['["a",0,"b"]']
865 - program: '[path(..)]'
866 input: '{"a":[{"b":1}]}'
867 output: ['[[],["a"],["a",0],["a",0,"b"]]']
868
869 - title: "`del(path_expression)`"
870 body: |
871
872 The builtin function `del` removes a key and its corresponding
873 value from an object.
874
875 examples:
876 - program: 'del(.foo)'
877 input: '{"foo": 42, "bar": 9001, "baz": 42}'
878 output: ['{"bar": 9001, "baz": 42}']
879 - program: 'del(.[1, 2])'
880 input: '["foo", "bar", "baz"]'
881 output: ['["foo"]']
882
883 - title: "`getpath(PATHS)`"
884 body: |
885
886 The builtin function `getpath` outputs the values in `.` found
887 at each path in `PATHS`.
888
889 examples:
890 - program: 'getpath(["a","b"])'
891 input: 'null'
892 output: ['null']
893 - program: '[getpath(["a","b"], ["a","c"])]'
894 input: '{"a":{"b":0, "c":1}}'
895 output: ['[0, 1]']
896
897 - title: "`setpath(PATHS; VALUE)`"
898 body: |
899
900 The builtin function `setpath` sets the `PATHS` in `.` to `VALUE`.
901
902 examples:
903 - program: 'setpath(["a","b"]; 1)'
904 input: 'null'
905 output: ['{"a": {"b": 1}}']
906 - program: 'setpath(["a","b"]; 1)'
907 input: '{"a":{"b":0}}'
908 output: ['{"a": {"b": 1}}']
909 - program: 'setpath([0,"a"]; 1)'
910 input: 'null'
911 output: ['[{"a":1}]']
912
913 - title: "`delpaths(PATHS)`"
914 body: |
915
916 The builtin function `delpaths` sets the `PATHS` in `.`.
917 `PATHS` must be an array of paths, where each path is an array
918 of strings and numbers.
919
920 examples:
921 - program: 'delpaths([["a","b"]])'
922 input: '{"a":{"b":1},"x":{"y":2}}'
923 output: ['{"a":{},"x":{"y":2}}']
924
925 - title: "`to_entries`, `from_entries`, `with_entries`"
926 body: |
927
928 These functions convert between an object and an array of
929 key-value pairs. If `to_entries` is passed an object, then
930 for each `k: v` entry in the input, the output array
931 includes `{"key": k, "value": v}`.
932
933 `from_entries` does the opposite conversion, and
934 `with_entries(foo)` is a shorthand for `to_entries |
935 map(foo) | from_entries`, useful for doing some operation to
936 all keys and values of an object. `from_entries` accepts key, Key,
937 name, Name, value and Value as keys.
938
939 examples:
940 - program: 'to_entries'
941 input: '{"a": 1, "b": 2}'
942 output: ['[{"key":"a", "value":1}, {"key":"b", "value":2}]']
943 - program: 'from_entries'
944 input: '[{"key":"a", "value":1}, {"key":"b", "value":2}]'
945 output: ['{"a": 1, "b": 2}']
946 - program: 'with_entries(.key |= "KEY_" + .)'
947 input: '{"a": 1, "b": 2}'
948 output: ['{"KEY_a": 1, "KEY_b": 2}']
949
950
951 - title: "`select(boolean_expression)`"
952 body: |
953
954 The function `select(foo)` produces its input unchanged if
955 `foo` returns true for that input, and produces no output
956 otherwise.
957
958 It's useful for filtering lists: `[1,2,3] | map(select(. >= 2))`
959 will give you `[2,3]`.
960
961 examples:
962 - program: 'map(select(. >= 2))'
963 input: '[1,5,3,0,7]'
964 output: ['[5,3,7]']
965 - program: '.[] | select(.id == "second")'
966 input: '[{"id": "first", "val": 1}, {"id": "second", "val": 2}]'
967 output: ['{"id": "second", "val": 2}']
968
969
970 - title: "`arrays`, `objects`, `iterables`, `booleans`, `numbers`, `normals`, `finites`, `strings`, `nulls`, `values`, `scalars`"
971 body: |
972
973 These built-ins select only inputs that are arrays, objects,
974 iterables (arrays or objects), booleans, numbers, normal
975 numbers, finite numbers, strings, null, non-null values, and
976 non-iterables, respectively.
977
978 examples:
979 - program: '.[]|numbers'
980 input: '[[],{},1,"foo",null,true,false]'
981 output: ['1']
982
983 - title: "`empty`"
984 body: |
985
986 `empty` returns no results. None at all. Not even `null`.
987
988 It's useful on occasion. You'll know if you need it :)
989
990 examples:
991 - program: '1, empty, 2'
992 input: 'null'
993 output: [1, 2]
994 - program: '[1,2,empty,3]'
995 input: 'null'
996 output: ['[1,2,3]']
997
998 - title: "`error(message)`"
999 body: |
1000
1001 Produces an error, just like `.a` applied to values other than
1002 null and objects would, but with the given message as the
1003 error's value. Errors can be caught with try/catch; see below.
1004
1005 - title: "`halt`"
1006 body: |
1007
1008 Stops the jq program with no further outputs. jq will exit
1009 with exit status `0`.
1010
1011 - title: "`halt_error`, `halt_error(exit_code)`"
1012 body: |
1013
1014 Stops the jq program with no further outputs. The input will
1015 be printed on `stderr` as raw output (i.e., strings will not
1016 have double quotes) with no decoration, not even a newline.
1017
1018 The given `exit_code` (defaulting to `5`) will be jq's exit
1019 status.
1020
1021 For example, `"Error: somthing went wrong\n"|halt_error(1)`.
1022
1023 - title: "`$__loc__`"
1024 body: |
1025
1026 Produces an object with a "file" key and a "line" key, with
1027 the filename and line number where `$__loc__` occurs, as
1028 values.
1029
1030 examples:
1031 - program: 'try error("\($__loc__)") catch .'
1032 input: 'null'
1033 output: ['"{\"file\":\"<top-level>\",\"line\":1}"']
1034
1035 - title: "`paths`, `paths(node_filter)`, `leaf_paths`"
1036 body: |
1037
1038 `paths` outputs the paths to all the elements in its input
1039 (except it does not output the empty list, representing .
1040 itself).
1041
1042 `paths(f)` outputs the paths to any values for which `f` is true.
1043 That is, `paths(numbers)` outputs the paths to all numeric
1044 values.
1045
1046 `leaf_paths` is an alias of `paths(scalars)`; `leaf_paths` is
1047 *deprecated* and will be removed in the next major release.
1048
1049 examples:
1050 - program: '[paths]'
1051 input: '[1,[[],{"a":2}]]'
1052 output: ['[[0],[1],[1,0],[1,1],[1,1,"a"]]']
1053 - program: '[paths(scalars)]'
1054 input: '[1,[[],{"a":2}]]'
1055 output: ['[[0],[1,1,"a"]]']
1056
1057 - title: "`add`"
1058 body: |
1059
1060 The filter `add` takes as input an array, and produces as
1061 output the elements of the array added together. This might
1062 mean summed, concatenated or merged depending on the types
1063 of the elements of the input array - the rules are the same
1064 as those for the `+` operator (described above).
1065
1066 If the input is an empty array, `add` returns `null`.
1067
1068 examples:
1069 - program: add
1070 input: '["a","b","c"]'
1071 output: ['"abc"']
1072 - program: add
1073 input: '[1, 2, 3]'
1074 output: [6]
1075 - program: add
1076 input: '[]'
1077 output: ["null"]
1078
1079 - title: "`any`, `any(condition)`, `any(generator; condition)`"
1080 body: |
1081
1082 The filter `any` takes as input an array of boolean values,
1083 and produces `true` as output if any of the elements of
1084 the array are `true`.
1085
1086 If the input is an empty array, `any` returns `false`.
1087
1088 The `any(condition)` form applies the given condition to the
1089 elements of the input array.
1090
1091 The `any(generator; condition)` form applies the given
1092 condition to all the outputs of the given generator.
1093
1094 examples:
1095 - program: any
1096 input: '[true, false]'
1097 output: ["true"]
1098 - program: any
1099 input: '[false, false]'
1100 output: ["false"]
1101 - program: any
1102 input: '[]'
1103 output: ["false"]
1104
1105 - title: "`all`, `all(condition)`, `all(generator; condition)`"
1106 body: |
1107
1108 The filter `all` takes as input an array of boolean values,
1109 and produces `true` as output if all of the elements of
1110 the array are `true`.
1111
1112 The `all(condition)` form applies the given condition to the
1113 elements of the input array.
1114
1115 The `all(generator; condition)` form applies the given
1116 condition to all the outputs of the given generator.
1117
1118 If the input is an empty array, `all` returns `true`.
1119
1120 examples:
1121 - program: all
1122 input: '[true, false]'
1123 output: ["false"]
1124 - program: all
1125 input: '[true, true]'
1126 output: ["true"]
1127 - program: all
1128 input: '[]'
1129 output: ["true"]
1130
1131 - title: "`flatten`, `flatten(depth)`"
1132 body: |
1133
1134 The filter `flatten` takes as input an array of nested arrays,
1135 and produces a flat array in which all arrays inside the original
1136 array have been recursively replaced by their values. You can pass
1137 an argument to it to specify how many levels of nesting to flatten.
1138
1139 `flatten(2)` is like `flatten`, but going only up to two
1140 levels deep.
1141
1142 examples:
1143 - program: flatten
1144 input: '[1, [2], [[3]]]'
1145 output: ["[1, 2, 3]"]
1146 - program: flatten(1)
1147 input: '[1, [2], [[3]]]'
1148 output: ["[1, 2, [3]]"]
1149 - program: flatten
1150 input: '[[]]'
1151 output: ["[]"]
1152 - program: flatten
1153 input: '[{"foo": "bar"}, [{"foo": "baz"}]]'
1154 output: ['[{"foo": "bar"}, {"foo": "baz"}]']
1155
1156 - title: "`range(upto)`, `range(from;upto)` `range(from;upto;by)`"
1157 body: |
1158
1159 The `range` function produces a range of numbers. `range(4;10)`
1160 produces 6 numbers, from 4 (inclusive) to 10 (exclusive). The numbers
1161 are produced as separate outputs. Use `[range(4;10)]` to get a range as
1162 an array.
1163
1164 The one argument form generates numbers from 0 to the given
1165 number, with an increment of 1.
1166
1167 The two argument form generates numbers from `from` to `upto`
1168 with an increment of 1.
1169
1170 The three argument form generates numbers `from` to `upto`
1171 with an increment of `by`.
1172
1173 examples:
1174 - program: 'range(2;4)'
1175 input: 'null'
1176 output: ['2', '3']
1177 - program: '[range(2;4)]'
1178 input: 'null'
1179 output: ['[2,3]']
1180 - program: '[range(4)]'
1181 input: 'null'
1182 output: ['[0,1,2,3]']
1183 - program: '[range(0;10;3)]'
1184 input: 'null'
1185 output: ['[0,3,6,9]']
1186 - program: '[range(0;10;-1)]'
1187 input: 'null'
1188 output: ['[]']
1189 - program: '[range(0;-5;-1)]'
1190 input: 'null'
1191 output: ['[0,-1,-2,-3,-4]']
1192
1193 - title: "`floor`"
1194 body: |
1195
1196 The `floor` function returns the floor of its numeric input.
1197
1198 examples:
1199 - program: 'floor'
1200 input: '3.14159'
1201 output: ['3']
1202
1203 - title: "`sqrt`"
1204 body: |
1205
1206 The `sqrt` function returns the square root of its numeric input.
1207
1208 examples:
1209 - program: 'sqrt'
1210 input: '9'
1211 output: ['3']
1212
1213 - title: "`tonumber`"
1214 body: |
1215
1216 The `tonumber` function parses its input as a number. It
1217 will convert correctly-formatted strings to their numeric
1218 equivalent, leave numbers alone, and give an error on all other input.
1219
1220 examples:
1221 - program: '.[] | tonumber'
1222 input: '[1, "1"]'
1223 output: [1, 1]
1224
1225 - title: "`tostring`"
1226 body: |
1227
1228 The `tostring` function prints its input as a
1229 string. Strings are left unchanged, and all other values are
1230 JSON-encoded.
1231
1232 examples:
1233 - program: '.[] | tostring'
1234 input: '[1, "1", [1]]'
1235 output: ['"1"', '"1"', '"[1]"']
1236
1237 - title: "`type`"
1238 body: |
1239
1240 The `type` function returns the type of its argument as a
1241 string, which is one of null, boolean, number, string, array
1242 or object.
1243
1244 examples:
1245 - program: 'map(type)'
1246 input: '[0, false, [], {}, null, "hello"]'
1247 output: ['["number", "boolean", "array", "object", "null", "string"]']
1248
1249 - title: "`infinite`, `nan`, `isinfinite`, `isnan`, `isfinite`, `isnormal`"
1250 body: |
1251
1252 Some arithmetic operations can yield infinities and "not a
1253 number" (NaN) values. The `isinfinite` builtin returns `true`
1254 if its input is infinite. The `isnan` builtin returns `true`
1255 if its input is a NaN. The `infinite` builtin returns a
1256 positive infinite value. The `nan` builtin returns a NaN.
1257 The `isnormal` builtin returns true if its input is a normal
1258 number.
1259
1260 Note that division by zero raises an error.
1261
1262 Currently most arithmetic operations operating on infinities,
1263 NaNs, and sub-normals do not raise errors.
1264
1265 examples:
1266 - program: '.[] | (infinite * .) < 0'
1267 input: '[-1, 1]'
1268 output: ['true', 'false']
1269 - program: 'infinite, nan | type'
1270 input: 'null'
1271 output: ['"number"', '"number"']
1272
1273 - title: "`sort, sort_by(path_expression)`"
1274 body: |
1275
1276 The `sort` functions sorts its input, which must be an
1277 array. Values are sorted in the following order:
1278
1279 * `null`
1280 * `false`
1281 * `true`
1282 * numbers
1283 * strings, in alphabetical order (by unicode codepoint value)
1284 * arrays, in lexical order
1285 * objects
1286
1287 The ordering for objects is a little complex: first they're
1288 compared by comparing their sets of keys (as arrays in
1289 sorted order), and if their keys are equal then the values
1290 are compared key by key.
1291
1292 `sort` may be used to sort by a particular field of an
1293 object, or by applying any jq filter.
1294
1295 `sort_by(foo)` compares two elements by comparing the result of
1296 `foo` on each element.
1297
1298 examples:
1299 - program: 'sort'
1300 input: '[8,3,null,6]'
1301 output: ['[null,3,6,8]']
1302 - program: 'sort_by(.foo)'
1303 input: '[{"foo":4, "bar":10}, {"foo":3, "bar":100}, {"foo":2, "bar":1}]'
1304 output: ['[{"foo":2, "bar":1}, {"foo":3, "bar":100}, {"foo":4, "bar":10}]']
1305
1306 - title: "`group_by(path_expression)`"
1307 body: |
1308
1309 `group_by(.foo)` takes as input an array, groups the
1310 elements having the same `.foo` field into separate arrays,
1311 and produces all of these arrays as elements of a larger
1312 array, sorted by the value of the `.foo` field.
1313
1314 Any jq expression, not just a field access, may be used in
1315 place of `.foo`. The sorting order is the same as described
1316 in the `sort` function above.
1317
1318 examples:
1319 - program: 'group_by(.foo)'
1320 input: '[{"foo":1, "bar":10}, {"foo":3, "bar":100}, {"foo":1, "bar":1}]'
1321 output: ['[[{"foo":1, "bar":10}, {"foo":1, "bar":1}], [{"foo":3, "bar":100}]]']
1322
1323 - title: "`min`, `max`, `min_by(path_exp)`, `max_by(path_exp)`"
1324 body: |
1325
1326 Find the minimum or maximum element of the input array.
1327
1328 The `min_by(path_exp)` and `max_by(path_exp)` functions allow
1329 you to specify a particular field or property to examine, e.g.
1330 `min_by(.foo)` finds the object with the smallest `foo` field.
1331
1332 examples:
1333 - program: 'min'
1334 input: '[5,4,2,7]'
1335 output: ['2']
1336 - program: 'max_by(.foo)'
1337 input: '[{"foo":1, "bar":14}, {"foo":2, "bar":3}]'
1338 output: ['{"foo":2, "bar":3}']
1339
1340 - title: "`unique`, `unique_by(path_exp)`"
1341 body: |
1342
1343 The `unique` function takes as input an array and produces
1344 an array of the same elements, in sorted order, with
1345 duplicates removed.
1346
1347 The `unique_by(path_exp)` function will keep only one element
1348 for each value obtained by applying the argument. Think of it
1349 as making an array by taking one element out of every group
1350 produced by `group`.
1351
1352 examples:
1353 - program: 'unique'
1354 input: '[1,2,5,3,5,3,1,3]'
1355 output: ['[1,2,3,5]']
1356 - program: 'unique_by(.foo)'
1357 input: '[{"foo": 1, "bar": 2}, {"foo": 1, "bar": 3}, {"foo": 4, "bar": 5}]'
1358 output: ['[{"foo": 1, "bar": 2}, {"foo": 4, "bar": 5}]']
1359 - program: 'unique_by(length)'
1360 input: '["chunky", "bacon", "kitten", "cicada", "asparagus"]'
1361 output: ['["bacon", "chunky", "asparagus"]']
1362
1363 - title: "`reverse`"
1364 body: |
1365
1366 This function reverses an array.
1367
1368 examples:
1369 - program: 'reverse'
1370 input: '[1,2,3,4]'
1371 output: ['[4,3,2,1]']
1372
1373 - title: "`contains(element)`"
1374 body: |
1375
1376 The filter `contains(b)` will produce true if b is
1377 completely contained within the input. A string B is
1378 contained in a string A if B is a substring of A. An array B
1379 is contained in an array A if all elements in B are
1380 contained in any element in A. An object B is contained in
1381 object A if all of the values in B are contained in the
1382 value in A with the same key. All other types are assumed to
1383 be contained in each other if they are equal.
1384
1385 examples:
1386 - program: 'contains("bar")'
1387 input: '"foobar"'
1388 output: ['true']
1389 - program: 'contains(["baz", "bar"])'
1390 input: '["foobar", "foobaz", "blarp"]'
1391 output: ['true']
1392 - program: 'contains(["bazzzzz", "bar"])'
1393 input: '["foobar", "foobaz", "blarp"]'
1394 output: ['false']
1395 - program: 'contains({foo: 12, bar: [{barp: 12}]})'
1396 input: '{"foo": 12, "bar":[1,2,{"barp":12, "blip":13}]}'
1397 output: ['true']
1398 - program: 'contains({foo: 12, bar: [{barp: 15}]})'
1399 input: '{"foo": 12, "bar":[1,2,{"barp":12, "blip":13}]}'
1400 output: ['false']
1401
1402 - title: "`indices(s)`"
1403 body: |
1404
1405 Outputs an array containing the indices in `.` where `s`
1406 occurs. The input may be an array, in which case if `s` is an
1407 array then the indices output will be those where all elements
1408 in `.` match those of `s`.
1409
1410 examples:
1411 - program: 'indices(", ")'
1412 input: '"a,b, cd, efg, hijk"'
1413 output: ['[3,7,12]']
1414 - program: 'indices(1)'
1415 input: '[0,1,2,1,3,1,4]'
1416 output: ['[1,3,5]']
1417 - program: 'indices([1,2])'
1418 input: '[0,1,2,3,1,4,2,5,1,2,6,7]'
1419 output: ['[1,8]']
1420
1421 - title: "`index(s)`, `rindex(s)`"
1422 body: |
1423
1424 Outputs the index of the first (`index`) or last (`rindex`)
1425 occurrence of `s` in the input.
1426
1427 examples:
1428 - program: 'index(", ")'
1429 input: '"a,b, cd, efg, hijk"'
1430 output: ['3']
1431 - program: 'rindex(", ")'
1432 input: '"a,b, cd, efg, hijk"'
1433 output: ['12']
1434
1435 - title: "`inside`"
1436 body: |
1437
1438 The filter `inside(b)` will produce true if the input is
1439 completely contained within b. It is, essentially, an
1440 inversed version of `contains`.
1441
1442 examples:
1443 - program: 'inside("foobar")'
1444 input: '"bar"'
1445 output: ['true']
1446 - program: 'inside(["foobar", "foobaz", "blarp"])'
1447 input: '["baz", "bar"]'
1448 output: ['true']
1449 - program: 'inside(["foobar", "foobaz", "blarp"])'
1450 input: '["bazzzzz", "bar"]'
1451 output: ['false']
1452 - program: 'inside({"foo": 12, "bar":[1,2,{"barp":12, "blip":13}]})'
1453 input: '{"foo": 12, "bar": [{"barp": 12}]}'
1454 output: ['true']
1455 - program: 'inside({"foo": 12, "bar":[1,2,{"barp":12, "blip":13}]})'
1456 input: '{"foo": 12, "bar": [{"barp": 15}]}'
1457 output: ['false']
1458
1459 - title: "`startswith(str)`"
1460 body: |
1461
1462 Outputs `true` if . starts with the given string argument.
1463
1464 examples:
1465 - program: '[.[]|startswith("foo")]'
1466 input: '["fo", "foo", "barfoo", "foobar", "barfoob"]'
1467 output: ['[false, true, false, true, false]']
1468
1469 - title: "`endswith(str)`"
1470 body: |
1471
1472 Outputs `true` if . ends with the given string argument.
1473
1474 examples:
1475 - program: '[.[]|endswith("foo")]'
1476 input: '["foobar", "barfoo"]'
1477 output: ['[false, true]']
1478
1479 - title: "`combinations`, `combinations(n)`"
1480 body: |
1481
1482 Outputs all combinations of the elements of the arrays in the
1483 input array. If given an argument `n`, it outputs all combinations
1484 of `n` repetitions of the input array.
1485
1486 examples:
1487 - program: 'combinations'
1488 input: '[[1,2], [3, 4]]'
1489 output: ['[1, 3]', '[1, 4]', '[2, 3]', '[2, 4]']
1490 - program: 'combinations(2)'
1491 input: '[0, 1]'
1492 output: ['[0, 0]', '[0, 1]', '[1, 0]', '[1, 1]']
1493
1494 - title: "`ltrimstr(str)`"
1495 body: |
1496
1497 Outputs its input with the given prefix string removed, if it
1498 starts with it.
1499
1500 examples:
1501 - program: '[.[]|ltrimstr("foo")]'
1502 input: '["fo", "foo", "barfoo", "foobar", "afoo"]'
1503 output: ['["fo","","barfoo","bar","afoo"]']
1504
1505 - title: "`rtrimstr(str)`"
1506 body: |
1507
1508 Outputs its input with the given suffix string removed, if it
1509 ends with it.
1510
1511 examples:
1512 - program: '[.[]|rtrimstr("foo")]'
1513 input: '["fo", "foo", "barfoo", "foobar", "foob"]'
1514 output: ['["fo","","bar","foobar","foob"]']
1515
1516 - title: "`explode`"
1517 body: |
1518
1519 Converts an input string into an array of the string's
1520 codepoint numbers.
1521
1522 examples:
1523 - program: 'explode'
1524 input: '"foobar"'
1525 output: ['[102,111,111,98,97,114]']
1526
1527 - title: "`implode`"
1528 body: |
1529
1530 The inverse of explode.
1531
1532 examples:
1533 - program: 'implode'
1534 input: '[65, 66, 67]'
1535 output: ['"ABC"']
1536
1537 - title: "`split(str)`"
1538 body: |
1539
1540 Splits an input string on the separator argument.
1541
1542 examples:
1543 - program: 'split(", ")'
1544 input: '"a, b,c,d, e, "'
1545 output: ['["a","b,c,d","e",""]']
1546
1547 - title: "`join(str)`"
1548 body: |
1549
1550 Joins the array of elements given as input, using the
1551 argument as separator. It is the inverse of `split`: that is,
1552 running `split("foo") | join("foo")` over any input string
1553 returns said input string.
1554
1555 Numbers and booleans in the input are converted to strings.
1556 Null values are treated as empty strings. Arrays and objects
1557 in the input are not supported.
1558
1559 examples:
1560 - program: 'join(", ")'
1561 input: '["a","b,c,d","e"]'
1562 output: ['"a, b,c,d, e"']
1563 - program: 'join(" ")'
1564 input: '["a",1,2.3,true,null,false]'
1565 output: ['"a 1 2.3 true false"']
1566
1567 - title: "`ascii_downcase`, `ascii_upcase`"
1568 body: |
1569
1570 Emit a copy of the input string with its alphabetic characters (a-z and A-Z)
1571 converted to the specified case.
1572
1573 example:
1574 - program: 'ascii_upcase'
1575 input: '"useful but not for é"'
1576 output: '"USEFUL BUT NOT FOR é"'
1577
1578 - title: "`while(cond; update)`"
1579 body: |
1580
1581 The `while(cond; update)` function allows you to repeatedly
1582 apply an update to `.` until `cond` is false.
1583
1584 Note that `while(cond; update)` is internally defined as a
1585 recursive jq function. Recursive calls within `while` will
1586 not consume additional memory if `update` produces at most one
1587 output for each input. See advanced topics below.
1588
1589 examples:
1590 - program: '[while(.<100; .*2)]'
1591 input: '1'
1592 output: ['[1,2,4,8,16,32,64]']
1593
1594 - title: "`until(cond; next)`"
1595 body: |
1596
1597 The `until(cond; next)` function allows you to repeatedly
1598 apply the expression `next`, initially to `.` then to its own
1599 output, until `cond` is true. For example, this can be used
1600 to implement a factorial function (see below).
1601
1602 Note that `until(cond; next)` is internally defined as a
1603 recursive jq function. Recursive calls within `until()` will
1604 not consume additional memory if `next` produces at most one
1605 output for each input. See advanced topics below.
1606
1607 examples:
1608 - program: '[.,1]|until(.[0] < 1; [.[0] - 1, .[1] * .[0]])|.[1]'
1609 input: '4'
1610 output: ['24']
1611
1612
1613 - title: "`recurse(f)`, `recurse`, `recurse(f; condition)`, `recurse_down`"
1614 body: |
1615
1616 The `recurse(f)` function allows you to search through a
1617 recursive structure, and extract interesting data from all
1618 levels. Suppose your input represents a filesystem:
1619
1620 {"name": "/", "children": [
1621 {"name": "/bin", "children": [
1622 {"name": "/bin/ls", "children": []},
1623 {"name": "/bin/sh", "children": []}]},
1624 {"name": "/home", "children": [
1625 {"name": "/home/stephen", "children": [
1626 {"name": "/home/stephen/jq", "children": []}]}]}]}
1627
1628 Now suppose you want to extract all of the filenames
1629 present. You need to retrieve `.name`, `.children[].name`,
1630 `.children[].children[].name`, and so on. You can do this
1631 with:
1632
1633 recurse(.children[]) | .name
1634
1635 When called without an argument, `recurse` is equivalent to
1636 `recurse(.[]?)`.
1637
1638 `recurse(f)` is identical to `recurse(f; . != null)` and can be
1639 used without concerns about recursion depth.
1640
1641 `recurse(f; condition)` is a generator which begins by
1642 emitting . and then emits in turn .|f, .|f|f, .|f|f|f, ... so long
1643 as the computed value satisfies the condition. For example,
1644 to generate all the integers, at least in principle, one
1645 could write `recurse(.+1; true)`.
1646
1647 For legacy reasons, `recurse_down` exists as an alias to
1648 calling `recurse` without arguments. This alias is considered
1649 *deprecated* and will be removed in the next major release.
1650
1651 The recursive calls in `recurse` will not consume additional
1652 memory whenever `f` produces at most a single output for each
1653 input.
1654
1655 examples:
1656 - program: 'recurse(.foo[])'
1657 input: '{"foo":[{"foo": []}, {"foo":[{"foo":[]}]}]}'
1658 output:
1659 - '{"foo":[{"foo":[]},{"foo":[{"foo":[]}]}]}'
1660 - '{"foo":[]}'
1661 - '{"foo":[{"foo":[]}]}'
1662 - '{"foo":[]}'
1663
1664 - program: 'recurse'
1665 input: '{"a":0,"b":[1]}'
1666 output:
1667 - '{"a":0,"b":[1]}'
1668 - '0'
1669 - '[1]'
1670 - '1'
1671
1672 - program: 'recurse(. * .; . < 20)'
1673 input: 2
1674 output:
1675 - 2
1676 - 4
1677 - 16
1678
1679 - title: "`walk(f)`"
1680 body: |
1681
1682 The `walk(f)` function applies f recursively to every
1683 component of the input entity. When an array is
1684 encountered, f is first applied to its elements and then to
1685 the array itself; when an object is encountered, f is first
1686 applied to all the values and then to the object. In
1687 practice, f will usually test the type of its input, as
1688 illustrated in the following examples. The first example
1689 highlights the usefulness of processing the elements of an
1690 array of arrays before processing the array itself. The second
1691 example shows how all the keys of all the objects within the
1692 input can be considered for alteration.
1693
1694 examples:
1695 - program: 'walk(if type == "array" then sort else . end)'
1696 input: '[[4, 1, 7], [8, 5, 2], [3, 6, 9]]'
1697 output:
1698 - '[[1,4,7],[2,5,8],[3,6,9]]'
1699
1700 - program: 'walk( if type == "object" then with_entries( .key |= sub( "^_+"; "") ) else . end )'
1701 input: '[ { "_a": { "__b": 2 } } ]'
1702 output:
1703 - '[{"a":{"b":2}}]'
1704
1705 - title: "`$ENV`, `env`"
1706 body: |
1707
1708 `$ENV` is an object representing the environment variables as
1709 set when the jq program started.
1710
1711 `env` outputs an object representing jq's current environment.
1712
1713 At the moment there is no builtin for setting environment
1714 variables.
1715
1716 examples:
1717 - program: '$ENV.PAGER'
1718 input: 'null'
1719 output: ['"less"']
1720
1721 - program: 'env.PAGER'
1722 input: 'null'
1723 output: ['"less"']
1724
1725 - title: "`transpose`"
1726 body: |
1727
1728 Transpose a possibly jagged matrix (an array of arrays).
1729 Rows are padded with nulls so the result is always rectangular.
1730
1731 examples:
1732 - program: 'transpose'
1733 input: '[[1], [2,3]]'
1734 output: ['[[1,2],[null,3]]']
1735
1736 - title: "`bsearch(x)`"
1737 body: |
1738
1739 bsearch(x) conducts a binary search for x in the input
1740 array. If the input is sorted and contains x, then
1741 bsearch(x) will return its index in the array; otherwise, if
1742 the array is sorted, it will return (-1 - ix) where ix is an
1743 insertion point such that the array would still be sorted
1744 after the insertion of x at ix. If the array is not sorted,
1745 bsearch(x) will return an integer that is probably of no
1746 interest.
1747
1748 examples:
1749 - program: 'bsearch(0)'
1750 input: '[0,1]'
1751 output: ['0']
1752 - program: 'bsearch(0)'
1753 input: '[1,2,3]'
1754 output: ['-1']
1755 - program: 'bsearch(4) as $ix | if $ix < 0 then .[-(1+$ix)] = 4 else . end'
1756 input: '[1,2,3]'
1757 output: ['[1,2,3,4]']
1758
1759 - title: "String interpolation - `\\(foo)`"
1760 body: |
1761
1762 Inside a string, you can put an expression inside parens
1763 after a backslash. Whatever the expression returns will be
1764 interpolated into the string.
1765
1766 examples:
1767 - program: '"The input was \(.), which is one less than \(.+1)"'
1768 input: '42'
1769 output: ['"The input was 42, which is one less than 43"']
1770
1771 - title: "Convert to/from JSON"
1772 body: |
1773
1774 The `tojson` and `fromjson` builtins dump values as JSON texts
1775 or parse JSON texts into values, respectively. The tojson
1776 builtin differs from tostring in that tostring returns strings
1777 unmodified, while tojson encodes strings as JSON strings.
1778
1779 examples:
1780 - program: '[.[]|tostring]'
1781 input: '[1, "foo", ["foo"]]'
1782 output: ['["1","foo","[\"foo\"]"]']
1783 - program: '[.[]|tojson]'
1784 input: '[1, "foo", ["foo"]]'
1785 output: ['["1","\"foo\"","[\"foo\"]"]']
1786 - program: '[.[]|tojson|fromjson]'
1787 input: '[1, "foo", ["foo"]]'
1788 output: ['[1,"foo",["foo"]]']
1789
1790 - title: "Format strings and escaping"
1791 body: |
1792
1793 The `@foo` syntax is used to format and escape strings,
1794 which is useful for building URLs, documents in a language
1795 like HTML or XML, and so forth. `@foo` can be used as a
1796 filter on its own, the possible escapings are:
1797
1798 * `@text`:
1799
1800 Calls `tostring`, see that function for details.
1801
1802 * `@json`:
1803
1804 Serializes the input as JSON.
1805
1806 * `@html`:
1807
1808 Applies HTML/XML escaping, by mapping the characters
1809 `<>&'"` to their entity equivalents `<`, `>`,
1810 `&`, `'`, `"`.
1811
1812 * `@uri`:
1813
1814 Applies percent-encoding, by mapping all reserved URI
1815 characters to a `%XX` sequence.
1816
1817 * `@csv`:
1818
1819 The input must be an array, and it is rendered as CSV
1820 with double quotes for strings, and quotes escaped by
1821 repetition.
1822
1823 * `@tsv`:
1824
1825 The input must be an array, and it is rendered as TSV
1826 (tab-separated values). Each input array will be printed as
1827 a single line. Fields are separated by a single
1828 tab (ascii `0x09`). Input characters line-feed (ascii `0x0a`),
1829 carriage-return (ascii `0x0d`), tab (ascii `0x09`) and
1830 backslash (ascii `0x5c`) will be output as escape sequences
1831 `\n`, `\r`, `\t`, `\\` respectively.
1832
1833 * `@sh`:
1834
1835 The input is escaped suitable for use in a command-line
1836 for a POSIX shell. If the input is an array, the output
1837 will be a series of space-separated strings.
1838
1839 * `@base64`:
1840
1841 The input is converted to base64 as specified by RFC 4648.
1842
1843 * `@base64d`:
1844
1845 The inverse of `@base64`, input is decoded as specified by RFC 4648.
1846 Note\: If the decoded string is not UTF-8, the results are undefined.
1847
1848 This syntax can be combined with string interpolation in a
1849 useful way. You can follow a `@foo` token with a string
1850 literal. The contents of the string literal will *not* be
1851 escaped. However, all interpolations made inside that string
1852 literal will be escaped. For instance,
1853
1854 @uri "https://www.google.com/search?q=\(.search)"
1855
1856 will produce the following output for the input
1857 `{"search":"what is jq?"}`:
1858
1859 "https://www.google.com/search?q=what%20is%20jq%3F"
1860
1861 Note that the slashes, question mark, etc. in the URL are
1862 not escaped, as they were part of the string literal.
1863
1864 examples:
1865 - program: '@html'
1866 input: '"This works if x < y"'
1867 output: ['"This works if x < y"']
1868
1869 # - program: '@html "<span>Anonymous said: \(.)</span>"'
1870 # input: '"<script>alert(\"lol hax\");</script>"'
1871 # output: ["<span>Anonymous said: <script>alert("lol hax");</script></span>"]
1872
1873 - program: '@sh "echo \(.)"'
1874 input: "\"O'Hara's Ale\""
1875 output: ["\"echo 'O'\\\\''Hara'\\\\''s Ale'\""]
1876
1877 - program: '@base64'
1878 input: '"This is a message"'
1879 output: ['"VGhpcyBpcyBhIG1lc3NhZ2U="']
1880
1881 - program: '@base64d'
1882 input: '"VGhpcyBpcyBhIG1lc3NhZ2U="'
1883 output: ['"This is a message"']
1884
1885 - title: "Dates"
1886 body: |
1887
1888 jq provides some basic date handling functionality, with some
1889 high-level and low-level builtins. In all cases these
1890 builtins deal exclusively with time in UTC.
1891
1892 The `fromdateiso8601` builtin parses datetimes in the ISO 8601
1893 format to a number of seconds since the Unix epoch
1894 (1970-01-01T00:00:00Z). The `todateiso8601` builtin does the
1895 inverse.
1896
1897 The `fromdate` builtin parses datetime strings. Currently
1898 `fromdate` only supports ISO 8601 datetime strings, but in the
1899 future it will attempt to parse datetime strings in more
1900 formats.
1901
1902 The `todate` builtin is an alias for `todateiso8601`.
1903
1904 The `now` builtin outputs the current time, in seconds since
1905 the Unix epoch.
1906
1907 Low-level jq interfaces to the C-library time functions are
1908 also provided: `strptime`, `strftime`, `strflocaltime`,
1909 `mktime`, `gmtime`, and `localtime`. Refer to your host
1910 operating system's documentation for the format strings used
1911 by `strptime` and `strftime`. Note: these are not necessarily
1912 stable interfaces in jq, particularly as to their localization
1913 functionality.
1914
1915 The `gmtime` builtin consumes a number of seconds since the
1916 Unix epoch and outputs a "broken down time" representation of
1917 Greenwhich Meridian time as an array of numbers representing
1918 (in this order): the year, the month (zero-based), the day of
1919 the month (one-based), the hour of the day, the minute of the
1920 hour, the second of the minute, the day of the week, and the
1921 day of the year -- all one-based unless otherwise stated. The
1922 day of the week number may be wrong on some systems for dates
1923 before March 1st 1900, or after December 31 2099.
1924
1925 The `localtime` builtin works like the `gmtime` builtin, but
1926 using the local timezone setting.
1927
1928 The `mktime` builtin consumes "broken down time"
1929 representations of time output by `gmtime` and `strptime`.
1930
1931 The `strptime(fmt)` builtin parses input strings matching the
1932 `fmt` argument. The output is in the "broken down time"
1933 representation consumed by `gmtime` and output by `mktime`.
1934
1935 The `strftime(fmt)` builtin formats a time (GMT) with the
1936 given format. The `strflocaltime` does the same, but using
1937 the local timezone setting.
1938
1939 The format strings for `strptime` and `strftime` are described
1940 in typical C library documentation. The format string for ISO
1941 8601 datetime is `"%Y-%m-%dT%H:%M:%SZ"`.
1942
1943 jq may not support some or all of this date functionality on
1944 some systems. In particular, the `%u` and `%j` specifiers for
1945 `strptime(fmt)` are not supported on macOS.
1946
1947 examples:
1948 - program: 'fromdate'
1949 input: '"2015-03-05T23:51:47Z"'
1950 output: ['1425599507']
1951
1952 - program: 'strptime("%Y-%m-%dT%H:%M:%SZ")'
1953 input: '"2015-03-05T23:51:47Z"'
1954 output: ['[2015,2,5,23,51,47,4,63]']
1955
1956 - program: 'strptime("%Y-%m-%dT%H:%M:%SZ")|mktime'
1957 input: '"2015-03-05T23:51:47Z"'
1958 output: ['1425599507']
1959
1960 - title: "SQL-Style Operators"
1961 body: |
1962
1963 jq provides a few SQL-style operators.
1964
1965 * INDEX(stream; index_expression):
1966
1967 This builtin produces an object whose keys are computed by
1968 the given index expression applied to each value from the
1969 given stream.
1970
1971 * JOIN($idx; stream; idx_expr; join_expr):
1972
1973 This builtin joins the values from the given stream to the
1974 given index. The index's keys are computed by applying the
1975 given index expression to each value from the given stream.
1976 An array of the value in the stream and the corresponding
1977 value from the index is fed to the given join expression to
1978 produce each result.
1979
1980 * JOIN($idx; stream; idx_expr):
1981
1982 Same as `JOIN($idx; stream; idx_expr; .)`.
1983
1984 * JOIN($idx; idx_expr):
1985
1986 This builtin joins the input `.` to the given index, applying
1987 the given index expression to `.` to compute the index key.
1988 The join operation is as described above.
1989
1990 * IN(s):
1991
1992 This builtin outputs `true` if `.` appears in the given
1993 stream, otherwise it outputs `false`.
1994
1995 * IN(source; s):
1996
1997 This builtin outputs `true` if any value in the source stream
1998 appears in the second stream, otherwise it outputs `false`.
1999
2000 - title: "`builtins`"
2001 body: |
2002
2003 Returns a list of all builtin functions in the format `name/arity`.
2004 Since functions with the same name but different arities are considered
2005 separate functions, `all/0`, `all/1`, and `all/2` would all be present
2006 in the list.
2007
2008 - title: Conditionals and Comparisons
2009 entries:
2010 - title: "`==`, `!=`"
2011 body: |
2012
2013 The expression 'a == b' will produce 'true' if the result of a and b
2014 are equal (that is, if they represent equivalent JSON documents) and
2015 'false' otherwise. In particular, strings are never considered equal
2016 to numbers. If you're coming from Javascript, jq's == is like
2017 Javascript's === - considering values equal only when they have the
2018 same type as well as the same value.
2019
2020 != is "not equal", and 'a != b' returns the opposite value of 'a == b'
2021
2022 examples:
2023 - program: '.[] == 1'
2024 input: '[1, 1.0, "1", "banana"]'
2025 output: ['true', 'true', 'false', 'false']
2026
2027 - title: if-then-else
2028 body: |
2029
2030 `if A then B else C end` will act the same as `B` if `A`
2031 produces a value other than false or null, but act the same
2032 as `C` otherwise.
2033
2034 Checking for false or null is a simpler notion of
2035 "truthiness" than is found in Javascript or Python, but it
2036 means that you'll sometimes have to be more explicit about
2037 the condition you want: you can't test whether, e.g. a
2038 string is empty using `if .name then A else B end`, you'll
2039 need something more like `if (.name | length) > 0 then A else
2040 B end` instead.
2041
2042 If the condition `A` produces multiple results, then `B` is evaluated
2043 once for each result that is not false or null, and `C` is evaluated
2044 once for each false or null.
2045
2046 More cases can be added to an if using `elif A then B` syntax.
2047
2048 examples:
2049 - program: |-
2050 if . == 0 then
2051 "zero"
2052 elif . == 1 then
2053 "one"
2054 else
2055 "many"
2056 end
2057 input: 2
2058 output: ['"many"']
2059
2060 - title: "`>, >=, <=, <`"
2061 body: |
2062
2063 The comparison operators `>`, `>=`, `<=`, `<` return whether
2064 their left argument is greater than, greater than or equal
2065 to, less than or equal to or less than their right argument
2066 (respectively).
2067
2068 The ordering is the same as that described for `sort`, above.
2069
2070 examples:
2071 - program: '. < 5'
2072 input: 2
2073 output: ['true']
2074
2075 - title: and/or/not
2076 body: |
2077
2078 jq supports the normal Boolean operators and/or/not. They have the
2079 same standard of truth as if expressions - false and null are
2080 considered "false values", and anything else is a "true value".
2081
2082 If an operand of one of these operators produces multiple
2083 results, the operator itself will produce a result for each input.
2084
2085 `not` is in fact a builtin function rather than an operator,
2086 so it is called as a filter to which things can be piped
2087 rather than with special syntax, as in `.foo and .bar |
2088 not`.
2089
2090 These three only produce the values "true" and "false", and
2091 so are only useful for genuine Boolean operations, rather
2092 than the common Perl/Python/Ruby idiom of
2093 "value_that_may_be_null or default". If you want to use this
2094 form of "or", picking between two values rather than
2095 evaluating a condition, see the "//" operator below.
2096
2097 examples:
2098 - program: '42 and "a string"'
2099 input: 'null'
2100 output: ['true']
2101 - program: '(true, false) or false'
2102 input: 'null'
2103 output: ['true', 'false']
2104 # - program: '(true, false) and (true, false)'
2105 # input: 'null'
2106 # output: ['true', 'false', 'false', 'false']
2107 - program: '(true, true) and (true, false)'
2108 input: 'null'
2109 output: ['true', 'false', 'true', 'false']
2110 - program: '[true, false | not]'
2111 input: 'null'
2112 output: ['[false, true]']
2113
2114 - title: "Alternative operator: `//`"
2115 body: |
2116
2117 A filter of the form `a // b` produces the same
2118 results as `a`, if `a` produces results other than `false`
2119 and `null`. Otherwise, `a // b` produces the same results as `b`.
2120
2121 This is useful for providing defaults: `.foo // 1` will
2122 evaluate to `1` if there's no `.foo` element in the
2123 input. It's similar to how `or` is sometimes used in Python
2124 (jq's `or` operator is reserved for strictly Boolean
2125 operations).
2126
2127 examples:
2128 - program: '.foo // 42'
2129 input: '{"foo": 19}'
2130 output: [19]
2131 - program: '.foo // 42'
2132 input: '{}'
2133 output: [42]
2134
2135 - title: try-catch
2136 body: |
2137
2138 Errors can be caught by using `try EXP catch EXP`. The first
2139 expression is executed, and if it fails then the second is
2140 executed with the error message. The output of the handler,
2141 if any, is output as if it had been the output of the
2142 expression to try.
2143
2144 The `try EXP` form uses `empty` as the exception handler.
2145
2146 examples:
2147 - program: 'try .a catch ". is not an object"'
2148 input: 'true'
2149 output: ['". is not an object"']
2150 - program: '[.[]|try .a]'
2151 input: '[{}, true, {"a":1}]'
2152 output: ['[null, 1]']
2153 - program: 'try error("some exception") catch .'
2154 input: 'true'
2155 output: ['"some exception"']
2156
2157 - title: Breaking out of control structures
2158 body: |
2159
2160 A convenient use of try/catch is to break out of control
2161 structures like `reduce`, `foreach`, `while`, and so on.
2162
2163 For example:
2164
2165 # Repeat an expression until it raises "break" as an
2166 # error, then stop repeating without re-raising the error.
2167 # But if the error caught is not "break" then re-raise it.
2168 try repeat(exp) catch .=="break" then empty else error;
2169
2170 jq has a syntax for named lexical labels to "break" or "go (back) to":
2171
2172 label $out | ... break $out ...
2173
2174 The `break $label_name` expression will cause the program to
2175 to act as though the nearest (to the left) `label $label_name`
2176 produced `empty`.
2177
2178 The relationship between the `break` and corresponding `label`
2179 is lexical: the label has to be "visible" from the break.
2180
2181 To break out of a `reduce`, for example:
2182
2183 label $out | reduce .[] as $item (null; if .==false then break $out else ... end)
2184
2185 The following jq program produces a syntax error:
2186
2187 break $out
2188
2189 because no label `$out` is visible.
2190
2191 - title: "Error Suppression / Optional Operator: `?`"
2192 body: |
2193
2194 The `?` operator, used as `EXP?`, is shorthand for `try EXP`.
2195
2196 examples:
2197 - program: '[.[]|(.a)?]'
2198 input: '[{}, true, {"a":1}]'
2199 output: ['[null, 1]']
2200
2201
2202 - title: Regular expressions (PCRE)
2203 body: |
2204
2205 jq uses the Oniguruma regular expression library, as do php,
2206 ruby, TextMate, Sublime Text, etc, so the description here
2207 will focus on jq specifics.
2208
2209 The jq regex filters are defined so that they can be used using
2210 one of these patterns:
2211
2212 STRING | FILTER( REGEX )
2213 STRING | FILTER( REGEX; FLAGS )
2214 STRING | FILTER( [REGEX] )
2215 STRING | FILTER( [REGEX, FLAGS] )
2216
2217 where:
2218 * STRING, REGEX and FLAGS are jq strings and subject to jq string interpolation;
2219 * REGEX, after string interpolation, should be a valid PCRE regex;
2220 * FILTER is one of `test`, `match`, or `capture`, as described below.
2221
2222 FLAGS is a string consisting of one of more of the supported flags:
2223
2224 * `g` - Global search (find all matches, not just the first)
2225 * `i` - Case insensitive search
2226 * `m` - Multi line mode ('.' will match newlines)
2227 * `n` - Ignore empty matches
2228 * `p` - Both s and m modes are enabled
2229 * `s` - Single line mode ('^' -> '\A', '$' -> '\Z')
2230 * `l` - Find longest possible matches
2231 * `x` - Extended regex format (ignore whitespace and comments)
2232
2233 To match whitespace in an x pattern use an escape such as \s, e.g.
2234
2235 * test( "a\\sb", "x" ).
2236
2237 Note that certain flags may also be specified within REGEX, e.g.
2238
2239 * jq -n '("test", "TEst", "teST", "TEST") | test( "(?i)te(?-i)st" )'
2240
2241 evaluates to: true, true, false, false.
2242
2243 entries:
2244 - title: "`test(val)`, `test(regex; flags)`"
2245 body: |
2246
2247 Like `match`, but does not return match objects, only `true` or `false`
2248 for whether or not the regex matches the input.
2249
2250 examples:
2251 - program: 'test("foo")'
2252 input: '"foo"'
2253 output: ['true']
2254 - program: '.[] | test("a b c # spaces are ignored"; "ix")'
2255 input: '["xabcd", "ABC"]'
2256 output: ['true', 'true']
2257
2258 - title: "`match(val)`, `match(regex; flags)`"
2259 body: |
2260
2261 **match** outputs an object for each match it finds. Matches have
2262 the following fields:
2263
2264 * `offset` - offset in UTF-8 codepoints from the beginning of the input
2265 * `length` - length in UTF-8 codepoints of the match
2266 * `string` - the string that it matched
2267 * `captures` - an array of objects representing capturing groups.
2268
2269 Capturing group objects have the following fields:
2270
2271 * `offset` - offset in UTF-8 codepoints from the beginning of the input
2272 * `length` - length in UTF-8 codepoints of this capturing group
2273 * `string` - the string that was captured
2274 * `name` - the name of the capturing group (or `null` if it was unnamed)
2275
2276 Capturing groups that did not match anything return an offset of -1
2277
2278 examples:
2279 - program: 'match("(abc)+"; "g")'
2280 input: '"abc abc"'
2281 output:
2282 - '{"offset": 0, "length": 3, "string": "abc", "captures": [{"offset": 0, "length": 3, "string": "abc", "name": null}]}'
2283 - '{"offset": 4, "length": 3, "string": "abc", "captures": [{"offset": 4, "length": 3, "string": "abc", "name": null}]}'
2284 - program: 'match("foo")'
2285 input: '"foo bar foo"'
2286 output: ['{"offset": 0, "length": 3, "string": "foo", "captures": []}']
2287 - program: 'match(["foo", "ig"])'
2288 input: '"foo bar FOO"'
2289 output:
2290 - '{"offset": 0, "length": 3, "string": "foo", "captures": []}'
2291 - '{"offset": 8, "length": 3, "string": "FOO", "captures": []}'
2292 - program: 'match("foo (?<bar123>bar)? foo"; "ig")'
2293 input: '"foo bar foo foo foo"'
2294 output:
2295 - '{"offset": 0, "length": 11, "string": "foo bar foo", "captures": [{"offset": 4, "length": 3, "string": "bar", "name": "bar123"}]}'
2296 - '{"offset": 12, "length": 8, "string": "foo foo", "captures": [{"offset": -1, "length": 0, "string": null, "name": "bar123"}]}'
2297
2298 - program: '[ match("."; "g")] | length'
2299 input: '"abc"'
2300 output: [3]
2301
2302
2303 - title: "`capture(val)`, `capture(regex; flags)`"
2304 body: |
2305
2306 Collects the named captures in a JSON object, with the name
2307 of each capture as the key, and the matched string as the
2308 corresponding value.
2309
2310 examples:
2311 - program: 'capture("(?<a>[a-z]+)-(?<n>[0-9]+)")'
2312 input: '"xyzzy-14"'
2313 output: ['{ "a": "xyzzy", "n": "14" }']
2314
2315 - title: "`scan(regex)`, `scan(regex; flags)`"
2316 body: |
2317
2318 Emit a stream of the non-overlapping substrings of the input
2319 that match the regex in accordance with the flags, if any
2320 have been specified. If there is no match, the stream is empty.
2321 To capture all the matches for each input string, use the idiom
2322 `[ expr ]`, e.g. `[ scan(regex) ]`.
2323
2324 example:
2325 - program: 'scan("c")'
2326 input: '"abcdefabc"'
2327 output: ['"c"', '"c"']
2328
2329 - program: 'scan("b")'
2330 input: ("", "")
2331 output: ['[]', '[]']
2332
2333 - title: "`split(regex; flags)`"
2334 body: |
2335
2336 For backwards compatibility, `split` splits on a string, not a regex.
2337
2338 example:
2339 - program: 'split(", *"; null)'
2340 input: '"ab,cd, ef"'
2341 output: ['"ab","cd","ef"']
2342
2343
2344 - title: "`splits(regex)`, `splits(regex; flags)`"
2345 body: |
2346
2347 These provide the same results as their `split` counterparts,
2348 but as a stream instead of an array.
2349
2350 example:
2351 - program: 'splits(", *")'
2352 input: '("ab,cd", "ef, gh")'
2353 output: ['"ab"', '"cd"', '"ef"', '"gh"']
2354
2355 - title: "`sub(regex; tostring)` `sub(regex; string; flags)`"
2356 body: |
2357
2358 Emit the string obtained by replacing the first match of regex in the
2359 input string with `tostring`, after interpolation. `tostring` should
2360 be a jq string, and may contain references to named captures. The
2361 named captures are, in effect, presented as a JSON object (as
2362 constructed by `capture`) to `tostring`, so a reference to a captured
2363 variable named "x" would take the form: "\(.x)".
2364
2365 example:
2366 - program: 'sub("^[^a-z]*(?<x>[a-z]*).*")'
2367 input: '"123abc456"'
2368 output: '"ZabcZabc"'
2369
2370
2371 - title: "`gsub(regex; string)`, `gsub(regex; string; flags)`"
2372 body: |
2373
2374 `gsub` is like `sub` but all the non-overlapping occurrences of the regex are
2375 replaced by the string, after interpolation.
2376
2377 example:
2378 - program: 'gsub("(?<x>.)[^a]*"; "+\(.x)-")'
2379 input: '"Abcabc"'
2380 output: '"+A-+a-"'
2381
2382
2383 - title: Advanced features
2384 body: |
2385 Variables are an absolute necessity in most programming languages, but
2386 they're relegated to an "advanced feature" in jq.
2387
2388 In most languages, variables are the only means of passing around
2389 data. If you calculate a value, and you want to use it more than once,
2390 you'll need to store it in a variable. To pass a value to another part
2391 of the program, you'll need that part of the program to define a
2392 variable (as a function parameter, object member, or whatever) in
2393 which to place the data.
2394
2395 It is also possible to define functions in jq, although this is
2396 is a feature whose biggest use is defining jq's standard library
2397 (many jq functions such as `map` and `find` are in fact written
2398 in jq).
2399
2400 jq has reduction operators, which are very powerful but a bit
2401 tricky. Again, these are mostly used internally, to define some
2402 useful bits of jq's standard library.
2403
2404 It may not be obvious at first, but jq is all about generators
2405 (yes, as often found in other languages). Some utilities are
2406 provided to help deal with generators.
2407
2408 Some minimal I/O support (besides reading JSON from standard
2409 input, and writing JSON to standard output) is available.
2410
2411 Finally, there is a module/library system.
2412
2413 entries:
2414 - title: "Variable / Symbolic Binding Operator: `... as $identifier | ...`"
2415 body: |
2416
2417 In jq, all filters have an input and an output, so manual
2418 plumbing is not necessary to pass a value from one part of a program
2419 to the next. Many expressions, for instance `a + b`, pass their input
2420 to two distinct subexpressions (here `a` and `b` are both passed the
2421 same input), so variables aren't usually necessary in order to use a
2422 value twice.
2423
2424 For instance, calculating the average value of an array of numbers
2425 requires a few variables in most languages - at least one to hold the
2426 array, perhaps one for each element or for a loop counter. In jq, it's
2427 simply `add / length` - the `add` expression is given the array and
2428 produces its sum, and the `length` expression is given the array and
2429 produces its length.
2430
2431 So, there's generally a cleaner way to solve most problems in jq than
2432 defining variables. Still, sometimes they do make things easier, so jq
2433 lets you define variables using `expression as $variable`. All
2434 variable names start with `$`. Here's a slightly uglier version of the
2435 array-averaging example:
2436
2437 length as $array_length | add / $array_length
2438
2439 We'll need a more complicated problem to find a situation where using
2440 variables actually makes our lives easier.
2441
2442
2443 Suppose we have an array of blog posts, with "author" and "title"
2444 fields, and another object which is used to map author usernames to
2445 real names. Our input looks like:
2446
2447 {"posts": [{"title": "Frist psot", "author": "anon"},
2448 {"title": "A well-written article", "author": "person1"}],
2449 "realnames": {"anon": "Anonymous Coward",
2450 "person1": "Person McPherson"}}
2451
2452 We want to produce the posts with the author field containing a real
2453 name, as in:
2454
2455 {"title": "Frist psot", "author": "Anonymous Coward"}
2456 {"title": "A well-written article", "author": "Person McPherson"}
2457
2458 We use a variable, $names, to store the realnames object, so that we
2459 can refer to it later when looking up author usernames:
2460
2461 .realnames as $names | .posts[] | {title, author: $names[.author]}
2462
2463 The expression `exp as $x | ...` means: for each value of expression
2464 `exp`, run the rest of the pipeline with the entire original input, and
2465 with `$x` set to that value. Thus `as` functions as something of a
2466 foreach loop.
2467
2468 Just as `{foo}` is a handy way of writing `{foo: .foo}`, so
2469 `{$foo}` is a handy way of writing `{foo:$foo}`.
2470
2471 Multiple variables may be declared using a single `as` expression by
2472 providing a pattern that matches the structure of the input
2473 (this is known as "destructuring"):
2474
2475 . as {realnames: $names, posts: [$first, $second]} | ...
2476
2477 The variable declarations in array patterns (e.g., `. as
2478 [$first, $second]`) bind to the elements of the array in from
2479 the element at index zero on up, in order. When there is no
2480 value at the index for an array pattern element, `null` is
2481 bound to that variable.
2482
2483 Variables are scoped over the rest of the expression that defines
2484 them, so
2485
2486 .realnames as $names | (.posts[] | {title, author: $names[.author]})
2487
2488 will work, but
2489
2490 (.realnames as $names | .posts[]) | {title, author: $names[.author]}
2491
2492 won't.
2493
2494 For programming language theorists, it's more accurate to
2495 say that jq variables are lexically-scoped bindings. In
2496 particular there's no way to change the value of a binding;
2497 one can only setup a new binding with the same name, but which
2498 will not be visible where the old one was.
2499
2500 examples:
2501 - program: '.bar as $x | .foo | . + $x'
2502 input: '{"foo":10, "bar":200}'
2503 output: ['210']
2504 - program: '. as $i|[(.*2|. as $i| $i), $i]'
2505 input: '5'
2506 output: ['[10,5]']
2507 - program: '. as [$a, $b, {c: $c}] | $a + $b + $c'
2508 input: '[2, 3, {"c": 4, "d": 5}]'
2509 output: ['9']
2510 - program: '.[] as [$a, $b] | {a: $a, b: $b}'
2511 input: '[[0], [0, 1], [2, 1, 0]]'
2512 output: ['{"a":0,"b":null}', '{"a":0,"b":1}', '{"a":2,"b":1}']
2513
2514 - title: 'Destructuring Alternative Operator: `?//`'
2515 body: |
2516
2517 The destructuring alternative operator provides a concise mechanism
2518 for destructuring an input that can take one of several forms.
2519
2520 Suppose we have an API that returns a list of resources and events
2521 associated with them, and we want to get the user_id and timestamp of
2522 the first event for each resource. The API (having been clumsily
2523 converted from XML) will only wrap the events in an array if the resource
2524 has multiple events:
2525
2526 {"resources": [{"id": 1, "kind": "widget", "events": {"action": "create", "user_id": 1, "ts": 13}},
2527 {"id": 2, "kind": "widget", "events": [{"action": "create", "user_id": 1, "ts": 14}, {"action": "destroy", "user_id": 1, "ts": 15}]}]}
2528
2529 We can use the destructuring alternative operator to handle this structural change simply:
2530
2531 .resources[] as {$id, $kind, events: {$user_id, $ts}} ?// {$id, $kind, events: [{$user_id, $ts}]} | {$user_id, $kind, $id, $ts}
2532
2533 Or, if we aren't sure if the input is an array of values or an object:
2534
2535 .[] as [$id, $kind, $user_id, $ts] ?// {$id, $kind, $user_id, $ts} | ...
2536
2537 Each alternative need not define all of the same variables, but all named
2538 variables will be available to the subsequent expression. Variables not
2539 matched in the alternative that succeeded will be `null`:
2540
2541 .resources[] as {$id, $kind, events: {$user_id, $ts}} ?// {$id, $kind, events: [{$first_user_id, $first_ts}]} | {$user_id, $first_user_id, $kind, $id, $ts, $first_ts}
2542
2543 Additionally, if the subsequent expression returns an error, the
2544 alternative operator will attempt to try the next binding. Errors
2545 that occur during the final alternative are passed through.
2546
2547 [[3]] | .[] as [$a] ?// [$b] | if $a != null then error("err: \($a)") else {$a,$b} end
2548
2549 examples:
2550 - program: '.[] as {$a, $b, c: {$d, $e}} ?// {$a, $b, c: [{$d, $e}]} | {$a, $b, $d, $e}'
2551 input: '[{"a": 1, "b": 2, "c": {"d": 3, "e": 4}}, {"a": 1, "b": 2, "c": [{"d": 3, "e": 4}]}]'
2552 output: ['{"a":1,"b":2,"d":3,"e":4}', '{"a":1,"b":2,"d":3,"e":4}']
2553 - program: '.[] as {$a, $b, c: {$d}} ?// {$a, $b, c: [{$e}]} | {$a, $b, $d, $e}'
2554 input: '[{"a": 1, "b": 2, "c": {"d": 3, "e": 4}}, {"a": 1, "b": 2, "c": [{"d": 3, "e": 4}]}]'
2555 output: ['{"a":1,"b":2,"d":3,"e":null}', '{"a":1,"b":2,"d":null,"e":4}']
2556 - program: '.[] as [$a] ?// [$b] | if $a != null then error("err: \($a)") else {$a,$b} end'
2557 input: '[[3]]'
2558 output: ['{"a":null,"b":3}']
2559
2560 - title: 'Defining Functions'
2561 body: |
2562
2563 You can give a filter a name using "def" syntax:
2564
2565 def increment: . + 1;
2566
2567 From then on, `increment` is usable as a filter just like a
2568 builtin function (in fact, this is how many of the builtins
2569 are defined). A function may take arguments:
2570
2571 def map(f): [.[] | f];
2572
2573 Arguments are passed as _filters_ (functions with no
2574 arguments), _not_ as values. The same argument may be
2575 referenced multiple times with different inputs (here `f` is
2576 run for each element of the input array). Arguments to a
2577 function work more like callbacks than like value arguments.
2578 This is important to understand. Consider:
2579
2580 def foo(f): f|f;
2581 5|foo(.*2)
2582
2583 The result will be 20 because `f` is `.*2`, and during the
2584 first invocation of `f` `.` will be 5, and the second time it
2585 will be 10 (5 * 2), so the result will be 20. Function
2586 arguments are filters, and filters expect an input when
2587 invoked.
2588
2589 If you want the value-argument behaviour for defining simple
2590 functions, you can just use a variable:
2591
2592 def addvalue(f): f as $f | map(. + $f);
2593
2594 Or use the short-hand:
2595
2596 def addvalue($f): ...;
2597
2598 With either definition, `addvalue(.foo)` will add the current
2599 input's `.foo` field to each element of the array. Do note
2600 that calling `addvalue(.[])` will cause the `map(. + $f)` part
2601 to be evaluated once per value in the value of `.` at the call
2602 site.
2603
2604 Multiple definitions using the same function name are allowed.
2605 Each re-definition replaces the previous one for the same
2606 number of function arguments, but only for references from
2607 functions (or main program) subsequent to the re-definition.
2608 See also the section below on scoping.
2609
2610 examples:
2611 - program: 'def addvalue(f): . + [f]; map(addvalue(.[0]))'
2612 input: '[[1,2],[10,20]]'
2613 output: ['[[1,2,1], [10,20,10]]']
2614 - program: 'def addvalue(f): f as $x | map(. + $x); addvalue(.[0])'
2615 input: '[[1,2],[10,20]]'
2616 output: ['[[1,2,1,2], [10,20,1,2]]']
2617
2618 - title: 'Scoping'
2619 body: |
2620
2621 There are two types of symbols in jq: value bindings (a.k.a.,
2622 "variables"), and functions. Both are scoped lexically,
2623 with expressions being able to refer only to symbols that
2624 have been defined "to the left" of them. The only exception
2625 to this rule is that functions can refer to themselves so as
2626 to be able to create recursive functions.
2627
2628 For example, in the following expression there is a binding
2629 which is visible "to the right" of it, `... | .*3 as
2630 $times_three | [. + $times_three] | ...`, but not "to the
2631 left". Consider this expression now, `... | (.*3 as
2632 $times_three | [.+ $times_three]) | ...`: here the binding
2633 `$times_three` is _not_ visible past the closing parenthesis.
2634
2635 - title: Reduce
2636 body: |
2637
2638 The `reduce` syntax in jq allows you to combine all of the
2639 results of an expression by accumulating them into a single
2640 answer. As an example, we'll pass `[3,2,1]` to this expression:
2641
2642 reduce .[] as $item (0; . + $item)
2643
2644 For each result that `.[]` produces, `. + $item` is run to
2645 accumulate a running total, starting from 0. In this
2646 example, `.[]` produces the results 3, 2, and 1, so the
2647 effect is similar to running something like this:
2648
2649 0 | (3 as $item | . + $item) |
2650 (2 as $item | . + $item) |
2651 (1 as $item | . + $item)
2652
2653 examples:
2654 - program: 'reduce .[] as $item (0; . + $item)'
2655 input: '[10,2,5,3]'
2656 output: ['20']
2657
2658 - title: "`isempty(exp)`"
2659 body: |
2660
2661 Returns true if `exp` produces no outputs, false otherwise.
2662
2663 examples:
2664 - program: 'isempty(empty)'
2665 input: 'null'
2666 output: ['true']
2667
2668 - title: "`limit(n; exp)`"
2669 body: |
2670
2671 The `limit` function extracts up to `n` outputs from `exp`.
2672
2673 examples:
2674 - program: '[limit(3;.[])]'
2675 input: '[0,1,2,3,4,5,6,7,8,9]'
2676 output: ['[0,1,2]']
2677
2678 - title: "`first(expr)`, `last(expr)`, `nth(n; expr)`"
2679 body: |
2680
2681 The `first(expr)` and `last(expr)` functions extract the first
2682 and last values from `expr`, respectively.
2683
2684 The `nth(n; expr)` function extracts the nth value output by
2685 `expr`. This can be defined as `def nth(n; expr):
2686 last(limit(n + 1; expr));`. Note that `nth(n; expr)` doesn't
2687 support negative values of `n`.
2688
2689 examples:
2690 - program: '[first(range(.)), last(range(.)), nth(./2; range(.))]'
2691 input: '10'
2692 output: ['[0,9,5]']
2693
2694 - title: "`first`, `last`, `nth(n)`"
2695 body: |
2696
2697 The `first` and `last` functions extract the first
2698 and last values from any array at `.`.
2699
2700 The `nth(n)` function extracts the nth value of any array at `.`.
2701
2702 examples:
2703 - program: '[range(.)]|[first, last, nth(5)]'
2704 input: '10'
2705 output: ['[0,9,5]']
2706
2707 - title: "`foreach`"
2708 body: |
2709
2710 The `foreach` syntax is similar to `reduce`, but intended to
2711 allow the construction of `limit` and reducers that produce
2712 intermediate results (see example).
2713
2714 The form is `foreach EXP as $var (INIT; UPDATE; EXTRACT)`.
2715 Like `reduce`, `INIT` is evaluated once to produce a state
2716 value, then each output of `EXP` is bound to `$var`, `UPDATE`
2717 is evaluated for each output of `EXP` with the current state
2718 and with `$var` visible. Each value output by `UPDATE`
2719 replaces the previous state. Finally, `EXTRACT` is evaluated
2720 for each new state to extract an output of `foreach`.
2721
2722 This is mostly useful only for constructing `reduce`- and
2723 `limit`-like functions. But it is much more general, as it
2724 allows for partial reductions (see the example below).
2725
2726 examples:
2727 - program: '[foreach .[] as $item
2728 ([[],[]];
2729 if $item == null then [[],.[0]] else [(.[0] + [$item]),[]] end;
2730 if $item == null then .[1] else empty end)]'
2731 input: '[1,2,3,4,null,"a","b",null]'
2732 output: ['[[1,2,3,4],["a","b"]]']
2733
2734 - title: Recursion
2735 body: |
2736
2737 As described above, `recurse` uses recursion, and any jq
2738 function can be recursive. The `while` builtin is also
2739 implemented in terms of recursion.
2740
2741 Tail calls are optimized whenever the expression to the left of
2742 the recursive call outputs its last value. In practice this
2743 means that the expression to the left of the recursive call
2744 should not produce more than one output for each input.
2745
2746 For example:
2747
2748 def recurse(f): def r: ., (f | select(. != null) | r); r;
2749
2750 def while(cond; update):
2751 def _while:
2752 if cond then ., (update | _while) else empty end;
2753 _while;
2754
2755 def repeat(exp):
2756 def _repeat:
2757 exp, _repeat;
2758 _repeat;
2759
2760 - title: Generators and iterators
2761 body: |
2762
2763 Some jq operators and functions are actually generators in
2764 that they can produce zero, one, or more values for each
2765 input, just as one might expect in other programming
2766 languages that have generators. For example, `.[]`
2767 generates all the values in its input (which must be an
2768 array or an object), `range(0; 10)` generates the integers
2769 between 0 and 10, and so on.
2770
2771 Even the comma operator is a generator, generating first the
2772 values generated by the expression to the left of the comma,
2773 then for each of those, the values generate by the
2774 expression on the right of the comma.
2775
2776 The `empty` builtin is the generator that produces zero
2777 outputs. The `empty` builtin backtracks to the preceding
2778 generator expression.
2779
2780 All jq functions can be generators just by using builtin
2781 generators. It is also possible to define new generators
2782 using only recursion and the comma operator. If the
2783 recursive call(s) is(are) "in tail position" then the
2784 generator will be efficient. In the example below the
2785 recursive call by `_range` to itself is in tail position.
2786 The example shows off three advanced topics: tail recursion,
2787 generator construction, and sub-functions.
2788
2789 examples:
2790 - program: 'def range(init; upto; by):
2791 def _range:
2792 if (by > 0 and . < upto) or (by < 0 and . > upto)
2793 then ., ((.+by)|_range)
2794 else . end;
2795 if by == 0 then init else init|_range end |
2796 select((by > 0 and . < upto) or (by < 0 and . > upto));
2797 range(0; 10; 3)'
2798 input: 'null'
2799 output: ['0', '3', '6', '9']
2800 - program: 'def while(cond; update):
2801 def _while:
2802 if cond then ., (update | _while) else empty end;
2803 _while;
2804 [while(.<100; .*2)]'
2805 input: '1'
2806 output: ['[1,2,4,8,16,32,64]']
2807
2808 - title: 'Math'
2809 body: |
2810
2811 jq currently only has IEEE754 double-precision (64-bit) floating
2812 point number support.
2813
2814 Besides simple arithmetic operators such as `+`, jq also has most
2815 standard math functions from the C math library. C math functions
2816 that take a single input argument (e.g., `sin()`) are available as
2817 zero-argument jq functions. C math functions that take two input
2818 arguments (e.g., `pow()`) are available as two-argument jq
2819 functions that ignore `.`. C math functions that take three input
2820 arguments are available as three-argument jq functions that ignore
2821 `.`.
2822
2823 Availability of standard math functions depends on the
2824 availability of the corresponding math functions in your operating
2825 system and C math library. Unavailable math functions will be
2826 defined but will raise an error.
2827
2828 One-input C math functions: `acos` `acosh` `asin` `asinh` `atan`
2829 `atanh` `cbrt` `ceil` `cos` `cosh` `erf` `erfc` `exp` `exp10`
2830 `exp2` `expm1` `fabs` `floor` `gamma` `j0` `j1` `lgamma` `log`
2831 `log10` `log1p` `log2` `logb` `nearbyint` `pow10` `rint` `round`
2832 `significand` `sin` `sinh` `sqrt` `tan` `tanh` `tgamma` `trunc`
2833 `y0` `y1`.
2834
2835 Two-input C math functions: `atan2` `copysign` `drem` `fdim`
2836 `fmax` `fmin` `fmod` `frexp` `hypot` `jn` `ldexp` `modf`
2837 `nextafter` `nexttoward` `pow` `remainder` `scalb` `scalbln` `yn`.
2838
2839 Three-input C math functions: `fma`.
2840
2841 See your system's manual for more information on each of these.
2842
2843 - title: 'I/O'
2844 body: |
2845
2846 At this time jq has minimal support for I/O, mostly in the
2847 form of control over when inputs are read. Two builtins functions
2848 are provided for this, `input` and `inputs`, that read from the
2849 same sources (e.g., `stdin`, files named on the command-line) as
2850 jq itself. These two builtins, and jq's own reading actions, can
2851 be interleaved with each other.
2852
2853 Two builtins provide minimal output capabilities, `debug`, and
2854 `stderr`. (Recall that a jq program's output values are always
2855 output as JSON texts on `stdout`.) The `debug` builtin can have
2856 application-specific behavior, such as for executables that use
2857 the libjq C API but aren't the jq executable itself. The `stderr`
2858 builtin outputs its input in raw mode to stder with no additional
2859 decoration, not even a newline.
2860
2861 Most jq builtins are referentially transparent, and yield constant
2862 and repeatable value streams when applied to constant inputs.
2863 This is not true of I/O builtins.
2864
2865 entries:
2866 - title: "`input`"
2867 body: |
2868
2869 Outputs one new input.
2870
2871 - title: "`inputs`"
2872 body: |
2873
2874 Outputs all remaining inputs, one by one.
2875
2876 This is primarily useful for reductions over a program's
2877 inputs.
2878
2879 - title: "`debug`"
2880 body: |
2881
2882 Causes a debug message based on the input value to be
2883 produced. The jq executable wraps the input value with
2884 `["DEBUG:", <input-value>]` and prints that and a newline on
2885 stderr, compactly. This may change in the future.
2886
2887 - title: "`stderr`"
2888 body: |
2889
2890 Prints its input in raw and compact mode to stderr with no
2891 additional decoration, not even a newline.
2892
2893 - title: "`input_filename`"
2894 body: |
2895
2896 Returns the name of the file whose input is currently being
2897 filtered. Note that this will not work well unless jq is
2898 running in a UTF-8 locale.
2899
2900 - title: "`input_line_number`"
2901 body: |
2902
2903 Returns the line number of the input currently being filtered.
2904
2905 - title: 'Streaming'
2906 body: |
2907
2908 With the `--stream` option jq can parse input texts in a streaming
2909 fashion, allowing jq programs to start processing large JSON texts
2910 immediately rather than after the parse completes. If you have a
2911 single JSON text that is 1GB in size, streaming it will allow you
2912 to process it much more quickly.
2913
2914 However, streaming isn't easy to deal with as the jq program will
2915 have `[<path>, <leaf-value>]` (and a few other forms) as inputs.
2916
2917 Several builtins are provided to make handling streams easier.
2918
2919 The examples below use the streamed form of `[0,[1]]`, which is
2920 `[[0],0],[[1,0],1],[[1,0]],[[1]]`.
2921
2922 Streaming forms include `[<path>, <leaf-value>]` (to indicate any
2923 scalar value, empty array, or empty object), and `[<path>]` (to
2924 indicate the end of an array or object). Future versions of jq
2925 run with `--stream` and `-seq` may output additional forms such as
2926 `["error message"]` when an input text fails to parse.
2927
2928 entries:
2929 - title: "`truncate_stream(stream_expression)`"
2930 body: |
2931
2932 Consumes a number as input and truncates the corresponding
2933 number of path elements from the left of the outputs of the
2934 given streaming expression.
2935
2936 examples:
2937 - program: '[1|truncate_stream([[0],1],[[1,0],2],[[1,0]],[[1]])]'
2938 input: '1'
2939 output: ['[[[0],2],[[0]]]']
2940
2941 - title: "`fromstream(stream_expression)`"
2942 body: |
2943
2944 Outputs values corresponding to the stream expression's
2945 outputs.
2946
2947 examples:
2948 - program: 'fromstream(1|truncate_stream([[0],1],[[1,0],2],[[1,0]],[[1]]))'
2949 input: 'null'
2950 output: ['[2]']
2951
2952 - title: "`tostream`"
2953 body: |
2954
2955 The `tostream` builtin outputs the streamed form of its input.
2956
2957 examples:
2958 - program: '. as $dot|fromstream($dot|tostream)|.==$dot'
2959 input: '[0,[1,{"a":1},{"b":2}]]'
2960 output: ['true']
2961
2962 - title: Assignment
2963 body: |
2964 Assignment works a little differently in jq than in most
2965 programming languages. jq doesn't distinguish between references
2966 to and copies of something - two objects or arrays are either
2967 equal or not equal, without any further notion of being "the
2968 same object" or "not the same object".
2969
2970 If an object has two fields which are arrays, `.foo` and `.bar`,
2971 and you append something to `.foo`, then `.bar` will not get
2972 bigger, even if you've previously set `.bar = .foo`. If you're
2973 used to programming in languages like Python, Java, Ruby,
2974 Javascript, etc. then you can think of it as though jq does a full
2975 deep copy of every object before it does the assignment (for
2976 performance it doesn't actually do that, but that's the general
2977 idea).
2978
2979 This means that it's impossible to build circular values in jq
2980 (such as an array whose first element is itself). This is quite
2981 intentional, and ensures that anything a jq program can produce
2982 can be represented in JSON.
2983
2984 All the assignment operators in jq have path expressions on the
2985 left-hand side (LHS). The right-hand side (RHS) provides values
2986 to set to the paths named by the LHS path expressions.
2987
2988 Values in jq are always immutable. Internally, assignment works
2989 by using a reduction to compute new, replacement values for `.` that
2990 have had all the desired assignments applied to `.`, then
2991 outputting the modified value. This might be made clear by this
2992 example: `{a:{b:{c:1}}} | (.a.b|=3), .`. This will output
2993 `{"a":{"b":3}}` and `{"a":{"b":{"c":1}}}` because the last
2994 sub-expression, `.`, sees the original value, not the modified
2995 value.
2996
2997 Most users will want to use modification assignment operators,
2998 such as `|=` or `+=`, rather than `=`.
2999
3000 Note that the LHS of assignment operators refers to a value in
3001 `.`. Thus `$var.foo = 1` won't work as expected (`$var.foo` is
3002 not a valid or useful path expression in `.`); use `$var | .foo =
3003 1` instead.
3004
3005 Note too that `.a,.b=0` does not set `.a` and `.b`, but
3006 `(.a,.b)=0` sets both.
3007
3008 entries:
3009 - title: "Update-assignment: `|=`"
3010 body: |
3011 This is the "update" operator '|='. It takes a filter on the
3012 right-hand side and works out the new value for the property
3013 of `.` being assigned to by running the old value through this
3014 expression. For instance, (.foo, .bar) |= .+1 will build an
3015 object with the "foo" field set to the input's "foo" plus 1,
3016 and the "bar" field set to the input's "bar" plus 1.
3017
3018 The left-hand side can be any general path expression; see `path()`.
3019
3020 Note that the left-hand side of '|=' refers to a value in `.`.
3021 Thus `$var.foo |= . + 1` won't work as expected (`$var.foo` is
3022 not a valid or useful path expression in `.`); use `$var |
3023 .foo |= . + 1` instead.
3024
3025 If the right-hand side outputs no values (i.e., `empty`), then
3026 the left-hand side path will be deleted, as with `del(path)`.
3027
3028 If the right-hand side outputs multiple values, only the first
3029 one will be used (COMPATIBILITY NOTE: in jq 1.5 and earlier
3030 releases, it used to be that only the last one was used).
3031
3032 examples:
3033 - program: '(..|select(type=="boolean")) |= if . then 1 else 0 end'
3034 input: '[true,false,[5,true,[true,[false]],false]]'
3035 output: ['[1,0,[5,1,[1,[0]],0]]']
3036
3037 - title: "Arithmetic update-assignment: `+=`, `-=`, `*=`, `/=`, `%=`, `//=`"
3038 body: |
3039
3040 jq has a few operators of the form `a op= b`, which are all
3041 equivalent to `a |= . op b`. So, `+= 1` can be used to
3042 increment values, being the same as `|= . + 1`.
3043
3044 examples:
3045 - program: .foo += 1
3046 input: '{"foo": 42}'
3047 output: ['{"foo": 43}']
3048
3049 - title: "Plain assignment: `=`"
3050 body: |
3051
3052 This is the plain assignment operator. Unlike the others, the
3053 input to the right-hand-side (RHS) is the same as the input to
3054 the left-hand-side (LHS) rather than the value at the LHS
3055 path, and all values output by the RHS will be used (as shown
3056 below).
3057
3058 If the RHS of '=' produces multiple values, then for each such
3059 value jq will set the paths on the left-hand side to the value
3060 and then it will output the modified `.`. For example,
3061 `(.a,.b)=range(2)` outputs `{"a":0,"b":0}`, then
3062 `{"a":1,"b":1}`. The "update" assignment forms (see above) do
3063 not do this.
3064
3065 This example should show the difference between '=' and '|=':
3066
3067 Provide input '{"a": {"b": 10}, "b": 20}' to the programs:
3068
3069 .a = .b
3070
3071 .a |= .b
3072
3073 The former will set the "a" field of the input to the "b"
3074 field of the input, and produce the output {"a": 20, "b": 20}.
3075 The latter will set the "a" field of the input to the "a"
3076 field's "b" field, producing {"a": 10, "b": 20}.
3077
3078 Another example of the difference between '=' and '|=':
3079
3080 null|(.a,.b)=range(3)
3081
3082 outputs '{"a":0,"b":0}', '{"a":1,"b":1}', and '{"a":2,"b":2}',
3083 while
3084
3085 null|(.a,.b)|=range(3)
3086
3087 outputs just '{"a":0,"b":0}'.
3088
3089 - title: Complex assignments
3090 body: |
3091 Lots more things are allowed on the left-hand side of a jq assignment
3092 than in most languages. We've already seen simple field accesses on
3093 the left hand side, and it's no surprise that array accesses work just
3094 as well:
3095
3096 .posts[0].title = "JQ Manual"
3097
3098 What may come as a surprise is that the expression on the left may
3099 produce multiple results, referring to different points in the input
3100 document:
3101
3102 .posts[].comments |= . + ["this is great"]
3103
3104 That example appends the string "this is great" to the "comments"
3105 array of each post in the input (where the input is an object with a
3106 field "posts" which is an array of posts).
3107
3108 When jq encounters an assignment like 'a = b', it records the "path"
3109 taken to select a part of the input document while executing a. This
3110 path is then used to find which part of the input to change while
3111 executing the assignment. Any filter may be used on the
3112 left-hand side of an equals - whichever paths it selects from the
3113 input will be where the assignment is performed.
3114
3115 This is a very powerful operation. Suppose we wanted to add a comment
3116 to blog posts, using the same "blog" input above. This time, we only
3117 want to comment on the posts written by "stedolan". We can find those
3118 posts using the "select" function described earlier:
3119
3120 .posts[] | select(.author == "stedolan")
3121
3122 The paths provided by this operation point to each of the posts that
3123 "stedolan" wrote, and we can comment on each of them in the same way
3124 that we did before:
3125
3126 (.posts[] | select(.author == "stedolan") | .comments) |=
3127 . + ["terrible."]
3128
3129 - title: Modules
3130 body: |
3131
3132 jq has a library/module system. Modules are files whose names end
3133 in `.jq`.
3134
3135 Modules imported by a program are searched for in a default search
3136 path (see below). The `import` and `include` directives allow the
3137 importer to alter this path.
3138
3139 Paths in the a search path are subject to various substitutions.
3140
3141 For paths starting with "~/", the user's home directory is
3142 substituted for "~".
3143
3144 For paths starting with "$ORIGIN/", the path of the jq executable
3145 is substituted for "$ORIGIN".
3146
3147 For paths starting with "./" or paths that are ".", the path of
3148 the including file is substituted for ".". For top-level programs
3149 given on the command-line, the current directory is used.
3150
3151 Import directives can optionally specify a search path to which
3152 the default is appended.
3153
3154 The default search path is the search path given to the `-L`
3155 command-line option, else `["~/.jq", "$ORIGIN/../lib/jq",
3156 "$ORIGIN/../lib"]`.
3157
3158 Null and empty string path elements terminate search path
3159 processing.
3160
3161 A dependency with relative path "foo/bar" would be searched for in
3162 "foo/bar.jq" and "foo/bar/bar.jq" in the given search path. This
3163 is intended to allow modules to be placed in a directory along
3164 with, for example, version control files, README files, and so on,
3165 but also to allow for single-file modules.
3166
3167 Consecutive components with the same name are not allowed to avoid
3168 ambiguities (e.g., "foo/foo").
3169
3170 For example, with `-L$HOME/.jq` a module `foo` can be found in
3171 `$HOME/.jq/foo.jq` and `$HOME/.jq/foo/foo.jq`.
3172
3173 If "$HOME/.jq" is a file, it is sourced into the main program.
3174
3175 entries:
3176 - title: "`import RelativePathString as NAME [<metadata>];`"
3177 body: |
3178
3179 Imports a module found at the given path relative to a
3180 directory in a search path. A ".jq" suffix will be added to
3181 the relative path string. The module's symbols are prefixed
3182 with "NAME::".
3183
3184 The optional metadata must be a constant jq expression. It
3185 should be an object with keys like "homepage" and so on. At
3186 this time jq only uses the "search" key/value of the metadata.
3187 The metadata is also made available to users via the
3188 `modulemeta` builtin.
3189
3190 The "search" key in the metadata, if present, should have a
3191 string or array value (array of strings); this is the search
3192 path to be prefixed to the top-level search path.
3193
3194 - title: "`include RelativePathString [<metadata>];`"
3195 body: |
3196
3197 Imports a module found at the given path relative to a
3198 directory in a search path as if it were included in place. A
3199 ".jq" suffix will be added to the relative path string. The
3200 module's symbols are imported into the caller's namespace as
3201 if the module's content had been included directly.
3202
3203 The optional metadata must be a constant jq expression. It
3204 should be an object with keys like "homepage" and so on. At
3205 this time jq only uses the "search" key/value of the metadata.
3206 The metadata is also made available to users via the
3207 `modulemeta` builtin.
3208
3209 - title: "`import RelativePathString as $NAME [<metadata>];`"
3210 body: |
3211
3212 Imports a JSON file found at the given path relative to a
3213 directory in a search path. A ".json" suffix will be added to
3214 the relative path string. The file's data will be available
3215 as `$NAME::NAME`.
3216
3217 The optional metadata must be a constant jq expression. It
3218 should be an object with keys like "homepage" and so on. At
3219 this time jq only uses the "search" key/value of the metadata.
3220 The metadata is also made available to users via the
3221 `modulemeta` builtin.
3222
3223 The "search" key in the metadata, if present, should have a
3224 string or array value (array of strings); this is the search
3225 path to be prefixed to the top-level search path.
3226
3227 - title: "`module <metadata>;`"
3228 body: |
3229
3230 This directive is entirely optional. It's not required for
3231 proper operation. It serves only the purpose of providing
3232 metadata that can be read with the `modulemeta` builtin.
3233
3234 The metadata must be a constant jq expression. It should be
3235 an object with keys like "homepage". At this time jq doesn't
3236 use this metadata, but it is made available to users via the
3237 `modulemeta` builtin.
3238
3239 - title: "`modulemeta`"
3240 body: |
3241
3242 Takes a module name as input and outputs the module's metadata
3243 as an object, with the module's imports (including metadata)
3244 as an array value for the "deps" key.
3245
3246 Programs can use this to query a module's metadata, which they
3247 could then use to, for example, search for, download, and
3248 install missing dependencies.
3249
3250 - title: Colors
3251 body: |
3252
3253 To configure alternative colors just set the `JQ_COLORS`
3254 environment variable to colon-delimited list of partial terminal
3255 escape sequences like `"1;31"`, in this order:
3256
3257 - color for `null`
3258 - color for `false`
3259 - color for `true`
3260 - color for numbers
3261 - color for strings
3262 - color for arrays
3263 - color for objects
3264
3265 The default color scheme is the same as setting
3266 `"JQ_COLORS=1;30:0;39:0;39:0;39:0;32:1;39:1;39"`.
3267
3268 This is not a manual for VT100/ANSI escapes. However, each of
3269 these color specifications should consist of two numbers separated
3270 by a semi-colon, where the first number is one of these:
3271
3272 - 1 (bright)
3273 - 2 (dim)
3274 - 4 (underscore)
3275 - 5 (blink)
3276 - 7 (reverse)
3277 - 8 (hidden)
3278
3279 and the second is one of these:
3280
3281 - 30 (black)
3282 - 31 (red)
3283 - 32 (green)
3284 - 33 (yellow)
3285 - 34 (blue)
3286 - 35 (magenta)
3287 - 36 (cyan)
3288 - 37 (white)