-------------------------------------------------------------------------------
JSON - JavaScript Object Notation

This is a very well defined data format, that is easy for both humans
and machines to generate, and allows simple automatic reformating.

Unlike XML does not provide any form of schema, or URL pointer to
semantic/scheme specification of the data. It does not even allow comments.

Its simplicity makes it good as a "data serialization language".

JSON Specification (simple)
  https://www.json.org/
For a quick summery of data format
  https://learnxinyminutes.com/docs/json/
"The JSON Saga" from creator, history and relation to other dta formats
  https://www.youtube.com/watch?v=-C-JoyNuQJs

JSON and YAML are often used interchangeably for data output by many programs.
Part of YAML includes JSON as part of its data format structure.

JSON does not care about whitespace while YAML does, as such programs that
output JSON often forgos whitespaces (especially newlines) so as to make
the output unreadable to humans.

---
JSON5 - JSON for humans

This is a extension, supposedly designed for better human editing of JSON-like
config files.  It includes things like: comments, mutli-line strings, floating
point, Hex-adecimal characters and more lenient quoting and white space.

  https://json5.org/

---
FireFox and Chrome and many other programs uses for its backup data storage.

The database MongoDB is specifically designed with JSON like databases,
in mind.

-------------------------------------------------------------------------------
Format

* Elements are comma seperated    "key":"value"  pairs
* The key must be quoted, but can be ANY string, not just a label
* Special values provided:  true, false, null
* Which are grouped (unordered) of key:values using    { ... }
* Or in a ordered list of values using    [ ... ]
* Characters escaped using  \
* White space outside quotes are ignored
  As such, many program do not include any newlines,
  the data is all just one single LONG line!

* No comments, though most parsers accept  // or /*...*/  C-style comments
  Removed from format in 2005 as it complicated the language uselessly.
  And people were using comments to add Meta Data, outside the data!
  All changes to JSON after 2005 is closed. Format is fix and unchanging.

  If you really need a comment, add it to the data...
    "_comment": "comment text goes here...",

Strings are all quoted (including keys):
  any Unicode character,  except ",  or \, or control-character
  \"
  \\
  \/        <-- optional escape to allow breaking html tags EG "<\/script>"
  \b        <-- backspace (not bell)
  \f
  \n
  \r
  \t
  \uHHHH    <-- unicode with 4 hex digits
  \xHH      <-- single hex char (not strict JSON, added in JSON5)

NOTE: JSON does NOT allow control chars in strings
  "this stings has a
   newline in it"
is NOT allowed!  The correct way to embed a new line is...
  "this stings has a\nnewline in it"

Most JSON parsers also allow you to use "\xHH" escapes
though that is not strictly part of the original JSON

NOTE: Bash $'..' understands most of these escapes (except for "\/")

-------------------------------------------------------------------------------
Example...

See the "json_example.json"  which will be used below

-------------------------------------------------------------------------------
Program Summary (using above json_example.json)
Also see bottom of page https://json.org/

jq

  Described as a 'sed' command for JSON (file or stdin)
  BUT hard to understand the search structure!

  Pretty Print...
    jq . json_example.json
  or even no arguments
    jq < json_example.json

  Simple extraction of one element is fine...

  One element
    jq '.object.array[4]'  json_example.json
    #> "Arrays can have anything in them."

    jq '.object | .comment'  json_example.json
    #> "Most of your structure will come from objects."

    jq '.object.obj_name'  json_example.json
    jq '.[object][obj_name]'  json_example.json
    #> "obj_value"
    # Note how  .obj_name  is short for  ["obj_name"]


  Multiple values...
    jq '.object | ._comment, .obj_name' json_example.json
    jq '.object | .["_comment","obj_name"]' json_example.json
    #> "Most of your structure will come from objects."
    #  "obj_value"
    # Order is as given

    jq '.object.array[4],  .object._comment'  json_example.json
    # get two objects (',') in that order
    #> "fourth element."
    #  "Most of your structure will come from objects."

    jq '.object | .array[4],._comment'  json_example.json
    # first extract .object , then specific sub-elements
    # to produce the same result as previous.
    #> "fourth element."
    #  "Most of your structure will come from objects."

  Multiple in hash contexts
    jq '.object | {obj_name}'  json_example.json
    #> {
    #    "obj_name": "obj_value"
    #  }

  Multiple in array contexts (no keys)
    az vm list | jq '.[] | [ .name, .resourceGroup ]'

  Multiple elements of a hash (json {..})
    jq '.object | {_comment,obj_name}' json_example.json
    #> {
    #    "_comment": "Most of your structure will come from objects.",
    #    "obj_name": "obj_value"
    #  }
    # note order is the order gven on command line

  If input file is an array, sepeate the elements of the arry first
  Elements of an array, (remove toplevel [] first)
    docker inspect {image} | jq '.[] | .Config.Env'
    az vm list | jq '.[] | .name, .resourceGroup'

  or concatenated strings
    jq '.object | .+_comment+" "+._array[4]' json_example.json

  Search and array of objects for specific ones...
    # get "_comment" from objects with "name" ending in "d"
    jq '.objects[] | select( .name | match("d$")) | ._comment' \
      json_example.json

niet

  python yaml/json extractor & converter
  https://github.com/openuado/niet
  Output is yaml-like (not exactly) indenting...
  It may convert unicode into quoted \xXX escapes (which is not json)

    niet . info/data/json_example.json
    #> { 
    #    "key": "value",
    #    "keys": "must always be enclosed in \"double quotes\".",
    #    "strings": "Unicode: Hell\xF8, w\u25CFrld, and \"escapes\".",
    #    "nothingness": null
    #    ...
    #  }

    niet .strings info/data/json_example.json
    #> Unicode: Hellø, w●rld, and "escapes".

gron

  json to discrete assignments (standard package)
  EG:
    json.key = "value";
    json.nothingness = null;
    json.objects.array[3] = 3
    json.arrays[0]["sources of potassium"][0] = "bananas";


jwalk   scripted parser,
        outputing one line per value,
        with array position, label type, value
        and a smimple filter for a sub-path

jason-table
        convert a json table to CSV structure

json.sh
        Parser written in shell (not a standard package)...
          https://github.com/dominictarr/JSON.sh
        one key-path and value per line

jshon   read JSON in shell scripts (not in RPM repos)
          http://kmkeen.com/jshon/
        Example
          jshon  -e whitespace < json_example.json


nushell
  A data processing shell for JSON, YAML, SQLite, Excel, etc
    https://www.nushell.sh/
  Example
     http get https://api.github.com/repos/<user>/<repo>/releases/latest |
       get tag_name

-----
Reformating
  json_reformat     from YAJL (Yet Another JSON Library)
                      all "key","values" and close syntax on seperate lines
                      But order of input remains the same.
                         -m    minimize, removing all white space and newlines
                         -s    work on a stream of json (BIG file handling)

  json_pp           perl using JSON::PP module
                      The same but may re-order the unordered {...} groupings


API's
  perl-JSON           decode into hash/array data tree

  python
    import json
    from pprint import pprint
    data = json.dumps(var)
    pprint(json.loads(data))

  php
    // Requires extra package:  dnf install php-json
    $data = json_encode(Array('task_name'=> 'doit',
                              'params'   => Array('arg1'=>'xyzzy',
                                                  'arg2'=>'foobar')));
    var_dump($data);
    $task = json_decode($data);
    var_dump($task);
    $json_pp = json_encode($info, JSON_PRETTY_PRINT );
    var_dump($json_pp);

-------------------------------------------------------------------------------
Formatting to make it more human readable

Perl, JSON module (not present by default)
  json_pp

  Equivelent to...
    echo '{"foo": "lorem", "bar": "ipsum"}' |
       perl -0007 -MJSON -nE'say to_json(from_json($_), {pretty=>1})'

Or (with color)
  jq . file.json
  cat file.json | jq . -

From yajl package (Yet Another JSON Library) - reformatter and verifier
installed by default in fedora
  json_reformat < file.json

Quick JSON formatter...    (good at locating a syntax fault)
  echo '{"foo": "lorem", "bar": "ipsum"}' | python -mjson.tool

On older versions of python use
  python -c'from simplejson.tool import main; main()'

jshon  (c program)  from http://kmkeen.com/jshon/  (not in repos)
which is designed to make json useful from shell scripts.
and pretty prints by default.

PROBLEM: ALL the above pretty prints into one line per item
with a 3 to 4 character indent

However often groups of items are small enough that they should can be
formatted onto the same line. Especially arrays of groups where a line per
array group, makes a lot more sense, especially when you align group element
into columns.

-------------------------------------------------------------------------------
Processing JSON (without an API relience)

JSON is a hierarchical data structure, which means information about
a containers can be placed before or after the children of the container.  It
is name tagged, so technically the order of the tags can also change.

Also when processing you generally want to process data in groups, chunks, or
records.  However it is data dependant on how the json data should be split up
and if it should be processed as a linear array or as a tree.

The only real way of processing it in all situations is to read it as
a complete data structure, then process it as appropriate.


Example Perl Processing...
Multi level JSON tree (Google Chrome Bookmarks file)

A recursive tree walk for records of "type:folder",
containing an array of "children".

=======8<--------
#!/bin/perl
use JSON;

$DEBUG=1;
my $json_file="$ENV{HOME}/.config/google-chrome/Default/Bookmarks";

# Grab the JSON using perl JSON module
local $/;
open(JSON, $json_file) || die;
my $bookmarks = decode_json scalar <JSON>;
close(JSON);

my $bar = $bookmarks->{roots}{bookmark_bar};

# For more restrictive folder processing
# for (@{$bar->{children}}) {          # list folders in "Bookmark Bar"
#   last if $_->{name} =~ /PRIVATE/;   # stop at this top level folder
#   do_folder( $_, 0 ) if $_->{type} eq 'folder';
# }

do_folder( $bar, 0 );

sub do_folder {
  my ( $folder, $level ) = @_;
  # setup folder here
  print STDERR "=="x$level, "> ", $folder->{name}, "\n"  if $DEBUG;

  # process folder contents
  $level++;
  for (@{$folder->{children}}) {

    if ( $_->{type} eq 'folder' ) {
      do_folder( $_, $level );
    }

    if ( $_->{type} eq 'url' ) {
      print STDERR "--"x$level, "> ", $_->{name}, "\n"  if $DEBUG;
    }

  }
  $level--;

  # finish up folder here
  print STDERR "=="x$level, "<\n"  if $DEBUG;
}

=======8<--------


-------------------------------------------------------------------------------
Simple but Horible Processing of JSON (old version)

Here I read a FireFox JSON bookmarks file as plain text, but break it into
records using '{' as the seperator.  I then process records in sequence.

I did increse/decrese tree depth basied on what I found.  Though it relied on
the fact that in FireFox place children at the end of a 'container' type
record.  Chrome Browser does not do that, so this failed with that
application.

Note this is a Kludge and very much depends on the JSON file being processed
to find and process the tree depth.

=======8<--------
my $level = 0;   # depth level within the tree at this point.

open(JSON, $json_file) || die;
$/='{';                       # Record separator (start character)

# Locate the start of my bookmarks
while( <JSON> ) {
  last if /"type":"text\/x-moz-place-separator"/;
}

# Process each book mark entry
while( <JSON> ) {
  if ( ! /"type"/  ) {  # the entry is incomplete - get more
    $_ .= <JSON>;  redo;
  }
  #
  # Process record here
  #
  my ( $title ) = /"title":"(.*?[^\\])"/;          # Title of this entry
  my ( $type ) = /"type":"text\/x-moz-place-?([^"]*)"/; # type of entry
  my ( $uri ) = /"uri":"(.*?[^\\])"/;              # any URI present?

  next if $uri =~ /^place:/;  # ignore special ones

  if ( $type eq "container" ) {
    # Process a container...
    # ...
    $level++;  # the following records are deeper
  }
  if ( $type eq "" ) {
    # Process a standard bookmark...
    # ...
  }
  # ... handle other types like "separator"

  # Look for container closures at the end of the current record
  if ( my (@count) = /]}/g ) {
    $level--;    # end of a folder
    # process end of a container
  }

  if ( $level < 0 ) {
    # Too many container Closures
  }
}
=======8<--------

-------------------------------------------------------------------------------