"Fossies" - the Fresh Open Source Software Archive

Member "salt-3003/salt/output/highstate.py" (24 Mar 2021, 22163 Bytes) of package /linux/misc/salt-3003.tar.gz:


As a special service "Fossies" has tried to format the requested source page into HTML format using (guessed) Python source code syntax highlighting (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file. For more information about "highstate.py" see the Fossies "Dox" file reference documentation and the latest Fossies "Diffs" side-by-side code changes report: 3002.6_vs_3003.

    1 """
    2 Outputter for displaying results of state runs
    3 ==============================================
    4 
    5 The return data from the Highstate command is a standard data structure
    6 which is parsed by the highstate outputter to deliver a clean and readable
    7 set of information about the HighState run on minions.
    8 
    9 Two configurations can be set to modify the highstate outputter. These values
   10 can be set in the master config to change the output of the ``salt`` command or
   11 set in the minion config to change the output of the ``salt-call`` command.
   12 
   13 state_verbose
   14     By default `state_verbose` is set to `True`, setting this to `False` will
   15     instruct the highstate outputter to omit displaying anything in green, this
   16     means that nothing with a result of True and no changes will not be printed
   17 state_output:
   18     The highstate outputter has six output modes,
   19     ``full``, ``terse``, ``mixed``, ``changes`` and ``filter``
   20 
   21     * The default is set to ``full``, which will display many lines of detailed
   22       information for each executed chunk.
   23 
   24     * If ``terse`` is used, then the output is greatly simplified and shown in
   25       only one line.
   26 
   27     * If ``mixed`` is used, then terse output will be used unless a state
   28       failed, in which case full output will be used.
   29 
   30     * If ``changes`` is used, then terse output will be used if there was no
   31       error and no changes, otherwise full output will be used.
   32 
   33     * If ``filter`` is used, then either or both of two different filters can be
   34       used: ``exclude`` or ``terse``.
   35 
   36         * for ``exclude``, state.highstate expects a list of states to be excluded (or ``None``)
   37           followed by ``True`` for terse output or ``False`` for regular output.
   38           Because of parsing nuances, if only one of these is used, it must still
   39           contain a comma. For instance: `exclude=True,`.
   40 
   41         * for ``terse``, state.highstate expects simply ``True`` or ``False``.
   42 
   43       These can be set as such from the command line, or in the Salt config as
   44       `state_output_exclude` or `state_output_terse`, respectively.
   45 
   46     The output modes have one modifier:
   47 
   48     ``full_id``, ``terse_id``, ``mixed_id``, ``changes_id`` and ``filter_id``
   49     If ``_id`` is used, then the corresponding form will be used, but the value for ``name``
   50     will be drawn from the state ID. This is useful for cases where the name
   51     value might be very long and hard to read.
   52 
   53 state_tabular:
   54     If `state_output` uses the terse output, set this to `True` for an aligned
   55     output format.  If you wish to use a custom format, this can be set to a
   56     string.
   57 
   58 Example usage:
   59 
   60 If ``state_output: filter`` is set in the configuration file:
   61 
   62 .. code-block:: bash
   63 
   64     salt '*' state.highstate exclude=None,True
   65 
   66 
   67 means to exclude no states from the highstate and turn on terse output.
   68 
   69 .. code-block:: bash
   70 
   71     salt twd state.highstate exclude=problemstate1,problemstate2,False
   72 
   73 
   74 means to exclude states ``problemstate1`` and ``problemstate2``
   75 from the highstate, and use regular output.
   76 
   77 Example output for the above highstate call when ``top.sls`` defines only
   78 one other state to apply to minion ``twd``:
   79 
   80 .. code-block:: text
   81 
   82     twd:
   83 
   84     Summary for twd
   85     ------------
   86     Succeeded: 1 (changed=1)
   87     Failed:    0
   88     ------------
   89     Total states run:     1
   90 
   91 
   92 Example output with no special settings in configuration files:
   93 
   94 .. code-block:: text
   95 
   96     myminion:
   97     ----------
   98               ID: test.ping
   99         Function: module.run
  100           Result: True
  101          Comment: Module function test.ping executed
  102          Changes:
  103                   ----------
  104                   ret:
  105                       True
  106 
  107     Summary for myminion
  108     ------------
  109     Succeeded: 1
  110     Failed:    0
  111     ------------
  112     Total:     0
  113 """
  114 
  115 
  116 import logging
  117 import pprint
  118 import re
  119 import textwrap
  120 
  121 import salt.output
  122 import salt.utils.color
  123 import salt.utils.data
  124 import salt.utils.stringutils
  125 
  126 log = logging.getLogger(__name__)
  127 
  128 
  129 def output(data, **kwargs):  # pylint: disable=unused-argument
  130     """
  131     The HighState Outputter is only meant to be used with the state.highstate
  132     function, or a function that returns highstate return data.
  133     """
  134     # If additional information is passed through via the "data" dictionary to
  135     # the highstate outputter, such as "outputter" or "retcode", discard it.
  136     # We only want the state data that was passed through, if it is wrapped up
  137     # in the "data" key, as the orchestrate runner does. See Issue #31330,
  138     # pull request #27838, and pull request #27175 for more information.
  139     # account for envelope data if being passed lookup_jid ret
  140     if isinstance(data, dict) and "return" in data:
  141         data = data["return"]
  142 
  143     if isinstance(data, dict) and "data" in data:
  144         data = data["data"]
  145 
  146     # account for envelope data if being passed lookup_jid ret
  147     if isinstance(data, dict) and len(data.keys()) == 1:
  148         _data = next(iter(data.values()))
  149 
  150         if isinstance(_data, dict):
  151             if "jid" in _data and "fun" in _data:
  152                 data = _data.get("return", {}).get("data", data)
  153 
  154     # output() is recursive, if we aren't passed a dict just return it
  155     if isinstance(data, int) or isinstance(data, str):
  156         return data
  157 
  158     if data is None:
  159         return "None"
  160 
  161     # Discard retcode in dictionary as present in orchestrate data
  162     local_masters = [key for key in data.keys() if key.endswith("_master")]
  163     orchestrator_output = "retcode" in data.keys() and len(local_masters) == 1
  164 
  165     if orchestrator_output:
  166         del data["retcode"]
  167 
  168     indent_level = kwargs.get("indent_level", 1)
  169     ret = [
  170         _format_host(host, hostdata, indent_level=indent_level)[0]
  171         for host, hostdata in data.items()
  172     ]
  173     if ret:
  174         return "\n".join(ret)
  175     log.error(
  176         "Data passed to highstate outputter is not a valid highstate return: %s", data
  177     )
  178     # We should not reach here, but if we do return empty string
  179     return ""
  180 
  181 
  182 def _format_host(host, data, indent_level=1):
  183     """
  184     Main highstate formatter. can be called recursively if a nested highstate
  185     contains other highstates (ie in an orchestration)
  186     """
  187     host = salt.utils.data.decode(host)
  188 
  189     colors = salt.utils.color.get_colors(
  190         __opts__.get("color"), __opts__.get("color_theme")
  191     )
  192     tabular = __opts__.get("state_tabular", False)
  193     rcounts = {}
  194     rdurations = []
  195     hcolor = colors["GREEN"]
  196     hstrs = []
  197     nchanges = 0
  198     strip_colors = __opts__.get("strip_colors", True)
  199 
  200     if isinstance(data, int):
  201         nchanges = 1
  202         hstrs.append("{0}    {1}{2[ENDC]}".format(hcolor, data, colors))
  203         hcolor = colors["CYAN"]  # Print the minion name in cyan
  204     elif isinstance(data, str):
  205         # Data in this format is from saltmod.function,
  206         # so it is always a 'change'
  207         nchanges = 1
  208         for data in data.splitlines():
  209             hstrs.append("{0}    {1}{2[ENDC]}".format(hcolor, data, colors))
  210         hcolor = colors["CYAN"]  # Print the minion name in cyan
  211     elif isinstance(data, list):
  212         # Errors have been detected, list them in RED!
  213         hcolor = colors["LIGHT_RED"]
  214         hstrs.append("    {0}Data failed to compile:{1[ENDC]}".format(hcolor, colors))
  215         for err in data:
  216             if strip_colors:
  217                 err = salt.output.strip_esc_sequence(salt.utils.data.decode(err))
  218             hstrs.append("{0}----------\n    {1}{2[ENDC]}".format(hcolor, err, colors))
  219     elif isinstance(data, dict):
  220         # Verify that the needed data is present
  221         data_tmp = {}
  222         for tname, info in data.items():
  223             if (
  224                 isinstance(info, dict)
  225                 and tname != "changes"
  226                 and info
  227                 and "__run_num__" not in info
  228             ):
  229                 err = (
  230                     "The State execution failed to record the order "
  231                     "in which all states were executed. The state "
  232                     "return missing data is:"
  233                 )
  234                 hstrs.insert(0, pprint.pformat(info))
  235                 hstrs.insert(0, err)
  236             if isinstance(info, dict) and "result" in info:
  237                 data_tmp[tname] = info
  238         data = data_tmp
  239         # Everything rendered as it should display the output
  240         for tname in sorted(data, key=lambda k: data[k].get("__run_num__", 0)):
  241             ret = data[tname]
  242             # Increment result counts
  243             rcounts.setdefault(ret["result"], 0)
  244             rcounts[ret["result"]] += 1
  245             rduration = ret.get("duration", 0)
  246             try:
  247                 rdurations.append(float(rduration))
  248             except ValueError:
  249                 rduration, _, _ = rduration.partition(" ms")
  250                 try:
  251                     rdurations.append(float(rduration))
  252                 except ValueError:
  253                     log.error(
  254                         "Cannot parse a float from duration %s", ret.get("duration", 0)
  255                     )
  256 
  257             tcolor = colors["GREEN"]
  258             if ret.get("name") in ["state.orch", "state.orchestrate", "state.sls"]:
  259                 nested = output(ret["changes"], indent_level=indent_level + 1)
  260                 ctext = re.sub(
  261                     "^", " " * 14 * indent_level, "\n" + nested, flags=re.MULTILINE
  262                 )
  263                 schanged = True
  264                 nchanges += 1
  265             else:
  266                 schanged, ctext = _format_changes(ret["changes"])
  267                 nchanges += 1 if schanged else 0
  268 
  269             # Skip this state if it was successful & diff output was requested
  270             if (
  271                 __opts__.get("state_output_diff", False)
  272                 and ret["result"]
  273                 and not schanged
  274             ):
  275                 continue
  276 
  277             # Skip this state if state_verbose is False, the result is True and
  278             # there were no changes made
  279             if (
  280                 not __opts__.get("state_verbose", False)
  281                 and ret["result"]
  282                 and not schanged
  283             ):
  284                 continue
  285 
  286             if schanged:
  287                 tcolor = colors["CYAN"]
  288             if ret["result"] is False:
  289                 hcolor = colors["RED"]
  290                 tcolor = colors["RED"]
  291             if ret["result"] is None:
  292                 hcolor = colors["LIGHT_YELLOW"]
  293                 tcolor = colors["LIGHT_YELLOW"]
  294 
  295             state_output = __opts__.get("state_output", "full").lower()
  296             comps = tname.split("_|-")
  297 
  298             if state_output.endswith("_id"):
  299                 # Swap in the ID for the name. Refs #35137
  300                 comps[2] = comps[1]
  301 
  302             if state_output.startswith("filter"):
  303                 # By default, full data is shown for all types. However, return
  304                 # data may be excluded by setting state_output_exclude to a
  305                 # comma-separated list of True, False or None, or including the
  306                 # same list with the exclude option on the command line. For
  307                 # now, this option must include a comma. For example:
  308                 #     exclude=True,
  309                 # The same functionality is also available for making return
  310                 # data terse, instead of excluding it.
  311                 cliargs = __opts__.get("arg", [])
  312                 clikwargs = {}
  313                 for item in cliargs:
  314                     if isinstance(item, dict) and "__kwarg__" in item:
  315                         clikwargs = item.copy()
  316 
  317                 exclude = clikwargs.get(
  318                     "exclude", __opts__.get("state_output_exclude", [])
  319                 )
  320                 if isinstance(exclude, str):
  321                     exclude = str(exclude).split(",")
  322 
  323                 terse = clikwargs.get("terse", __opts__.get("state_output_terse", []))
  324                 if isinstance(terse, str):
  325                     terse = str(terse).split(",")
  326 
  327                 if str(ret["result"]) in terse:
  328                     msg = _format_terse(tcolor, comps, ret, colors, tabular)
  329                     hstrs.append(msg)
  330                     continue
  331                 if str(ret["result"]) in exclude:
  332                     continue
  333 
  334             elif any(
  335                 (
  336                     state_output.startswith("terse"),
  337                     state_output.startswith("mixed")
  338                     and ret["result"] is not False,  # only non-error'd
  339                     state_output.startswith("changes")
  340                     and ret["result"]
  341                     and not schanged,  # non-error'd non-changed
  342                 )
  343             ):
  344                 # Print this chunk in a terse way and continue in the loop
  345                 msg = _format_terse(tcolor, comps, ret, colors, tabular)
  346                 hstrs.append(msg)
  347                 continue
  348 
  349             state_lines = [
  350                 "{tcolor}----------{colors[ENDC]}",
  351                 "    {tcolor}      ID: {comps[1]}{colors[ENDC]}",
  352                 "    {tcolor}Function: {comps[0]}.{comps[3]}{colors[ENDC]}",
  353                 "    {tcolor}  Result: {ret[result]!s}{colors[ENDC]}",
  354                 "    {tcolor} Comment: {comment}{colors[ENDC]}",
  355             ]
  356             if __opts__.get("state_output_profile") and "start_time" in ret:
  357                 state_lines.extend(
  358                     [
  359                         "    {tcolor} Started: {ret[start_time]!s}{colors[ENDC]}",
  360                         "    {tcolor}Duration: {ret[duration]!s}{colors[ENDC]}",
  361                     ]
  362                 )
  363             # This isn't the prettiest way of doing this, but it's readable.
  364             if comps[1] != comps[2]:
  365                 state_lines.insert(3, "    {tcolor}    Name: {comps[2]}{colors[ENDC]}")
  366             # be sure that ret['comment'] is utf-8 friendly
  367             try:
  368                 if not isinstance(ret["comment"], str):
  369                     ret["comment"] = str(ret["comment"])
  370             except UnicodeDecodeError:
  371                 # If we got here, we're on Python 2 and ret['comment'] somehow
  372                 # contained a str type with unicode content.
  373                 ret["comment"] = salt.utils.stringutils.to_unicode(ret["comment"])
  374             try:
  375                 comment = salt.utils.data.decode(ret["comment"])
  376                 comment = comment.strip().replace("\n", "\n" + " " * 14)
  377             except AttributeError:  # Assume comment is a list
  378                 try:
  379                     comment = ret["comment"].join(" ").replace("\n", "\n" + " " * 13)
  380                 except AttributeError:
  381                     # Comment isn't a list either, just convert to string
  382                     comment = str(ret["comment"])
  383                     comment = comment.strip().replace("\n", "\n" + " " * 14)
  384             # If there is a data attribute, append it to the comment
  385             if "data" in ret:
  386                 if isinstance(ret["data"], list):
  387                     for item in ret["data"]:
  388                         comment = "{} {}".format(comment, item)
  389                 elif isinstance(ret["data"], dict):
  390                     for key, value in ret["data"].items():
  391                         comment = "{}\n\t\t{}: {}".format(comment, key, value)
  392                 else:
  393                     comment = "{} {}".format(comment, ret["data"])
  394             for detail in ["start_time", "duration"]:
  395                 ret.setdefault(detail, "")
  396             if ret["duration"] != "":
  397                 ret["duration"] = "{} ms".format(ret["duration"])
  398             svars = {
  399                 "tcolor": tcolor,
  400                 "comps": comps,
  401                 "ret": ret,
  402                 "comment": salt.utils.data.decode(comment),
  403                 # This nukes any trailing \n and indents the others.
  404                 "colors": colors,
  405             }
  406             hstrs.extend([sline.format(**svars) for sline in state_lines])
  407             changes = "     Changes:   " + ctext
  408             hstrs.append("{0}{1}{2[ENDC]}".format(tcolor, changes, colors))
  409 
  410             if "warnings" in ret:
  411                 rcounts.setdefault("warnings", 0)
  412                 rcounts["warnings"] += 1
  413                 wrapper = textwrap.TextWrapper(
  414                     width=80, initial_indent=" " * 14, subsequent_indent=" " * 14
  415                 )
  416                 hstrs.append(
  417                     "   {colors[LIGHT_RED]} Warnings: {0}{colors[ENDC]}".format(
  418                         wrapper.fill("\n".join(ret["warnings"])).lstrip(), colors=colors
  419                     )
  420                 )
  421 
  422         # Append result counts to end of output
  423         colorfmt = "{0}{1}{2[ENDC]}"
  424         rlabel = {
  425             True: "Succeeded",
  426             False: "Failed",
  427             None: "Not Run",
  428             "warnings": "Warnings",
  429         }
  430         count_max_len = max([len(str(x)) for x in rcounts.values()] or [0])
  431         label_max_len = max([len(x) for x in rlabel.values()] or [0])
  432         line_max_len = label_max_len + count_max_len + 2  # +2 for ': '
  433         hstrs.append(
  434             colorfmt.format(
  435                 colors["CYAN"],
  436                 "\nSummary for {}\n{}".format(host, "-" * line_max_len),
  437                 colors,
  438             )
  439         )
  440 
  441         def _counts(label, count):
  442             return "{0}: {1:>{2}}".format(label, count, line_max_len - (len(label) + 2))
  443 
  444         # Successful states
  445         changestats = []
  446         if None in rcounts and rcounts.get(None, 0) > 0:
  447             # test=True states
  448             changestats.append(
  449                 colorfmt.format(
  450                     colors["LIGHT_YELLOW"],
  451                     "unchanged={}".format(rcounts.get(None, 0)),
  452                     colors,
  453                 )
  454             )
  455         if nchanges > 0:
  456             changestats.append(
  457                 colorfmt.format(colors["GREEN"], "changed={}".format(nchanges), colors)
  458             )
  459         if changestats:
  460             changestats = " ({})".format(", ".join(changestats))
  461         else:
  462             changestats = ""
  463         hstrs.append(
  464             colorfmt.format(
  465                 colors["GREEN"],
  466                 _counts(rlabel[True], rcounts.get(True, 0) + rcounts.get(None, 0)),
  467                 colors,
  468             )
  469             + changestats
  470         )
  471 
  472         # Failed states
  473         num_failed = rcounts.get(False, 0)
  474         hstrs.append(
  475             colorfmt.format(
  476                 colors["RED"] if num_failed else colors["CYAN"],
  477                 _counts(rlabel[False], num_failed),
  478                 colors,
  479             )
  480         )
  481 
  482         num_warnings = rcounts.get("warnings", 0)
  483         if num_warnings:
  484             hstrs.append(
  485                 colorfmt.format(
  486                     colors["LIGHT_RED"],
  487                     _counts(rlabel["warnings"], num_warnings),
  488                     colors,
  489                 )
  490             )
  491         totals = "{0}\nTotal states run: {1:>{2}}".format(
  492             "-" * line_max_len,
  493             sum(rcounts.values()) - rcounts.get("warnings", 0),
  494             line_max_len - 7,
  495         )
  496         hstrs.append(colorfmt.format(colors["CYAN"], totals, colors))
  497 
  498         if __opts__.get("state_output_profile"):
  499             sum_duration = sum(rdurations)
  500             duration_unit = "ms"
  501             # convert to seconds if duration is 1000ms or more
  502             if sum_duration > 999:
  503                 sum_duration /= 1000
  504                 duration_unit = "s"
  505             total_duration = "Total run time: {} {}".format(
  506                 "{:.3f}".format(sum_duration).rjust(line_max_len - 5), duration_unit
  507             )
  508             hstrs.append(colorfmt.format(colors["CYAN"], total_duration, colors))
  509 
  510     if strip_colors:
  511         host = salt.output.strip_esc_sequence(host)
  512     hstrs.insert(0, ("{0}{1}:{2[ENDC]}".format(hcolor, host, colors)))
  513     return "\n".join(hstrs), nchanges > 0
  514 
  515 
  516 def _nested_changes(changes):
  517     """
  518     Print the changes data using the nested outputter
  519     """
  520     ret = "\n"
  521     ret += salt.output.out_format(changes, "nested", __opts__, nested_indent=14)
  522     return ret
  523 
  524 
  525 def _format_changes(changes, orchestration=False):
  526     """
  527     Format the changes dict based on what the data is
  528     """
  529     if not changes:
  530         return False, ""
  531 
  532     if orchestration:
  533         return True, _nested_changes(changes)
  534 
  535     if not isinstance(changes, dict):
  536         return True, "Invalid Changes data: {}".format(changes)
  537 
  538     ret = changes.get("ret")
  539     if ret is not None and changes.get("out") == "highstate":
  540         ctext = ""
  541         changed = False
  542         for host, hostdata in ret.items():
  543             s, c = _format_host(host, hostdata)
  544             ctext += "\n" + "\n".join((" " * 14 + l) for l in s.splitlines())
  545             changed = changed or c
  546     else:
  547         changed = True
  548         ctext = _nested_changes(changes)
  549     return changed, ctext
  550 
  551 
  552 def _format_terse(tcolor, comps, ret, colors, tabular):
  553     """
  554     Terse formatting of a message.
  555     """
  556     result = "Clean"
  557     if ret["changes"]:
  558         result = "Changed"
  559     if ret["result"] is False:
  560         result = "Failed"
  561     elif ret["result"] is None:
  562         result = "Differs"
  563     if tabular is True:
  564         fmt_string = ""
  565         if "warnings" in ret:
  566             fmt_string += "{c[LIGHT_RED]}Warnings:\n{w}{c[ENDC]}\n".format(
  567                 c=colors, w="\n".join(ret["warnings"])
  568             )
  569         fmt_string += "{0}"
  570         if __opts__.get("state_output_profile") and "start_time" in ret:
  571             fmt_string += "{6[start_time]!s} [{6[duration]!s:>7} ms] "
  572         fmt_string += "{2:>10}.{3:<10} {4:7}   Name: {1}{5}"
  573     elif isinstance(tabular, str):
  574         fmt_string = tabular
  575     else:
  576         fmt_string = ""
  577         if "warnings" in ret:
  578             fmt_string += "{c[LIGHT_RED]}Warnings:\n{w}{c[ENDC]}".format(
  579                 c=colors, w="\n".join(ret["warnings"])
  580             )
  581         fmt_string += " {0} Name: {1} - Function: {2}.{3} - Result: {4}"
  582         if __opts__.get("state_output_profile") and "start_time" in ret:
  583             fmt_string += " Started: - {6[start_time]!s} Duration: {6[duration]!s} ms"
  584         fmt_string += "{5}"
  585 
  586     msg = fmt_string.format(
  587         tcolor, comps[2], comps[0], comps[-1], result, colors["ENDC"], ret
  588     )
  589     return msg