"Fossies" - the Fresh Open Source Software Archive 
Member "salt-3003/salt/output/highstate.py" (24 Mar 2021, 22163 Bytes) of package /linux/misc/salt-3003.tar.gz:
As a special service "Fossies" has tried to format the requested source page into HTML format using (guessed) Python source code syntax highlighting (style:
standard) with prefixed line numbers.
Alternatively you can here
view or
download the uninterpreted source code file.
For more information about "highstate.py" see the
Fossies "Dox" file reference documentation and the latest
Fossies "Diffs" side-by-side code changes report:
3002.6_vs_3003.
1 """
2 Outputter for displaying results of state runs
3 ==============================================
4
5 The return data from the Highstate command is a standard data structure
6 which is parsed by the highstate outputter to deliver a clean and readable
7 set of information about the HighState run on minions.
8
9 Two configurations can be set to modify the highstate outputter. These values
10 can be set in the master config to change the output of the ``salt`` command or
11 set in the minion config to change the output of the ``salt-call`` command.
12
13 state_verbose
14 By default `state_verbose` is set to `True`, setting this to `False` will
15 instruct the highstate outputter to omit displaying anything in green, this
16 means that nothing with a result of True and no changes will not be printed
17 state_output:
18 The highstate outputter has six output modes,
19 ``full``, ``terse``, ``mixed``, ``changes`` and ``filter``
20
21 * The default is set to ``full``, which will display many lines of detailed
22 information for each executed chunk.
23
24 * If ``terse`` is used, then the output is greatly simplified and shown in
25 only one line.
26
27 * If ``mixed`` is used, then terse output will be used unless a state
28 failed, in which case full output will be used.
29
30 * If ``changes`` is used, then terse output will be used if there was no
31 error and no changes, otherwise full output will be used.
32
33 * If ``filter`` is used, then either or both of two different filters can be
34 used: ``exclude`` or ``terse``.
35
36 * for ``exclude``, state.highstate expects a list of states to be excluded (or ``None``)
37 followed by ``True`` for terse output or ``False`` for regular output.
38 Because of parsing nuances, if only one of these is used, it must still
39 contain a comma. For instance: `exclude=True,`.
40
41 * for ``terse``, state.highstate expects simply ``True`` or ``False``.
42
43 These can be set as such from the command line, or in the Salt config as
44 `state_output_exclude` or `state_output_terse`, respectively.
45
46 The output modes have one modifier:
47
48 ``full_id``, ``terse_id``, ``mixed_id``, ``changes_id`` and ``filter_id``
49 If ``_id`` is used, then the corresponding form will be used, but the value for ``name``
50 will be drawn from the state ID. This is useful for cases where the name
51 value might be very long and hard to read.
52
53 state_tabular:
54 If `state_output` uses the terse output, set this to `True` for an aligned
55 output format. If you wish to use a custom format, this can be set to a
56 string.
57
58 Example usage:
59
60 If ``state_output: filter`` is set in the configuration file:
61
62 .. code-block:: bash
63
64 salt '*' state.highstate exclude=None,True
65
66
67 means to exclude no states from the highstate and turn on terse output.
68
69 .. code-block:: bash
70
71 salt twd state.highstate exclude=problemstate1,problemstate2,False
72
73
74 means to exclude states ``problemstate1`` and ``problemstate2``
75 from the highstate, and use regular output.
76
77 Example output for the above highstate call when ``top.sls`` defines only
78 one other state to apply to minion ``twd``:
79
80 .. code-block:: text
81
82 twd:
83
84 Summary for twd
85 ------------
86 Succeeded: 1 (changed=1)
87 Failed: 0
88 ------------
89 Total states run: 1
90
91
92 Example output with no special settings in configuration files:
93
94 .. code-block:: text
95
96 myminion:
97 ----------
98 ID: test.ping
99 Function: module.run
100 Result: True
101 Comment: Module function test.ping executed
102 Changes:
103 ----------
104 ret:
105 True
106
107 Summary for myminion
108 ------------
109 Succeeded: 1
110 Failed: 0
111 ------------
112 Total: 0
113 """
114
115
116 import logging
117 import pprint
118 import re
119 import textwrap
120
121 import salt.output
122 import salt.utils.color
123 import salt.utils.data
124 import salt.utils.stringutils
125
126 log = logging.getLogger(__name__)
127
128
129 def output(data, **kwargs): # pylint: disable=unused-argument
130 """
131 The HighState Outputter is only meant to be used with the state.highstate
132 function, or a function that returns highstate return data.
133 """
134 # If additional information is passed through via the "data" dictionary to
135 # the highstate outputter, such as "outputter" or "retcode", discard it.
136 # We only want the state data that was passed through, if it is wrapped up
137 # in the "data" key, as the orchestrate runner does. See Issue #31330,
138 # pull request #27838, and pull request #27175 for more information.
139 # account for envelope data if being passed lookup_jid ret
140 if isinstance(data, dict) and "return" in data:
141 data = data["return"]
142
143 if isinstance(data, dict) and "data" in data:
144 data = data["data"]
145
146 # account for envelope data if being passed lookup_jid ret
147 if isinstance(data, dict) and len(data.keys()) == 1:
148 _data = next(iter(data.values()))
149
150 if isinstance(_data, dict):
151 if "jid" in _data and "fun" in _data:
152 data = _data.get("return", {}).get("data", data)
153
154 # output() is recursive, if we aren't passed a dict just return it
155 if isinstance(data, int) or isinstance(data, str):
156 return data
157
158 if data is None:
159 return "None"
160
161 # Discard retcode in dictionary as present in orchestrate data
162 local_masters = [key for key in data.keys() if key.endswith("_master")]
163 orchestrator_output = "retcode" in data.keys() and len(local_masters) == 1
164
165 if orchestrator_output:
166 del data["retcode"]
167
168 indent_level = kwargs.get("indent_level", 1)
169 ret = [
170 _format_host(host, hostdata, indent_level=indent_level)[0]
171 for host, hostdata in data.items()
172 ]
173 if ret:
174 return "\n".join(ret)
175 log.error(
176 "Data passed to highstate outputter is not a valid highstate return: %s", data
177 )
178 # We should not reach here, but if we do return empty string
179 return ""
180
181
182 def _format_host(host, data, indent_level=1):
183 """
184 Main highstate formatter. can be called recursively if a nested highstate
185 contains other highstates (ie in an orchestration)
186 """
187 host = salt.utils.data.decode(host)
188
189 colors = salt.utils.color.get_colors(
190 __opts__.get("color"), __opts__.get("color_theme")
191 )
192 tabular = __opts__.get("state_tabular", False)
193 rcounts = {}
194 rdurations = []
195 hcolor = colors["GREEN"]
196 hstrs = []
197 nchanges = 0
198 strip_colors = __opts__.get("strip_colors", True)
199
200 if isinstance(data, int):
201 nchanges = 1
202 hstrs.append("{0} {1}{2[ENDC]}".format(hcolor, data, colors))
203 hcolor = colors["CYAN"] # Print the minion name in cyan
204 elif isinstance(data, str):
205 # Data in this format is from saltmod.function,
206 # so it is always a 'change'
207 nchanges = 1
208 for data in data.splitlines():
209 hstrs.append("{0} {1}{2[ENDC]}".format(hcolor, data, colors))
210 hcolor = colors["CYAN"] # Print the minion name in cyan
211 elif isinstance(data, list):
212 # Errors have been detected, list them in RED!
213 hcolor = colors["LIGHT_RED"]
214 hstrs.append(" {0}Data failed to compile:{1[ENDC]}".format(hcolor, colors))
215 for err in data:
216 if strip_colors:
217 err = salt.output.strip_esc_sequence(salt.utils.data.decode(err))
218 hstrs.append("{0}----------\n {1}{2[ENDC]}".format(hcolor, err, colors))
219 elif isinstance(data, dict):
220 # Verify that the needed data is present
221 data_tmp = {}
222 for tname, info in data.items():
223 if (
224 isinstance(info, dict)
225 and tname != "changes"
226 and info
227 and "__run_num__" not in info
228 ):
229 err = (
230 "The State execution failed to record the order "
231 "in which all states were executed. The state "
232 "return missing data is:"
233 )
234 hstrs.insert(0, pprint.pformat(info))
235 hstrs.insert(0, err)
236 if isinstance(info, dict) and "result" in info:
237 data_tmp[tname] = info
238 data = data_tmp
239 # Everything rendered as it should display the output
240 for tname in sorted(data, key=lambda k: data[k].get("__run_num__", 0)):
241 ret = data[tname]
242 # Increment result counts
243 rcounts.setdefault(ret["result"], 0)
244 rcounts[ret["result"]] += 1
245 rduration = ret.get("duration", 0)
246 try:
247 rdurations.append(float(rduration))
248 except ValueError:
249 rduration, _, _ = rduration.partition(" ms")
250 try:
251 rdurations.append(float(rduration))
252 except ValueError:
253 log.error(
254 "Cannot parse a float from duration %s", ret.get("duration", 0)
255 )
256
257 tcolor = colors["GREEN"]
258 if ret.get("name") in ["state.orch", "state.orchestrate", "state.sls"]:
259 nested = output(ret["changes"], indent_level=indent_level + 1)
260 ctext = re.sub(
261 "^", " " * 14 * indent_level, "\n" + nested, flags=re.MULTILINE
262 )
263 schanged = True
264 nchanges += 1
265 else:
266 schanged, ctext = _format_changes(ret["changes"])
267 nchanges += 1 if schanged else 0
268
269 # Skip this state if it was successful & diff output was requested
270 if (
271 __opts__.get("state_output_diff", False)
272 and ret["result"]
273 and not schanged
274 ):
275 continue
276
277 # Skip this state if state_verbose is False, the result is True and
278 # there were no changes made
279 if (
280 not __opts__.get("state_verbose", False)
281 and ret["result"]
282 and not schanged
283 ):
284 continue
285
286 if schanged:
287 tcolor = colors["CYAN"]
288 if ret["result"] is False:
289 hcolor = colors["RED"]
290 tcolor = colors["RED"]
291 if ret["result"] is None:
292 hcolor = colors["LIGHT_YELLOW"]
293 tcolor = colors["LIGHT_YELLOW"]
294
295 state_output = __opts__.get("state_output", "full").lower()
296 comps = tname.split("_|-")
297
298 if state_output.endswith("_id"):
299 # Swap in the ID for the name. Refs #35137
300 comps[2] = comps[1]
301
302 if state_output.startswith("filter"):
303 # By default, full data is shown for all types. However, return
304 # data may be excluded by setting state_output_exclude to a
305 # comma-separated list of True, False or None, or including the
306 # same list with the exclude option on the command line. For
307 # now, this option must include a comma. For example:
308 # exclude=True,
309 # The same functionality is also available for making return
310 # data terse, instead of excluding it.
311 cliargs = __opts__.get("arg", [])
312 clikwargs = {}
313 for item in cliargs:
314 if isinstance(item, dict) and "__kwarg__" in item:
315 clikwargs = item.copy()
316
317 exclude = clikwargs.get(
318 "exclude", __opts__.get("state_output_exclude", [])
319 )
320 if isinstance(exclude, str):
321 exclude = str(exclude).split(",")
322
323 terse = clikwargs.get("terse", __opts__.get("state_output_terse", []))
324 if isinstance(terse, str):
325 terse = str(terse).split(",")
326
327 if str(ret["result"]) in terse:
328 msg = _format_terse(tcolor, comps, ret, colors, tabular)
329 hstrs.append(msg)
330 continue
331 if str(ret["result"]) in exclude:
332 continue
333
334 elif any(
335 (
336 state_output.startswith("terse"),
337 state_output.startswith("mixed")
338 and ret["result"] is not False, # only non-error'd
339 state_output.startswith("changes")
340 and ret["result"]
341 and not schanged, # non-error'd non-changed
342 )
343 ):
344 # Print this chunk in a terse way and continue in the loop
345 msg = _format_terse(tcolor, comps, ret, colors, tabular)
346 hstrs.append(msg)
347 continue
348
349 state_lines = [
350 "{tcolor}----------{colors[ENDC]}",
351 " {tcolor} ID: {comps[1]}{colors[ENDC]}",
352 " {tcolor}Function: {comps[0]}.{comps[3]}{colors[ENDC]}",
353 " {tcolor} Result: {ret[result]!s}{colors[ENDC]}",
354 " {tcolor} Comment: {comment}{colors[ENDC]}",
355 ]
356 if __opts__.get("state_output_profile") and "start_time" in ret:
357 state_lines.extend(
358 [
359 " {tcolor} Started: {ret[start_time]!s}{colors[ENDC]}",
360 " {tcolor}Duration: {ret[duration]!s}{colors[ENDC]}",
361 ]
362 )
363 # This isn't the prettiest way of doing this, but it's readable.
364 if comps[1] != comps[2]:
365 state_lines.insert(3, " {tcolor} Name: {comps[2]}{colors[ENDC]}")
366 # be sure that ret['comment'] is utf-8 friendly
367 try:
368 if not isinstance(ret["comment"], str):
369 ret["comment"] = str(ret["comment"])
370 except UnicodeDecodeError:
371 # If we got here, we're on Python 2 and ret['comment'] somehow
372 # contained a str type with unicode content.
373 ret["comment"] = salt.utils.stringutils.to_unicode(ret["comment"])
374 try:
375 comment = salt.utils.data.decode(ret["comment"])
376 comment = comment.strip().replace("\n", "\n" + " " * 14)
377 except AttributeError: # Assume comment is a list
378 try:
379 comment = ret["comment"].join(" ").replace("\n", "\n" + " " * 13)
380 except AttributeError:
381 # Comment isn't a list either, just convert to string
382 comment = str(ret["comment"])
383 comment = comment.strip().replace("\n", "\n" + " " * 14)
384 # If there is a data attribute, append it to the comment
385 if "data" in ret:
386 if isinstance(ret["data"], list):
387 for item in ret["data"]:
388 comment = "{} {}".format(comment, item)
389 elif isinstance(ret["data"], dict):
390 for key, value in ret["data"].items():
391 comment = "{}\n\t\t{}: {}".format(comment, key, value)
392 else:
393 comment = "{} {}".format(comment, ret["data"])
394 for detail in ["start_time", "duration"]:
395 ret.setdefault(detail, "")
396 if ret["duration"] != "":
397 ret["duration"] = "{} ms".format(ret["duration"])
398 svars = {
399 "tcolor": tcolor,
400 "comps": comps,
401 "ret": ret,
402 "comment": salt.utils.data.decode(comment),
403 # This nukes any trailing \n and indents the others.
404 "colors": colors,
405 }
406 hstrs.extend([sline.format(**svars) for sline in state_lines])
407 changes = " Changes: " + ctext
408 hstrs.append("{0}{1}{2[ENDC]}".format(tcolor, changes, colors))
409
410 if "warnings" in ret:
411 rcounts.setdefault("warnings", 0)
412 rcounts["warnings"] += 1
413 wrapper = textwrap.TextWrapper(
414 width=80, initial_indent=" " * 14, subsequent_indent=" " * 14
415 )
416 hstrs.append(
417 " {colors[LIGHT_RED]} Warnings: {0}{colors[ENDC]}".format(
418 wrapper.fill("\n".join(ret["warnings"])).lstrip(), colors=colors
419 )
420 )
421
422 # Append result counts to end of output
423 colorfmt = "{0}{1}{2[ENDC]}"
424 rlabel = {
425 True: "Succeeded",
426 False: "Failed",
427 None: "Not Run",
428 "warnings": "Warnings",
429 }
430 count_max_len = max([len(str(x)) for x in rcounts.values()] or [0])
431 label_max_len = max([len(x) for x in rlabel.values()] or [0])
432 line_max_len = label_max_len + count_max_len + 2 # +2 for ': '
433 hstrs.append(
434 colorfmt.format(
435 colors["CYAN"],
436 "\nSummary for {}\n{}".format(host, "-" * line_max_len),
437 colors,
438 )
439 )
440
441 def _counts(label, count):
442 return "{0}: {1:>{2}}".format(label, count, line_max_len - (len(label) + 2))
443
444 # Successful states
445 changestats = []
446 if None in rcounts and rcounts.get(None, 0) > 0:
447 # test=True states
448 changestats.append(
449 colorfmt.format(
450 colors["LIGHT_YELLOW"],
451 "unchanged={}".format(rcounts.get(None, 0)),
452 colors,
453 )
454 )
455 if nchanges > 0:
456 changestats.append(
457 colorfmt.format(colors["GREEN"], "changed={}".format(nchanges), colors)
458 )
459 if changestats:
460 changestats = " ({})".format(", ".join(changestats))
461 else:
462 changestats = ""
463 hstrs.append(
464 colorfmt.format(
465 colors["GREEN"],
466 _counts(rlabel[True], rcounts.get(True, 0) + rcounts.get(None, 0)),
467 colors,
468 )
469 + changestats
470 )
471
472 # Failed states
473 num_failed = rcounts.get(False, 0)
474 hstrs.append(
475 colorfmt.format(
476 colors["RED"] if num_failed else colors["CYAN"],
477 _counts(rlabel[False], num_failed),
478 colors,
479 )
480 )
481
482 num_warnings = rcounts.get("warnings", 0)
483 if num_warnings:
484 hstrs.append(
485 colorfmt.format(
486 colors["LIGHT_RED"],
487 _counts(rlabel["warnings"], num_warnings),
488 colors,
489 )
490 )
491 totals = "{0}\nTotal states run: {1:>{2}}".format(
492 "-" * line_max_len,
493 sum(rcounts.values()) - rcounts.get("warnings", 0),
494 line_max_len - 7,
495 )
496 hstrs.append(colorfmt.format(colors["CYAN"], totals, colors))
497
498 if __opts__.get("state_output_profile"):
499 sum_duration = sum(rdurations)
500 duration_unit = "ms"
501 # convert to seconds if duration is 1000ms or more
502 if sum_duration > 999:
503 sum_duration /= 1000
504 duration_unit = "s"
505 total_duration = "Total run time: {} {}".format(
506 "{:.3f}".format(sum_duration).rjust(line_max_len - 5), duration_unit
507 )
508 hstrs.append(colorfmt.format(colors["CYAN"], total_duration, colors))
509
510 if strip_colors:
511 host = salt.output.strip_esc_sequence(host)
512 hstrs.insert(0, ("{0}{1}:{2[ENDC]}".format(hcolor, host, colors)))
513 return "\n".join(hstrs), nchanges > 0
514
515
516 def _nested_changes(changes):
517 """
518 Print the changes data using the nested outputter
519 """
520 ret = "\n"
521 ret += salt.output.out_format(changes, "nested", __opts__, nested_indent=14)
522 return ret
523
524
525 def _format_changes(changes, orchestration=False):
526 """
527 Format the changes dict based on what the data is
528 """
529 if not changes:
530 return False, ""
531
532 if orchestration:
533 return True, _nested_changes(changes)
534
535 if not isinstance(changes, dict):
536 return True, "Invalid Changes data: {}".format(changes)
537
538 ret = changes.get("ret")
539 if ret is not None and changes.get("out") == "highstate":
540 ctext = ""
541 changed = False
542 for host, hostdata in ret.items():
543 s, c = _format_host(host, hostdata)
544 ctext += "\n" + "\n".join((" " * 14 + l) for l in s.splitlines())
545 changed = changed or c
546 else:
547 changed = True
548 ctext = _nested_changes(changes)
549 return changed, ctext
550
551
552 def _format_terse(tcolor, comps, ret, colors, tabular):
553 """
554 Terse formatting of a message.
555 """
556 result = "Clean"
557 if ret["changes"]:
558 result = "Changed"
559 if ret["result"] is False:
560 result = "Failed"
561 elif ret["result"] is None:
562 result = "Differs"
563 if tabular is True:
564 fmt_string = ""
565 if "warnings" in ret:
566 fmt_string += "{c[LIGHT_RED]}Warnings:\n{w}{c[ENDC]}\n".format(
567 c=colors, w="\n".join(ret["warnings"])
568 )
569 fmt_string += "{0}"
570 if __opts__.get("state_output_profile") and "start_time" in ret:
571 fmt_string += "{6[start_time]!s} [{6[duration]!s:>7} ms] "
572 fmt_string += "{2:>10}.{3:<10} {4:7} Name: {1}{5}"
573 elif isinstance(tabular, str):
574 fmt_string = tabular
575 else:
576 fmt_string = ""
577 if "warnings" in ret:
578 fmt_string += "{c[LIGHT_RED]}Warnings:\n{w}{c[ENDC]}".format(
579 c=colors, w="\n".join(ret["warnings"])
580 )
581 fmt_string += " {0} Name: {1} - Function: {2}.{3} - Result: {4}"
582 if __opts__.get("state_output_profile") and "start_time" in ret:
583 fmt_string += " Started: - {6[start_time]!s} Duration: {6[duration]!s} ms"
584 fmt_string += "{5}"
585
586 msg = fmt_string.format(
587 tcolor, comps[2], comps[0], comps[-1], result, colors["ENDC"], ret
588 )
589 return msg