"Fossies" - the Fresh Open Source Software Archive 
Member "salt-3000.3/salt/output/highstate.py" (13 May 2020, 22088 Bytes) of package /linux/misc/salt-3000.3.tar.gz:
As a special service "Fossies" has tried to format the requested source page into HTML format using (guessed) Python source code syntax highlighting (style:
standard) with prefixed line numbers.
Alternatively you can here
view or
download the uninterpreted source code file.
For more information about "highstate.py" see the
Fossies "Dox" file reference documentation.
1 # -*- coding: utf-8 -*-
2 '''
3 Outputter for displaying results of state runs
4 ==============================================
5
6 The return data from the Highstate command is a standard data structure
7 which is parsed by the highstate outputter to deliver a clean and readable
8 set of information about the HighState run on minions.
9
10 Two configurations can be set to modify the highstate outputter. These values
11 can be set in the master config to change the output of the ``salt`` command or
12 set in the minion config to change the output of the ``salt-call`` command.
13
14 state_verbose
15 By default `state_verbose` is set to `True`, setting this to `False` will
16 instruct the highstate outputter to omit displaying anything in green, this
17 means that nothing with a result of True and no changes will not be printed
18 state_output:
19 The highstate outputter has six output modes,
20 ``full``, ``terse``, ``mixed``, ``changes`` and ``filter``
21
22 * The default is set to ``full``, which will display many lines of detailed
23 information for each executed chunk.
24
25 * If ``terse`` is used, then the output is greatly simplified and shown in
26 only one line.
27
28 * If ``mixed`` is used, then terse output will be used unless a state
29 failed, in which case full output will be used.
30
31 * If ``changes`` is used, then terse output will be used if there was no
32 error and no changes, otherwise full output will be used.
33
34 * If ``filter`` is used, then either or both of two different filters can be
35 used: ``exclude`` or ``terse``.
36
37 * for ``exclude``, state.highstate expects a list of states to be excluded (or ``None``)
38 followed by ``True`` for terse output or ``False`` for regular output.
39 Because of parsing nuances, if only one of these is used, it must still
40 contain a comma. For instance: `exclude=True,`.
41
42 * for ``terse``, state.highstate expects simply ``True`` or ``False``.
43
44 These can be set as such from the command line, or in the Salt config as
45 `state_output_exclude` or `state_output_terse`, respectively.
46
47 The output modes have one modifier:
48
49 ``full_id``, ``terse_id``, ``mixed_id``, ``changes_id`` and ``filter_id``
50 If ``_id`` is used, then the corresponding form will be used, but the value for ``name``
51 will be drawn from the state ID. This is useful for cases where the name
52 value might be very long and hard to read.
53
54 state_tabular:
55 If `state_output` uses the terse output, set this to `True` for an aligned
56 output format. If you wish to use a custom format, this can be set to a
57 string.
58
59 Example usage:
60
61 If ``state_output: filter`` is set in the configuration file:
62
63 .. code-block:: bash
64
65 salt '*' state.highstate exclude=None,True
66
67
68 means to exclude no states from the highstate and turn on terse output.
69
70 .. code-block:: bash
71
72 salt twd state.highstate exclude=problemstate1,problemstate2,False
73
74
75 means to exclude states ``problemstate1`` and ``problemstate2``
76 from the highstate, and use regular output.
77
78 Example output for the above highstate call when ``top.sls`` defines only
79 one other state to apply to minion ``twd``:
80
81 .. code-block:: text
82
83 twd:
84
85 Summary for twd
86 ------------
87 Succeeded: 1 (changed=1)
88 Failed: 0
89 ------------
90 Total states run: 1
91
92
93 Example output with no special settings in configuration files:
94
95 .. code-block:: text
96
97 myminion:
98 ----------
99 ID: test.ping
100 Function: module.run
101 Result: True
102 Comment: Module function test.ping executed
103 Changes:
104 ----------
105 ret:
106 True
107
108 Summary for myminion
109 ------------
110 Succeeded: 1
111 Failed: 0
112 ------------
113 Total: 0
114 '''
115
116 # Import python libs
117 from __future__ import absolute_import, print_function, unicode_literals
118 import pprint
119 import re
120 import textwrap
121
122 # Import salt libs
123 import salt.utils.color
124 import salt.utils.data
125 import salt.utils.stringutils
126 import salt.output
127
128 # Import 3rd-party libs
129 from salt.ext import six
130
131 import logging
132
133 log = logging.getLogger(__name__)
134
135
136 def output(data, **kwargs): # pylint: disable=unused-argument
137 '''
138 The HighState Outputter is only meant to be used with the state.highstate
139 function, or a function that returns highstate return data.
140 '''
141 # Discard retcode in dictionary as present in orchestrate data
142 local_masters = [key for key in data.keys() if key.endswith('.local_master')]
143 orchestrator_output = 'retcode' in data.keys() and len(local_masters) == 1
144
145 if orchestrator_output:
146 del data['retcode']
147
148 # If additional information is passed through via the "data" dictionary to
149 # the highstate outputter, such as "outputter" or "retcode", discard it.
150 # We only want the state data that was passed through, if it is wrapped up
151 # in the "data" key, as the orchestrate runner does. See Issue #31330,
152 # pull request #27838, and pull request #27175 for more information.
153 if 'data' in data:
154 data = data.pop('data')
155
156 indent_level = kwargs.get('indent_level', 1)
157 ret = [
158 _format_host(host, hostdata, indent_level=indent_level)[0]
159 for host, hostdata in six.iteritems(data)
160 ]
161 if ret:
162 return "\n".join(ret)
163 log.error(
164 'Data passed to highstate outputter is not a valid highstate return: %s',
165 data
166 )
167 # We should not reach here, but if we do return empty string
168 return ''
169
170
171 def _format_host(host, data, indent_level=1):
172 '''
173 Main highstate formatter. can be called recursively if a nested highstate
174 contains other highstates (ie in an orchestration)
175 '''
176 host = salt.utils.data.decode(host)
177
178 colors = salt.utils.color.get_colors(
179 __opts__.get('color'),
180 __opts__.get('color_theme'))
181 tabular = __opts__.get('state_tabular', False)
182 rcounts = {}
183 rdurations = []
184 hcolor = colors['GREEN']
185 hstrs = []
186 nchanges = 0
187 strip_colors = __opts__.get('strip_colors', True)
188
189 if isinstance(data, int) or isinstance(data, six.string_types):
190 # Data in this format is from saltmod.function,
191 # so it is always a 'change'
192 nchanges = 1
193 hstrs.append(('{0} {1}{2[ENDC]}'
194 .format(hcolor, data, colors)))
195 hcolor = colors['CYAN'] # Print the minion name in cyan
196 if isinstance(data, list):
197 # Errors have been detected, list them in RED!
198 hcolor = colors['LIGHT_RED']
199 hstrs.append((' {0}Data failed to compile:{1[ENDC]}'
200 .format(hcolor, colors)))
201 for err in data:
202 if strip_colors:
203 err = salt.output.strip_esc_sequence(
204 salt.utils.data.decode(err)
205 )
206 hstrs.append(('{0}----------\n {1}{2[ENDC]}'
207 .format(hcolor, err, colors)))
208 if isinstance(data, dict):
209 # Verify that the needed data is present
210 data_tmp = {}
211 for tname, info in six.iteritems(data):
212 if isinstance(info, dict) and tname is not 'changes' and info and '__run_num__' not in info:
213 err = ('The State execution failed to record the order '
214 'in which all states were executed. The state '
215 'return missing data is:')
216 hstrs.insert(0, pprint.pformat(info))
217 hstrs.insert(0, err)
218 if isinstance(info, dict) and 'result' in info:
219 data_tmp[tname] = info
220 data = data_tmp
221 # Everything rendered as it should display the output
222 for tname in sorted(
223 data,
224 key=lambda k: data[k].get('__run_num__', 0)):
225 ret = data[tname]
226 # Increment result counts
227 rcounts.setdefault(ret['result'], 0)
228 rcounts[ret['result']] += 1
229 rduration = ret.get('duration', 0)
230 try:
231 rdurations.append(float(rduration))
232 except ValueError:
233 rduration, _, _ = rduration.partition(' ms')
234 try:
235 rdurations.append(float(rduration))
236 except ValueError:
237 log.error('Cannot parse a float from duration %s', ret.get('duration', 0))
238
239 tcolor = colors['GREEN']
240 if ret.get('name') in ['state.orch', 'state.orchestrate', 'state.sls']:
241 nested = output(ret['changes']['return'], indent_level=indent_level+1)
242 ctext = re.sub('^', ' ' * 14 * indent_level, '\n'+nested, flags=re.MULTILINE)
243 schanged = True
244 nchanges += 1
245 else:
246 schanged, ctext = _format_changes(ret['changes'])
247 nchanges += 1 if schanged else 0
248
249 # Skip this state if it was successful & diff output was requested
250 if __opts__.get('state_output_diff', False) and \
251 ret['result'] and not schanged:
252 continue
253
254 # Skip this state if state_verbose is False, the result is True and
255 # there were no changes made
256 if not __opts__.get('state_verbose', False) and \
257 ret['result'] and not schanged:
258 continue
259
260 if schanged:
261 tcolor = colors['CYAN']
262 if ret['result'] is False:
263 hcolor = colors['RED']
264 tcolor = colors['RED']
265 if ret['result'] is None:
266 hcolor = colors['LIGHT_YELLOW']
267 tcolor = colors['LIGHT_YELLOW']
268
269 state_output = __opts__.get('state_output', 'full').lower()
270 comps = tname.split('_|-')
271
272 if state_output.endswith('_id'):
273 # Swap in the ID for the name. Refs #35137
274 comps[2] = comps[1]
275
276 if state_output.startswith('filter'):
277 # By default, full data is shown for all types. However, return
278 # data may be excluded by setting state_output_exclude to a
279 # comma-separated list of True, False or None, or including the
280 # same list with the exclude option on the command line. For
281 # now, this option must include a comma. For example:
282 # exclude=True,
283 # The same functionality is also available for making return
284 # data terse, instead of excluding it.
285 cliargs = __opts__.get('arg', [])
286 clikwargs = {}
287 for item in cliargs:
288 if isinstance(item, dict) and '__kwarg__' in item:
289 clikwargs = item.copy()
290
291 exclude = clikwargs.get(
292 'exclude', __opts__.get('state_output_exclude', [])
293 )
294 if isinstance(exclude, six.string_types):
295 exclude = six.text_type(exclude).split(',')
296
297 terse = clikwargs.get(
298 'terse', __opts__.get('state_output_terse', [])
299 )
300 if isinstance(terse, six.string_types):
301 terse = six.text_type(terse).split(',')
302
303 if six.text_type(ret['result']) in terse:
304 msg = _format_terse(tcolor, comps, ret, colors, tabular)
305 hstrs.append(msg)
306 continue
307 if six.text_type(ret['result']) in exclude:
308 continue
309
310 elif any((
311 state_output.startswith('terse'),
312 state_output.startswith('mixed') and ret['result'] is not False, # only non-error'd
313 state_output.startswith('changes') and ret['result'] and not schanged # non-error'd non-changed
314 )):
315 # Print this chunk in a terse way and continue in the loop
316 msg = _format_terse(tcolor, comps, ret, colors, tabular)
317 hstrs.append(msg)
318 continue
319
320 state_lines = [
321 '{tcolor}----------{colors[ENDC]}',
322 ' {tcolor} ID: {comps[1]}{colors[ENDC]}',
323 ' {tcolor}Function: {comps[0]}.{comps[3]}{colors[ENDC]}',
324 ' {tcolor} Result: {ret[result]!s}{colors[ENDC]}',
325 ' {tcolor} Comment: {comment}{colors[ENDC]}',
326 ]
327 if __opts__.get('state_output_profile', True) and 'start_time' in ret:
328 state_lines.extend([
329 ' {tcolor} Started: {ret[start_time]!s}{colors[ENDC]}',
330 ' {tcolor}Duration: {ret[duration]!s}{colors[ENDC]}',
331 ])
332 # This isn't the prettiest way of doing this, but it's readable.
333 if comps[1] != comps[2]:
334 state_lines.insert(
335 3, ' {tcolor} Name: {comps[2]}{colors[ENDC]}')
336 # be sure that ret['comment'] is utf-8 friendly
337 try:
338 if not isinstance(ret['comment'], six.text_type):
339 ret['comment'] = six.text_type(ret['comment'])
340 except UnicodeDecodeError:
341 # If we got here, we're on Python 2 and ret['comment'] somehow
342 # contained a str type with unicode content.
343 ret['comment'] = salt.utils.stringutils.to_unicode(ret['comment'])
344 try:
345 comment = salt.utils.data.decode(ret['comment'])
346 comment = comment.strip().replace(
347 '\n',
348 '\n' + ' ' * 14)
349 except AttributeError: # Assume comment is a list
350 try:
351 comment = ret['comment'].join(' ').replace(
352 '\n',
353 '\n' + ' ' * 13)
354 except AttributeError:
355 # Comment isn't a list either, just convert to string
356 comment = six.text_type(ret['comment'])
357 comment = comment.strip().replace(
358 '\n',
359 '\n' + ' ' * 14)
360 # If there is a data attribute, append it to the comment
361 if 'data' in ret:
362 if isinstance(ret['data'], list):
363 for item in ret['data']:
364 comment = '{0} {1}'.format(comment, item)
365 elif isinstance(ret['data'], dict):
366 for key, value in ret['data'].items():
367 comment = '{0}\n\t\t{1}: {2}'.format(comment, key, value)
368 else:
369 comment = '{0} {1}'.format(comment, ret['data'])
370 for detail in ['start_time', 'duration']:
371 ret.setdefault(detail, '')
372 if ret['duration'] != '':
373 ret['duration'] = '{0} ms'.format(ret['duration'])
374 svars = {
375 'tcolor': tcolor,
376 'comps': comps,
377 'ret': ret,
378 'comment': salt.utils.data.decode(comment),
379 # This nukes any trailing \n and indents the others.
380 'colors': colors
381 }
382 hstrs.extend([sline.format(**svars) for sline in state_lines])
383 changes = ' Changes: ' + ctext
384 hstrs.append(('{0}{1}{2[ENDC]}'
385 .format(tcolor, changes, colors)))
386
387 if 'warnings' in ret:
388 rcounts.setdefault('warnings', 0)
389 rcounts['warnings'] += 1
390 wrapper = textwrap.TextWrapper(
391 width=80,
392 initial_indent=' ' * 14,
393 subsequent_indent=' ' * 14
394 )
395 hstrs.append(
396 ' {colors[LIGHT_RED]} Warnings: {0}{colors[ENDC]}'.format(
397 wrapper.fill('\n'.join(ret['warnings'])).lstrip(),
398 colors=colors
399 )
400 )
401
402 # Append result counts to end of output
403 colorfmt = '{0}{1}{2[ENDC]}'
404 rlabel = {True: 'Succeeded', False: 'Failed', None: 'Not Run', 'warnings': 'Warnings'}
405 count_max_len = max([len(six.text_type(x)) for x in six.itervalues(rcounts)] or [0])
406 label_max_len = max([len(x) for x in six.itervalues(rlabel)] or [0])
407 line_max_len = label_max_len + count_max_len + 2 # +2 for ': '
408 hstrs.append(
409 colorfmt.format(
410 colors['CYAN'],
411 '\nSummary for {0}\n{1}'.format(host, '-' * line_max_len),
412 colors
413 )
414 )
415
416 def _counts(label, count):
417 return '{0}: {1:>{2}}'.format(
418 label,
419 count,
420 line_max_len - (len(label) + 2)
421 )
422
423 # Successful states
424 changestats = []
425 if None in rcounts and rcounts.get(None, 0) > 0:
426 # test=True states
427 changestats.append(
428 colorfmt.format(
429 colors['LIGHT_YELLOW'],
430 'unchanged={0}'.format(rcounts.get(None, 0)),
431 colors
432 )
433 )
434 if nchanges > 0:
435 changestats.append(
436 colorfmt.format(
437 colors['GREEN'],
438 'changed={0}'.format(nchanges),
439 colors
440 )
441 )
442 if changestats:
443 changestats = ' ({0})'.format(', '.join(changestats))
444 else:
445 changestats = ''
446 hstrs.append(
447 colorfmt.format(
448 colors['GREEN'],
449 _counts(
450 rlabel[True],
451 rcounts.get(True, 0) + rcounts.get(None, 0)
452 ),
453 colors
454 ) + changestats
455 )
456
457 # Failed states
458 num_failed = rcounts.get(False, 0)
459 hstrs.append(
460 colorfmt.format(
461 colors['RED'] if num_failed else colors['CYAN'],
462 _counts(rlabel[False], num_failed),
463 colors
464 )
465 )
466
467 num_warnings = rcounts.get('warnings', 0)
468 if num_warnings:
469 hstrs.append(
470 colorfmt.format(
471 colors['LIGHT_RED'],
472 _counts(rlabel['warnings'], num_warnings),
473 colors
474 )
475 )
476 totals = '{0}\nTotal states run: {1:>{2}}'.format('-' * line_max_len,
477 sum(six.itervalues(rcounts)) - rcounts.get('warnings', 0),
478 line_max_len - 7)
479 hstrs.append(colorfmt.format(colors['CYAN'], totals, colors))
480
481 if __opts__.get('state_output_profile', True):
482 sum_duration = sum(rdurations)
483 duration_unit = 'ms'
484 # convert to seconds if duration is 1000ms or more
485 if sum_duration > 999:
486 sum_duration /= 1000
487 duration_unit = 's'
488 total_duration = 'Total run time: {0} {1}'.format(
489 '{0:.3f}'.format(sum_duration).rjust(line_max_len - 5),
490 duration_unit)
491 hstrs.append(colorfmt.format(colors['CYAN'], total_duration, colors))
492
493 if strip_colors:
494 host = salt.output.strip_esc_sequence(host)
495 hstrs.insert(0, ('{0}{1}:{2[ENDC]}'.format(hcolor, host, colors)))
496 return '\n'.join(hstrs), nchanges > 0
497
498
499 def _nested_changes(changes):
500 '''
501 Print the changes data using the nested outputter
502 '''
503 ret = '\n'
504 ret += salt.output.out_format(
505 changes,
506 'nested',
507 __opts__,
508 nested_indent=14)
509 return ret
510
511
512 def _format_changes(changes, orchestration=False):
513 '''
514 Format the changes dict based on what the data is
515 '''
516 if not changes:
517 return False, ''
518
519 if orchestration:
520 return True, _nested_changes(changes)
521
522 if not isinstance(changes, dict):
523 return True, 'Invalid Changes data: {0}'.format(changes)
524
525 ret = changes.get('ret')
526 if ret is not None and changes.get('out') == 'highstate':
527 ctext = ''
528 changed = False
529 for host, hostdata in six.iteritems(ret):
530 s, c = _format_host(host, hostdata)
531 ctext += '\n' + '\n'.join((' ' * 14 + l) for l in s.splitlines())
532 changed = changed or c
533 else:
534 changed = True
535 ctext = _nested_changes(changes)
536 return changed, ctext
537
538
539 def _format_terse(tcolor, comps, ret, colors, tabular):
540 '''
541 Terse formatting of a message.
542 '''
543 result = 'Clean'
544 if ret['changes']:
545 result = 'Changed'
546 if ret['result'] is False:
547 result = 'Failed'
548 elif ret['result'] is None:
549 result = 'Differs'
550 if tabular is True:
551 fmt_string = ''
552 if 'warnings' in ret:
553 fmt_string += '{c[LIGHT_RED]}Warnings:\n{w}{c[ENDC]}\n'.format(
554 c=colors, w='\n'.join(ret['warnings'])
555 )
556 fmt_string += '{0}'
557 if __opts__.get('state_output_profile', True) and 'start_time' in ret:
558 fmt_string += '{6[start_time]!s} [{6[duration]!s:>7} ms] '
559 fmt_string += '{2:>10}.{3:<10} {4:7} Name: {1}{5}'
560 elif isinstance(tabular, six.string_types):
561 fmt_string = tabular
562 else:
563 fmt_string = ''
564 if 'warnings' in ret:
565 fmt_string += '{c[LIGHT_RED]}Warnings:\n{w}{c[ENDC]}'.format(
566 c=colors, w='\n'.join(ret['warnings'])
567 )
568 fmt_string += ' {0} Name: {1} - Function: {2}.{3} - Result: {4}'
569 if __opts__.get('state_output_profile', True) and 'start_time' in ret:
570 fmt_string += ' Started: - {6[start_time]!s} Duration: {6[duration]!s} ms'
571 fmt_string += '{5}'
572
573 msg = fmt_string.format(tcolor,
574 comps[2],
575 comps[0],
576 comps[-1],
577 result,
578 colors['ENDC'],
579 ret)
580 return msg