"Fossies" - the Fresh Open Source Software Archive 
Member "salt-2018.3.3/salt/output/highstate.py" (25 Sep 2018, 21732 Bytes) of package /linux/misc/salt-2018.3.3.tar.gz:
As a special service "Fossies" has tried to format the requested source page into HTML format using (guessed) Python source code syntax highlighting (style:
standard) with prefixed line numbers.
Alternatively you can here
view or
download the uninterpreted source code file.
For more information about "highstate.py" see the
Fossies "Dox" file reference documentation and the latest
Fossies "Diffs" side-by-side code changes report:
2018.3.2_vs_2018.3.3.
1 # -*- coding: utf-8 -*-
2 '''
3 Outputter for displaying results of state runs
4 ==============================================
5
6 The return data from the Highstate command is a standard data structure
7 which is parsed by the highstate outputter to deliver a clean and readable
8 set of information about the HighState run on minions.
9
10 Two configurations can be set to modify the highstate outputter. These values
11 can be set in the master config to change the output of the ``salt`` command or
12 set in the minion config to change the output of the ``salt-call`` command.
13
14 state_verbose
15 By default `state_verbose` is set to `True`, setting this to `False` will
16 instruct the highstate outputter to omit displaying anything in green, this
17 means that nothing with a result of True and no changes will not be printed
18 state_output:
19 The highstate outputter has six output modes,
20 ``full``, ``terse``, ``mixed``, ``changes`` and ``filter``
21
22 * The default is set to ``full``, which will display many lines of detailed
23 information for each executed chunk.
24
25 * If ``terse`` is used, then the output is greatly simplified and shown in
26 only one line.
27
28 * If ``mixed`` is used, then terse output will be used unless a state
29 failed, in which case full output will be used.
30
31 * If ``changes`` is used, then terse output will be used if there was no
32 error and no changes, otherwise full output will be used.
33
34 * If ``filter`` is used, then either or both of two different filters can be
35 used: ``exclude`` or ``terse``.
36
37 * for ``exclude``, state.highstate expects a list of states to be excluded (or ``None``)
38 followed by ``True`` for terse output or ``False`` for regular output.
39 Because of parsing nuances, if only one of these is used, it must still
40 contain a comma. For instance: `exclude=True,`.
41
42 * for ``terse``, state.highstate expects simply ``True`` or ``False``.
43
44 These can be set as such from the command line, or in the Salt config as
45 `state_output_exclude` or `state_output_terse`, respectively.
46
47 The output modes have one modifier:
48
49 ``full_id``, ``terse_id``, ``mixed_id``, ``changes_id`` and ``filter_id``
50 If ``_id`` is used, then the corresponding form will be used, but the value for ``name``
51 will be drawn from the state ID. This is useful for cases where the name
52 value might be very long and hard to read.
53
54 state_tabular:
55 If `state_output` uses the terse output, set this to `True` for an aligned
56 output format. If you wish to use a custom format, this can be set to a
57 string.
58
59 Example usage:
60
61 If ``state_output: filter`` is set in the configuration file:
62
63 .. code-block:: bash
64
65 salt '*' state.highstate exclude=None,True
66
67
68 means to exclude no states from the highstate and turn on terse output.
69
70 .. code-block:: bash
71
72 salt twd state.highstate exclude=problemstate1,problemstate2,False
73
74
75 means to exclude states ``problemstate1`` and ``problemstate2``
76 from the highstate, and use regular output.
77
78 Example output for the above highstate call when ``top.sls`` defines only
79 one other state to apply to minion ``twd``:
80
81 .. code-block:: text
82
83 twd:
84
85 Summary for twd
86 ------------
87 Succeeded: 1 (changed=1)
88 Failed: 0
89 ------------
90 Total states run: 1
91
92
93 Example output with no special settings in configuration files:
94
95 .. code-block:: text
96
97 myminion:
98 ----------
99 ID: test.ping
100 Function: module.run
101 Result: True
102 Comment: Module function test.ping executed
103 Changes:
104 ----------
105 ret:
106 True
107
108 Summary for myminion
109 ------------
110 Succeeded: 1
111 Failed: 0
112 ------------
113 Total: 0
114 '''
115
116 # Import python libs
117 from __future__ import absolute_import, print_function, unicode_literals
118 import pprint
119 import textwrap
120
121 # Import salt libs
122 import salt.utils.color
123 import salt.utils.data
124 import salt.utils.stringutils
125 import salt.output
126
127 # Import 3rd-party libs
128 from salt.ext import six
129
130 import logging
131
132 log = logging.getLogger(__name__)
133
134
135 def output(data, **kwargs): # pylint: disable=unused-argument
136 '''
137 The HighState Outputter is only meant to be used with the state.highstate
138 function, or a function that returns highstate return data.
139 '''
140 # Discard retcode in dictionary as present in orchestrate data
141 local_masters = [key for key in data.keys() if key.endswith('.local_master')]
142 orchestrator_output = 'retcode' in data.keys() and len(local_masters) == 1
143
144 if orchestrator_output:
145 del data['retcode']
146
147 # If additional information is passed through via the "data" dictionary to
148 # the highstate outputter, such as "outputter" or "retcode", discard it.
149 # We only want the state data that was passed through, if it is wrapped up
150 # in the "data" key, as the orchestrate runner does. See Issue #31330,
151 # pull request #27838, and pull request #27175 for more information.
152 if 'data' in data:
153 data = data.pop('data')
154
155 ret = [
156 _format_host(host, hostdata)[0]
157 for host, hostdata in six.iteritems(data)
158 ]
159 if ret:
160 return "\n".join(ret)
161 log.error(
162 'Data passed to highstate outputter is not a valid highstate return: %s',
163 data
164 )
165 # We should not reach here, but if we do return empty string
166 return ''
167
168
169 def _format_host(host, data):
170 host = salt.utils.data.decode(host)
171
172 colors = salt.utils.color.get_colors(
173 __opts__.get('color'),
174 __opts__.get('color_theme'))
175 tabular = __opts__.get('state_tabular', False)
176 rcounts = {}
177 rdurations = []
178 hcolor = colors['GREEN']
179 hstrs = []
180 nchanges = 0
181 strip_colors = __opts__.get('strip_colors', True)
182
183 if isinstance(data, int) or isinstance(data, six.string_types):
184 # Data in this format is from saltmod.function,
185 # so it is always a 'change'
186 nchanges = 1
187 hstrs.append(('{0} {1}{2[ENDC]}'
188 .format(hcolor, data, colors)))
189 hcolor = colors['CYAN'] # Print the minion name in cyan
190 if isinstance(data, list):
191 # Errors have been detected, list them in RED!
192 hcolor = colors['LIGHT_RED']
193 hstrs.append((' {0}Data failed to compile:{1[ENDC]}'
194 .format(hcolor, colors)))
195 for err in data:
196 if strip_colors:
197 err = salt.output.strip_esc_sequence(
198 salt.utils.data.decode(err)
199 )
200 hstrs.append(('{0}----------\n {1}{2[ENDC]}'
201 .format(hcolor, err, colors)))
202 if isinstance(data, dict):
203 # Verify that the needed data is present
204 data_tmp = {}
205 for tname, info in six.iteritems(data):
206 if isinstance(info, dict) and tname is not 'changes' and info and '__run_num__' not in info:
207 err = ('The State execution failed to record the order '
208 'in which all states were executed. The state '
209 'return missing data is:')
210 hstrs.insert(0, pprint.pformat(info))
211 hstrs.insert(0, err)
212 if isinstance(info, dict) and 'result' in info:
213 data_tmp[tname] = info
214 data = data_tmp
215 # Everything rendered as it should display the output
216 for tname in sorted(
217 data,
218 key=lambda k: data[k].get('__run_num__', 0)):
219 ret = data[tname]
220 # Increment result counts
221 rcounts.setdefault(ret['result'], 0)
222 rcounts[ret['result']] += 1
223 rduration = ret.get('duration', 0)
224 try:
225 rdurations.append(float(rduration))
226 except ValueError:
227 rduration, _, _ = rduration.partition(' ms')
228 try:
229 rdurations.append(float(rduration))
230 except ValueError:
231 log.error('Cannot parse a float from duration {0}'
232 .format(ret.get('duration', 0)))
233
234 tcolor = colors['GREEN']
235 orchestration = ret.get('__orchestration__', False)
236 schanged, ctext = _format_changes(ret['changes'], orchestration)
237 if not ctext and 'pchanges' in ret:
238 schanged, ctext = _format_changes(ret['pchanges'], orchestration)
239 nchanges += 1 if schanged else 0
240
241 # Skip this state if it was successful & diff output was requested
242 if __opts__.get('state_output_diff', False) and \
243 ret['result'] and not schanged:
244 continue
245
246 # Skip this state if state_verbose is False, the result is True and
247 # there were no changes made
248 if not __opts__.get('state_verbose', False) and \
249 ret['result'] and not schanged:
250 continue
251
252 if schanged:
253 tcolor = colors['CYAN']
254 if ret['result'] is False:
255 hcolor = colors['RED']
256 tcolor = colors['RED']
257 if ret['result'] is None:
258 hcolor = colors['LIGHT_YELLOW']
259 tcolor = colors['LIGHT_YELLOW']
260
261 state_output = __opts__.get('state_output', 'full').lower()
262 comps = tname.split('_|-')
263
264 if state_output.endswith('_id'):
265 # Swap in the ID for the name. Refs #35137
266 comps[2] = comps[1]
267
268 if state_output.startswith('filter'):
269 # By default, full data is shown for all types. However, return
270 # data may be excluded by setting state_output_exclude to a
271 # comma-separated list of True, False or None, or including the
272 # same list with the exclude option on the command line. For
273 # now, this option must include a comma. For example:
274 # exclude=True,
275 # The same functionality is also available for making return
276 # data terse, instead of excluding it.
277 cliargs = __opts__.get('arg', [])
278 clikwargs = {}
279 for item in cliargs:
280 if isinstance(item, dict) and '__kwarg__' in item:
281 clikwargs = item.copy()
282
283 exclude = clikwargs.get(
284 'exclude', __opts__.get('state_output_exclude', [])
285 )
286 if isinstance(exclude, six.string_types):
287 exclude = six.text_type(exclude).split(',')
288
289 terse = clikwargs.get(
290 'terse', __opts__.get('state_output_terse', [])
291 )
292 if isinstance(terse, six.string_types):
293 terse = six.text_type(terse).split(',')
294
295 if six.text_type(ret['result']) in terse:
296 msg = _format_terse(tcolor, comps, ret, colors, tabular)
297 hstrs.append(msg)
298 continue
299 if six.text_type(ret['result']) in exclude:
300 continue
301
302 elif any((
303 state_output.startswith('terse'),
304 state_output.startswith('mixed') and ret['result'] is not False, # only non-error'd
305 state_output.startswith('changes') and ret['result'] and not schanged # non-error'd non-changed
306 )):
307 # Print this chunk in a terse way and continue in the loop
308 msg = _format_terse(tcolor, comps, ret, colors, tabular)
309 hstrs.append(msg)
310 continue
311
312 state_lines = [
313 '{tcolor}----------{colors[ENDC]}',
314 ' {tcolor} ID: {comps[1]}{colors[ENDC]}',
315 ' {tcolor}Function: {comps[0]}.{comps[3]}{colors[ENDC]}',
316 ' {tcolor} Result: {ret[result]!s}{colors[ENDC]}',
317 ' {tcolor} Comment: {comment}{colors[ENDC]}',
318 ]
319 if __opts__.get('state_output_profile', True) and 'start_time' in ret:
320 state_lines.extend([
321 ' {tcolor} Started: {ret[start_time]!s}{colors[ENDC]}',
322 ' {tcolor}Duration: {ret[duration]!s}{colors[ENDC]}',
323 ])
324 # This isn't the prettiest way of doing this, but it's readable.
325 if comps[1] != comps[2]:
326 state_lines.insert(
327 3, ' {tcolor} Name: {comps[2]}{colors[ENDC]}')
328 # be sure that ret['comment'] is utf-8 friendly
329 try:
330 if not isinstance(ret['comment'], six.text_type):
331 ret['comment'] = six.text_type(ret['comment'])
332 except UnicodeDecodeError:
333 # If we got here, we're on Python 2 and ret['comment'] somehow
334 # contained a str type with unicode content.
335 ret['comment'] = salt.utils.stringutils.to_unicode(ret['comment'])
336 try:
337 comment = salt.utils.data.decode(ret['comment'])
338 comment = comment.strip().replace(
339 '\n',
340 '\n' + ' ' * 14)
341 except AttributeError: # Assume comment is a list
342 try:
343 comment = ret['comment'].join(' ').replace(
344 '\n',
345 '\n' + ' ' * 13)
346 except AttributeError:
347 # Comment isn't a list either, just convert to string
348 comment = six.text_type(ret['comment'])
349 comment = comment.strip().replace(
350 '\n',
351 '\n' + ' ' * 14)
352 # If there is a data attribute, append it to the comment
353 if 'data' in ret:
354 if isinstance(ret['data'], list):
355 for item in ret['data']:
356 comment = '{0} {1}'.format(comment, item)
357 elif isinstance(ret['data'], dict):
358 for key, value in ret['data'].items():
359 comment = '{0}\n\t\t{1}: {2}'.format(comment, key, value)
360 else:
361 comment = '{0} {1}'.format(comment, ret['data'])
362 for detail in ['start_time', 'duration']:
363 ret.setdefault(detail, '')
364 if ret['duration'] != '':
365 ret['duration'] = '{0} ms'.format(ret['duration'])
366 svars = {
367 'tcolor': tcolor,
368 'comps': comps,
369 'ret': ret,
370 'comment': salt.utils.data.decode(comment),
371 # This nukes any trailing \n and indents the others.
372 'colors': colors
373 }
374 hstrs.extend([sline.format(**svars) for sline in state_lines])
375 changes = ' Changes: ' + ctext
376 hstrs.append(('{0}{1}{2[ENDC]}'
377 .format(tcolor, changes, colors)))
378
379 if 'warnings' in ret:
380 rcounts.setdefault('warnings', 0)
381 rcounts['warnings'] += 1
382 wrapper = textwrap.TextWrapper(
383 width=80,
384 initial_indent=' ' * 14,
385 subsequent_indent=' ' * 14
386 )
387 hstrs.append(
388 ' {colors[LIGHT_RED]} Warnings: {0}{colors[ENDC]}'.format(
389 wrapper.fill('\n'.join(ret['warnings'])).lstrip(),
390 colors=colors
391 )
392 )
393
394 # Append result counts to end of output
395 colorfmt = '{0}{1}{2[ENDC]}'
396 rlabel = {True: 'Succeeded', False: 'Failed', None: 'Not Run', 'warnings': 'Warnings'}
397 count_max_len = max([len(six.text_type(x)) for x in six.itervalues(rcounts)] or [0])
398 label_max_len = max([len(x) for x in six.itervalues(rlabel)] or [0])
399 line_max_len = label_max_len + count_max_len + 2 # +2 for ': '
400 hstrs.append(
401 colorfmt.format(
402 colors['CYAN'],
403 '\nSummary for {0}\n{1}'.format(host, '-' * line_max_len),
404 colors
405 )
406 )
407
408 def _counts(label, count):
409 return '{0}: {1:>{2}}'.format(
410 label,
411 count,
412 line_max_len - (len(label) + 2)
413 )
414
415 # Successful states
416 changestats = []
417 if None in rcounts and rcounts.get(None, 0) > 0:
418 # test=True states
419 changestats.append(
420 colorfmt.format(
421 colors['LIGHT_YELLOW'],
422 'unchanged={0}'.format(rcounts.get(None, 0)),
423 colors
424 )
425 )
426 if nchanges > 0:
427 changestats.append(
428 colorfmt.format(
429 colors['GREEN'],
430 'changed={0}'.format(nchanges),
431 colors
432 )
433 )
434 if changestats:
435 changestats = ' ({0})'.format(', '.join(changestats))
436 else:
437 changestats = ''
438 hstrs.append(
439 colorfmt.format(
440 colors['GREEN'],
441 _counts(
442 rlabel[True],
443 rcounts.get(True, 0) + rcounts.get(None, 0)
444 ),
445 colors
446 ) + changestats
447 )
448
449 # Failed states
450 num_failed = rcounts.get(False, 0)
451 hstrs.append(
452 colorfmt.format(
453 colors['RED'] if num_failed else colors['CYAN'],
454 _counts(rlabel[False], num_failed),
455 colors
456 )
457 )
458
459 num_warnings = rcounts.get('warnings', 0)
460 if num_warnings:
461 hstrs.append(
462 colorfmt.format(
463 colors['LIGHT_RED'],
464 _counts(rlabel['warnings'], num_warnings),
465 colors
466 )
467 )
468 totals = '{0}\nTotal states run: {1:>{2}}'.format('-' * line_max_len,
469 sum(six.itervalues(rcounts)) - rcounts.get('warnings', 0),
470 line_max_len - 7)
471 hstrs.append(colorfmt.format(colors['CYAN'], totals, colors))
472
473 if __opts__.get('state_output_profile', True):
474 sum_duration = sum(rdurations)
475 duration_unit = 'ms'
476 # convert to seconds if duration is 1000ms or more
477 if sum_duration > 999:
478 sum_duration /= 1000
479 duration_unit = 's'
480 total_duration = 'Total run time: {0} {1}'.format(
481 '{0:.3f}'.format(sum_duration).rjust(line_max_len - 5),
482 duration_unit)
483 hstrs.append(colorfmt.format(colors['CYAN'], total_duration, colors))
484
485 if strip_colors:
486 host = salt.output.strip_esc_sequence(host)
487 hstrs.insert(0, ('{0}{1}:{2[ENDC]}'.format(hcolor, host, colors)))
488 return '\n'.join(hstrs), nchanges > 0
489
490
491 def _nested_changes(changes):
492 '''
493 Print the changes data using the nested outputter
494 '''
495 ret = '\n'
496 ret += salt.output.out_format(
497 changes,
498 'nested',
499 __opts__,
500 nested_indent=14)
501 return ret
502
503
504 def _format_changes(changes, orchestration=False):
505 '''
506 Format the changes dict based on what the data is
507 '''
508 if not changes:
509 return False, ''
510
511 if orchestration:
512 return True, _nested_changes(changes)
513
514 if not isinstance(changes, dict):
515 return True, 'Invalid Changes data: {0}'.format(changes)
516
517 ret = changes.get('ret')
518 if ret is not None and changes.get('out') == 'highstate':
519 ctext = ''
520 changed = False
521 for host, hostdata in six.iteritems(ret):
522 s, c = _format_host(host, hostdata)
523 ctext += '\n' + '\n'.join((' ' * 14 + l) for l in s.splitlines())
524 changed = changed or c
525 else:
526 changed = True
527 ctext = _nested_changes(changes)
528 return changed, ctext
529
530
531 def _format_terse(tcolor, comps, ret, colors, tabular):
532 '''
533 Terse formatting of a message.
534 '''
535 result = 'Clean'
536 if ret['changes']:
537 result = 'Changed'
538 if ret['result'] is False:
539 result = 'Failed'
540 elif ret['result'] is None:
541 result = 'Differs'
542 if tabular is True:
543 fmt_string = ''
544 if 'warnings' in ret:
545 fmt_string += '{c[LIGHT_RED]}Warnings:\n{w}{c[ENDC]}\n'.format(
546 c=colors, w='\n'.join(ret['warnings'])
547 )
548 fmt_string += '{0}'
549 if __opts__.get('state_output_profile', True) and 'start_time' in ret:
550 fmt_string += '{6[start_time]!s} [{6[duration]!s:>7} ms] '
551 fmt_string += '{2:>10}.{3:<10} {4:7} Name: {1}{5}'
552 elif isinstance(tabular, six.string_types):
553 fmt_string = tabular
554 else:
555 fmt_string = ''
556 if 'warnings' in ret:
557 fmt_string += '{c[LIGHT_RED]}Warnings:\n{w}{c[ENDC]}'.format(
558 c=colors, w='\n'.join(ret['warnings'])
559 )
560 fmt_string += ' {0} Name: {1} - Function: {2}.{3} - Result: {4}'
561 if __opts__.get('state_output_profile', True) and 'start_time' in ret:
562 fmt_string += ' Started: - {6[start_time]!s} Duration: {6[duration]!s} ms'
563 fmt_string += '{5}'
564
565 msg = fmt_string.format(tcolor,
566 comps[2],
567 comps[0],
568 comps[-1],
569 result,
570 colors['ENDC'],
571 ret)
572 return msg