"Fossies" - the Fresh Open Source Software Archive

Member "salt-2018.3.1/salt/output/highstate.py" (6 Jun 2018, 21722 Bytes) of package /linux/misc/salt-2018.3.1.tar.gz:


As a special service "Fossies" has tried to format the requested source page into HTML format using (guessed) Python source code syntax highlighting (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file. For more information about "highstate.py" see the Fossies "Dox" file reference documentation and the last Fossies "Diffs" side-by-side code changes report: 2017.7.5_vs_2018.3.0.

    1 # -*- coding: utf-8 -*-
    2 '''
    3 Outputter for displaying results of state runs
    4 ==============================================
    5 
    6 The return data from the Highstate command is a standard data structure
    7 which is parsed by the highstate outputter to deliver a clean and readable
    8 set of information about the HighState run on minions.
    9 
   10 Two configurations can be set to modify the highstate outputter. These values
   11 can be set in the master config to change the output of the ``salt`` command or
   12 set in the minion config to change the output of the ``salt-call`` command.
   13 
   14 state_verbose:
   15     By default `state_verbose` is set to `True`, setting this to `False` will
   16     instruct the highstate outputter to omit displaying anything in green, this
   17     means that nothing with a result of True and no changes will not be printed
   18 state_output:
   19     The highstate outputter has six output modes,
   20     ``full``, ``terse``, ``mixed``, ``changes`` and ``filter``
   21     * The default is set to ``full``, which will display many lines of detailed
   22       information for each executed chunk.
   23     * If ``terse`` is used, then the output is greatly simplified and shown in
   24       only one line.
   25     * If ``mixed`` is used, then terse output will be used unless a state
   26       failed, in which case full output will be used.
   27     * If ``changes`` is used, then terse output will be used if there was no
   28       error and no changes, otherwise full output will be used.
   29     * If ``filter`` is used, then either or both of two different filters can be
   30       used: ``exclude`` or ``terse``.
   31         * for ``exclude``, state.highstate expects a list of states to be excluded (or ``None``)
   32           followed by ``True`` for terse output or ``False`` for regular output.
   33           Because of parsing nuances, if only one of these is used, it must still
   34           contain a comma. For instance: `exclude=True,`.
   35         * for ``terse``, state.highstate expects simply ``True`` or ``False``.
   36       These can be set as such from the command line, or in the Salt config as
   37       `state_output_exclude` or `state_output_terse`, respectively.
   38     The output modes have one modifier:
   39     ``full_id``, ``terse_id``, ``mixed_id``, ``changes_id`` and ``filter_id``
   40     If ``_id`` is used, then the corresponding form will be used, but the value for ``name``
   41     will be drawn from the state ID. This is useful for cases where the name
   42     value might be very long and hard to read.
   43 state_tabular:
   44     If `state_output` uses the terse output, set this to `True` for an aligned
   45     output format.  If you wish to use a custom format, this can be set to a
   46     string.
   47 
   48 Example usage:
   49 
   50 If ``state_output: filter`` is set in the configuration file:
   51 
   52 .. code-block:: bash
   53 
   54     salt '*' state.highstate exclude=None,True
   55 
   56 
   57 means to exclude no states from the highstate and turn on terse output.
   58 
   59 .. code-block:: bash
   60 
   61     salt twd state.highstate exclude=problemstate1,problemstate2,False
   62 
   63 
   64 means to exclude states ``problemstate1`` and ``problemstate2``
   65 from the highstate, and use regular output.
   66 
   67 Example output for the above highstate call when ``top.sls`` defines only
   68 one other state to apply to minion ``twd``:
   69 
   70 .. code-block:: text
   71 
   72     twd:
   73 
   74     Summary for twd
   75     ------------
   76     Succeeded: 1 (changed=1)
   77     Failed:    0
   78     ------------
   79     Total states run:     1
   80 
   81 
   82 Example output with no special settings in configuration files:
   83 
   84 .. code-block:: text
   85 
   86     myminion:
   87     ----------
   88               ID: test.ping
   89         Function: module.run
   90           Result: True
   91          Comment: Module function test.ping executed
   92          Changes:
   93                   ----------
   94                   ret:
   95                       True
   96 
   97     Summary for myminion
   98     ------------
   99     Succeeded: 1
  100     Failed:    0
  101     ------------
  102     Total:     0
  103 '''
  104 
  105 # Import python libs
  106 from __future__ import absolute_import, print_function, unicode_literals
  107 import pprint
  108 import textwrap
  109 
  110 # Import salt libs
  111 import salt.utils.color
  112 import salt.utils.data
  113 import salt.utils.stringutils
  114 import salt.output
  115 
  116 # Import 3rd-party libs
  117 from salt.ext import six
  118 
  119 import logging
  120 
  121 log = logging.getLogger(__name__)
  122 
  123 
  124 def output(data, **kwargs):  # pylint: disable=unused-argument
  125     '''
  126     The HighState Outputter is only meant to be used with the state.highstate
  127     function, or a function that returns highstate return data.
  128     '''
  129     # Discard retcode in dictionary as present in orchestrate data
  130     local_masters = [key for key in data.keys() if key.endswith('.local_master')]
  131     orchestrator_output = 'retcode' in data.keys() and len(local_masters) == 1
  132 
  133     if orchestrator_output:
  134         del data['retcode']
  135 
  136     # If additional information is passed through via the "data" dictionary to
  137     # the highstate outputter, such as "outputter" or "retcode", discard it.
  138     # We only want the state data that was passed through, if it is wrapped up
  139     # in the "data" key, as the orchestrate runner does. See Issue #31330,
  140     # pull request #27838, and pull request #27175 for more information.
  141     if 'data' in data:
  142         data = data.pop('data')
  143 
  144     ret = [
  145         _format_host(host, hostdata)[0]
  146         for host, hostdata in six.iteritems(data)
  147     ]
  148     if ret:
  149         return "\n".join(ret)
  150     log.error(
  151         'Data passed to highstate outputter is not a valid highstate return: %s',
  152         data
  153     )
  154     # We should not reach here, but if we do return empty string
  155     return ''
  156 
  157 
  158 def _format_host(host, data):
  159     host = salt.utils.data.decode(host)
  160 
  161     colors = salt.utils.color.get_colors(
  162             __opts__.get('color'),
  163             __opts__.get('color_theme'))
  164     tabular = __opts__.get('state_tabular', False)
  165     rcounts = {}
  166     rdurations = []
  167     hcolor = colors['GREEN']
  168     hstrs = []
  169     nchanges = 0
  170     strip_colors = __opts__.get('strip_colors', True)
  171 
  172     if isinstance(data, int) or isinstance(data, six.string_types):
  173         # Data in this format is from saltmod.function,
  174         # so it is always a 'change'
  175         nchanges = 1
  176         hstrs.append(('{0}    {1}{2[ENDC]}'
  177                       .format(hcolor, data, colors)))
  178         hcolor = colors['CYAN']  # Print the minion name in cyan
  179     if isinstance(data, list):
  180         # Errors have been detected, list them in RED!
  181         hcolor = colors['LIGHT_RED']
  182         hstrs.append(('    {0}Data failed to compile:{1[ENDC]}'
  183                       .format(hcolor, colors)))
  184         for err in data:
  185             if strip_colors:
  186                 err = salt.output.strip_esc_sequence(
  187                     salt.utils.data.decode(err)
  188                 )
  189             hstrs.append(('{0}----------\n    {1}{2[ENDC]}'
  190                           .format(hcolor, err, colors)))
  191     if isinstance(data, dict):
  192         # Verify that the needed data is present
  193         data_tmp = {}
  194         for tname, info in six.iteritems(data):
  195             if isinstance(info, dict) and tname is not 'changes' and info and '__run_num__' not in info:
  196                 err = ('The State execution failed to record the order '
  197                        'in which all states were executed. The state '
  198                        'return missing data is:')
  199                 hstrs.insert(0, pprint.pformat(info))
  200                 hstrs.insert(0, err)
  201             if isinstance(info, dict) and 'result' in info:
  202                 data_tmp[tname] = info
  203         data = data_tmp
  204         # Everything rendered as it should display the output
  205         for tname in sorted(
  206                 data,
  207                 key=lambda k: data[k].get('__run_num__', 0)):
  208             ret = data[tname]
  209             # Increment result counts
  210             rcounts.setdefault(ret['result'], 0)
  211             rcounts[ret['result']] += 1
  212             rduration = ret.get('duration', 0)
  213             try:
  214                 rdurations.append(float(rduration))
  215             except ValueError:
  216                 rduration, _, _ = rduration.partition(' ms')
  217                 try:
  218                     rdurations.append(float(rduration))
  219                 except ValueError:
  220                     log.error('Cannot parse a float from duration {0}'
  221                               .format(ret.get('duration', 0)))
  222 
  223             tcolor = colors['GREEN']
  224             orchestration = ret.get('__orchestration__', False)
  225             schanged, ctext = _format_changes(ret['changes'], orchestration)
  226             if not ctext and 'pchanges' in ret:
  227                 schanged, ctext = _format_changes(ret['pchanges'], orchestration)
  228             nchanges += 1 if schanged else 0
  229 
  230             # Skip this state if it was successful & diff output was requested
  231             if __opts__.get('state_output_diff', False) and \
  232                ret['result'] and not schanged:
  233                 continue
  234 
  235             # Skip this state if state_verbose is False, the result is True and
  236             # there were no changes made
  237             if not __opts__.get('state_verbose', False) and \
  238                ret['result'] and not schanged:
  239                 continue
  240 
  241             if schanged:
  242                 tcolor = colors['CYAN']
  243             if ret['result'] is False:
  244                 hcolor = colors['RED']
  245                 tcolor = colors['RED']
  246             if ret['result'] is None:
  247                 hcolor = colors['LIGHT_YELLOW']
  248                 tcolor = colors['LIGHT_YELLOW']
  249 
  250             state_output = __opts__.get('state_output', 'full').lower()
  251             comps = tname.split('_|-')
  252 
  253             if state_output.endswith('_id'):
  254                 # Swap in the ID for the name. Refs #35137
  255                 comps[2] = comps[1]
  256 
  257             if state_output.startswith('filter'):
  258                 # By default, full data is shown for all types. However, return
  259                 # data may be excluded by setting state_output_exclude to a
  260                 # comma-separated list of True, False or None, or including the
  261                 # same list with the exclude option on the command line. For
  262                 # now, this option must include a comma. For example:
  263                 #     exclude=True,
  264                 # The same functionality is also available for making return
  265                 # data terse, instead of excluding it.
  266                 cliargs = __opts__.get('arg', [])
  267                 clikwargs = {}
  268                 for item in cliargs:
  269                     if isinstance(item, dict) and '__kwarg__' in item:
  270                         clikwargs = item.copy()
  271 
  272                 exclude = clikwargs.get(
  273                     'exclude', __opts__.get('state_output_exclude', [])
  274                 )
  275                 if isinstance(exclude, six.string_types):
  276                     exclude = six.text_type(exclude).split(',')
  277 
  278                 terse = clikwargs.get(
  279                     'terse', __opts__.get('state_output_terse', [])
  280                 )
  281                 if isinstance(terse, six.string_types):
  282                     terse = six.text_type(terse).split(',')
  283 
  284                 if six.text_type(ret['result']) in terse:
  285                     msg = _format_terse(tcolor, comps, ret, colors, tabular)
  286                     hstrs.append(msg)
  287                     continue
  288                 if six.text_type(ret['result']) in exclude:
  289                     continue
  290 
  291             elif any((
  292                 state_output.startswith('terse'),
  293                 state_output.startswith('mixed') and ret['result'] is not False,  # only non-error'd
  294                 state_output.startswith('changes') and ret['result'] and not schanged  # non-error'd non-changed
  295             )):
  296                 # Print this chunk in a terse way and continue in the loop
  297                 msg = _format_terse(tcolor, comps, ret, colors, tabular)
  298                 hstrs.append(msg)
  299                 continue
  300 
  301             state_lines = [
  302                 '{tcolor}----------{colors[ENDC]}',
  303                 '    {tcolor}      ID: {comps[1]}{colors[ENDC]}',
  304                 '    {tcolor}Function: {comps[0]}.{comps[3]}{colors[ENDC]}',
  305                 '    {tcolor}  Result: {ret[result]!s}{colors[ENDC]}',
  306                 '    {tcolor} Comment: {comment}{colors[ENDC]}',
  307             ]
  308             if __opts__.get('state_output_profile', True) and 'start_time' in ret:
  309                 state_lines.extend([
  310                     '    {tcolor} Started: {ret[start_time]!s}{colors[ENDC]}',
  311                     '    {tcolor}Duration: {ret[duration]!s}{colors[ENDC]}',
  312                 ])
  313             # This isn't the prettiest way of doing this, but it's readable.
  314             if comps[1] != comps[2]:
  315                 state_lines.insert(
  316                     3, '    {tcolor}    Name: {comps[2]}{colors[ENDC]}')
  317             # be sure that ret['comment'] is utf-8 friendly
  318             try:
  319                 if not isinstance(ret['comment'], six.text_type):
  320                     ret['comment'] = six.text_type(ret['comment'])
  321             except UnicodeDecodeError:
  322                 # If we got here, we're on Python 2 and ret['comment'] somehow
  323                 # contained a str type with unicode content.
  324                 ret['comment'] = salt.utils.stringutils.to_unicode(ret['comment'])
  325             try:
  326                 comment = salt.utils.data.decode(ret['comment'])
  327                 comment = comment.strip().replace(
  328                         '\n',
  329                         '\n' + ' ' * 14)
  330             except AttributeError:  # Assume comment is a list
  331                 try:
  332                     comment = ret['comment'].join(' ').replace(
  333                         '\n',
  334                         '\n' + ' ' * 13)
  335                 except AttributeError:
  336                     # Comment isn't a list either, just convert to string
  337                     comment = six.text_type(ret['comment'])
  338                     comment = comment.strip().replace(
  339                         '\n',
  340                         '\n' + ' ' * 14)
  341             # If there is a data attribute, append it to the comment
  342             if 'data' in ret:
  343                 if isinstance(ret['data'], list):
  344                     for item in ret['data']:
  345                         comment = '{0} {1}'.format(comment, item)
  346                 elif isinstance(ret['data'], dict):
  347                     for key, value in ret['data'].items():
  348                         comment = '{0}\n\t\t{1}: {2}'.format(comment, key, value)
  349                 else:
  350                     comment = '{0} {1}'.format(comment, ret['data'])
  351             for detail in ['start_time', 'duration']:
  352                 ret.setdefault(detail, '')
  353             if ret['duration'] != '':
  354                 ret['duration'] = '{0} ms'.format(ret['duration'])
  355             svars = {
  356                 'tcolor': tcolor,
  357                 'comps': comps,
  358                 'ret': ret,
  359                 'comment': salt.utils.data.decode(comment),
  360                 # This nukes any trailing \n and indents the others.
  361                 'colors': colors
  362             }
  363             hstrs.extend([sline.format(**svars) for sline in state_lines])
  364             changes = '     Changes:   ' + ctext
  365             hstrs.append(('{0}{1}{2[ENDC]}'
  366                           .format(tcolor, changes, colors)))
  367 
  368             if 'warnings' in ret:
  369                 rcounts.setdefault('warnings', 0)
  370                 rcounts['warnings'] += 1
  371                 wrapper = textwrap.TextWrapper(
  372                     width=80,
  373                     initial_indent=' ' * 14,
  374                     subsequent_indent=' ' * 14
  375                 )
  376                 hstrs.append(
  377                     '   {colors[LIGHT_RED]} Warnings: {0}{colors[ENDC]}'.format(
  378                         wrapper.fill('\n'.join(ret['warnings'])).lstrip(),
  379                         colors=colors
  380                     )
  381                 )
  382 
  383         # Append result counts to end of output
  384         colorfmt = '{0}{1}{2[ENDC]}'
  385         rlabel = {True: 'Succeeded', False: 'Failed', None: 'Not Run', 'warnings': 'Warnings'}
  386         count_max_len = max([len(six.text_type(x)) for x in six.itervalues(rcounts)] or [0])
  387         label_max_len = max([len(x) for x in six.itervalues(rlabel)] or [0])
  388         line_max_len = label_max_len + count_max_len + 2  # +2 for ': '
  389         hstrs.append(
  390             colorfmt.format(
  391                 colors['CYAN'],
  392                 '\nSummary for {0}\n{1}'.format(host, '-' * line_max_len),
  393                 colors
  394             )
  395         )
  396 
  397         def _counts(label, count):
  398             return '{0}: {1:>{2}}'.format(
  399                 label,
  400                 count,
  401                 line_max_len - (len(label) + 2)
  402             )
  403 
  404         # Successful states
  405         changestats = []
  406         if None in rcounts and rcounts.get(None, 0) > 0:
  407             # test=True states
  408             changestats.append(
  409                 colorfmt.format(
  410                     colors['LIGHT_YELLOW'],
  411                     'unchanged={0}'.format(rcounts.get(None, 0)),
  412                     colors
  413                 )
  414             )
  415         if nchanges > 0:
  416             changestats.append(
  417                 colorfmt.format(
  418                     colors['GREEN'],
  419                     'changed={0}'.format(nchanges),
  420                     colors
  421                 )
  422             )
  423         if changestats:
  424             changestats = ' ({0})'.format(', '.join(changestats))
  425         else:
  426             changestats = ''
  427         hstrs.append(
  428             colorfmt.format(
  429                 colors['GREEN'],
  430                 _counts(
  431                     rlabel[True],
  432                     rcounts.get(True, 0) + rcounts.get(None, 0)
  433                 ),
  434                 colors
  435             ) + changestats
  436         )
  437 
  438         # Failed states
  439         num_failed = rcounts.get(False, 0)
  440         hstrs.append(
  441             colorfmt.format(
  442                 colors['RED'] if num_failed else colors['CYAN'],
  443                 _counts(rlabel[False], num_failed),
  444                 colors
  445             )
  446         )
  447 
  448         num_warnings = rcounts.get('warnings', 0)
  449         if num_warnings:
  450             hstrs.append(
  451                 colorfmt.format(
  452                     colors['LIGHT_RED'],
  453                     _counts(rlabel['warnings'], num_warnings),
  454                     colors
  455                 )
  456             )
  457         totals = '{0}\nTotal states run: {1:>{2}}'.format('-' * line_max_len,
  458                                                sum(six.itervalues(rcounts)) - rcounts.get('warnings', 0),
  459                                                line_max_len - 7)
  460         hstrs.append(colorfmt.format(colors['CYAN'], totals, colors))
  461 
  462         if __opts__.get('state_output_profile', True):
  463             sum_duration = sum(rdurations)
  464             duration_unit = 'ms'
  465             # convert to seconds if duration is 1000ms or more
  466             if sum_duration > 999:
  467                 sum_duration /= 1000
  468                 duration_unit = 's'
  469             total_duration = 'Total run time: {0} {1}'.format(
  470                 '{0:.3f}'.format(sum_duration).rjust(line_max_len - 5),
  471                 duration_unit)
  472             hstrs.append(colorfmt.format(colors['CYAN'], total_duration, colors))
  473 
  474     if strip_colors:
  475         host = salt.output.strip_esc_sequence(host)
  476     hstrs.insert(0, ('{0}{1}:{2[ENDC]}'.format(hcolor, host, colors)))
  477     return '\n'.join(hstrs), nchanges > 0
  478 
  479 
  480 def _nested_changes(changes):
  481     '''
  482     Print the changes data using the nested outputter
  483     '''
  484     ret = '\n'
  485     ret += salt.output.out_format(
  486             changes,
  487             'nested',
  488             __opts__,
  489             nested_indent=14)
  490     return ret
  491 
  492 
  493 def _format_changes(changes, orchestration=False):
  494     '''
  495     Format the changes dict based on what the data is
  496     '''
  497     if not changes:
  498         return False, ''
  499 
  500     if orchestration:
  501         return True, _nested_changes(changes)
  502 
  503     if not isinstance(changes, dict):
  504         return True, 'Invalid Changes data: {0}'.format(changes)
  505 
  506     ret = changes.get('ret')
  507     if ret is not None and changes.get('out') == 'highstate':
  508         ctext = ''
  509         changed = False
  510         for host, hostdata in six.iteritems(ret):
  511             s, c = _format_host(host, hostdata)
  512             ctext += '\n' + '\n'.join((' ' * 14 + l) for l in s.splitlines())
  513             changed = changed or c
  514     else:
  515         changed = True
  516         ctext = _nested_changes(changes)
  517     return changed, ctext
  518 
  519 
  520 def _format_terse(tcolor, comps, ret, colors, tabular):
  521     '''
  522     Terse formatting of a message.
  523     '''
  524     result = 'Clean'
  525     if ret['changes']:
  526         result = 'Changed'
  527     if ret['result'] is False:
  528         result = 'Failed'
  529     elif ret['result'] is None:
  530         result = 'Differs'
  531     if tabular is True:
  532         fmt_string = ''
  533         if 'warnings' in ret:
  534             fmt_string += '{c[LIGHT_RED]}Warnings:\n{w}{c[ENDC]}\n'.format(
  535                 c=colors, w='\n'.join(ret['warnings'])
  536             )
  537         fmt_string += '{0}'
  538         if __opts__.get('state_output_profile', True) and 'start_time' in ret:
  539             fmt_string += '{6[start_time]!s} [{6[duration]!s:>7} ms] '
  540         fmt_string += '{2:>10}.{3:<10} {4:7}   Name: {1}{5}'
  541     elif isinstance(tabular, six.string_types):
  542         fmt_string = tabular
  543     else:
  544         fmt_string = ''
  545         if 'warnings' in ret:
  546             fmt_string += '{c[LIGHT_RED]}Warnings:\n{w}{c[ENDC]}'.format(
  547                 c=colors, w='\n'.join(ret['warnings'])
  548             )
  549         fmt_string += ' {0} Name: {1} - Function: {2}.{3} - Result: {4}'
  550         if __opts__.get('state_output_profile', True) and 'start_time' in ret:
  551             fmt_string += ' Started: - {6[start_time]!s} Duration: {6[duration]!s} ms'
  552         fmt_string += '{5}'
  553 
  554     msg = fmt_string.format(tcolor,
  555                             comps[2],
  556                             comps[0],
  557                             comps[-1],
  558                             result,
  559                             colors['ENDC'],
  560                             ret)
  561     return msg