"Fossies" - the Fresh Open Source Software Archive

Member "salt-3000.3/salt/output/highstate.py" (13 May 2020, 22088 Bytes) of package /linux/misc/salt-3000.3.tar.gz:


As a special service "Fossies" has tried to format the requested source page into HTML format using (guessed) Python source code syntax highlighting (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file. For more information about "highstate.py" see the Fossies "Dox" file reference documentation.

    1 # -*- coding: utf-8 -*-
    2 '''
    3 Outputter for displaying results of state runs
    4 ==============================================
    5 
    6 The return data from the Highstate command is a standard data structure
    7 which is parsed by the highstate outputter to deliver a clean and readable
    8 set of information about the HighState run on minions.
    9 
   10 Two configurations can be set to modify the highstate outputter. These values
   11 can be set in the master config to change the output of the ``salt`` command or
   12 set in the minion config to change the output of the ``salt-call`` command.
   13 
   14 state_verbose
   15     By default `state_verbose` is set to `True`, setting this to `False` will
   16     instruct the highstate outputter to omit displaying anything in green, this
   17     means that nothing with a result of True and no changes will not be printed
   18 state_output:
   19     The highstate outputter has six output modes,
   20     ``full``, ``terse``, ``mixed``, ``changes`` and ``filter``
   21 
   22     * The default is set to ``full``, which will display many lines of detailed
   23       information for each executed chunk.
   24 
   25     * If ``terse`` is used, then the output is greatly simplified and shown in
   26       only one line.
   27 
   28     * If ``mixed`` is used, then terse output will be used unless a state
   29       failed, in which case full output will be used.
   30 
   31     * If ``changes`` is used, then terse output will be used if there was no
   32       error and no changes, otherwise full output will be used.
   33 
   34     * If ``filter`` is used, then either or both of two different filters can be
   35       used: ``exclude`` or ``terse``.
   36 
   37         * for ``exclude``, state.highstate expects a list of states to be excluded (or ``None``)
   38           followed by ``True`` for terse output or ``False`` for regular output.
   39           Because of parsing nuances, if only one of these is used, it must still
   40           contain a comma. For instance: `exclude=True,`.
   41 
   42         * for ``terse``, state.highstate expects simply ``True`` or ``False``.
   43 
   44       These can be set as such from the command line, or in the Salt config as
   45       `state_output_exclude` or `state_output_terse`, respectively.
   46 
   47     The output modes have one modifier:
   48 
   49     ``full_id``, ``terse_id``, ``mixed_id``, ``changes_id`` and ``filter_id``
   50     If ``_id`` is used, then the corresponding form will be used, but the value for ``name``
   51     will be drawn from the state ID. This is useful for cases where the name
   52     value might be very long and hard to read.
   53 
   54 state_tabular:
   55     If `state_output` uses the terse output, set this to `True` for an aligned
   56     output format.  If you wish to use a custom format, this can be set to a
   57     string.
   58 
   59 Example usage:
   60 
   61 If ``state_output: filter`` is set in the configuration file:
   62 
   63 .. code-block:: bash
   64 
   65     salt '*' state.highstate exclude=None,True
   66 
   67 
   68 means to exclude no states from the highstate and turn on terse output.
   69 
   70 .. code-block:: bash
   71 
   72     salt twd state.highstate exclude=problemstate1,problemstate2,False
   73 
   74 
   75 means to exclude states ``problemstate1`` and ``problemstate2``
   76 from the highstate, and use regular output.
   77 
   78 Example output for the above highstate call when ``top.sls`` defines only
   79 one other state to apply to minion ``twd``:
   80 
   81 .. code-block:: text
   82 
   83     twd:
   84 
   85     Summary for twd
   86     ------------
   87     Succeeded: 1 (changed=1)
   88     Failed:    0
   89     ------------
   90     Total states run:     1
   91 
   92 
   93 Example output with no special settings in configuration files:
   94 
   95 .. code-block:: text
   96 
   97     myminion:
   98     ----------
   99               ID: test.ping
  100         Function: module.run
  101           Result: True
  102          Comment: Module function test.ping executed
  103          Changes:
  104                   ----------
  105                   ret:
  106                       True
  107 
  108     Summary for myminion
  109     ------------
  110     Succeeded: 1
  111     Failed:    0
  112     ------------
  113     Total:     0
  114 '''
  115 
  116 # Import python libs
  117 from __future__ import absolute_import, print_function, unicode_literals
  118 import pprint
  119 import re
  120 import textwrap
  121 
  122 # Import salt libs
  123 import salt.utils.color
  124 import salt.utils.data
  125 import salt.utils.stringutils
  126 import salt.output
  127 
  128 # Import 3rd-party libs
  129 from salt.ext import six
  130 
  131 import logging
  132 
  133 log = logging.getLogger(__name__)
  134 
  135 
  136 def output(data, **kwargs):  # pylint: disable=unused-argument
  137     '''
  138     The HighState Outputter is only meant to be used with the state.highstate
  139     function, or a function that returns highstate return data.
  140     '''
  141     # Discard retcode in dictionary as present in orchestrate data
  142     local_masters = [key for key in data.keys() if key.endswith('.local_master')]
  143     orchestrator_output = 'retcode' in data.keys() and len(local_masters) == 1
  144 
  145     if orchestrator_output:
  146         del data['retcode']
  147 
  148     # If additional information is passed through via the "data" dictionary to
  149     # the highstate outputter, such as "outputter" or "retcode", discard it.
  150     # We only want the state data that was passed through, if it is wrapped up
  151     # in the "data" key, as the orchestrate runner does. See Issue #31330,
  152     # pull request #27838, and pull request #27175 for more information.
  153     if 'data' in data:
  154         data = data.pop('data')
  155 
  156     indent_level = kwargs.get('indent_level', 1)
  157     ret = [
  158         _format_host(host, hostdata, indent_level=indent_level)[0]
  159         for host, hostdata in six.iteritems(data)
  160     ]
  161     if ret:
  162         return "\n".join(ret)
  163     log.error(
  164         'Data passed to highstate outputter is not a valid highstate return: %s',
  165         data
  166     )
  167     # We should not reach here, but if we do return empty string
  168     return ''
  169 
  170 
  171 def _format_host(host, data, indent_level=1):
  172     '''
  173     Main highstate formatter. can be called recursively if a nested highstate
  174     contains other highstates (ie in an orchestration)
  175     '''
  176     host = salt.utils.data.decode(host)
  177 
  178     colors = salt.utils.color.get_colors(
  179             __opts__.get('color'),
  180             __opts__.get('color_theme'))
  181     tabular = __opts__.get('state_tabular', False)
  182     rcounts = {}
  183     rdurations = []
  184     hcolor = colors['GREEN']
  185     hstrs = []
  186     nchanges = 0
  187     strip_colors = __opts__.get('strip_colors', True)
  188 
  189     if isinstance(data, int) or isinstance(data, six.string_types):
  190         # Data in this format is from saltmod.function,
  191         # so it is always a 'change'
  192         nchanges = 1
  193         hstrs.append(('{0}    {1}{2[ENDC]}'
  194                       .format(hcolor, data, colors)))
  195         hcolor = colors['CYAN']  # Print the minion name in cyan
  196     if isinstance(data, list):
  197         # Errors have been detected, list them in RED!
  198         hcolor = colors['LIGHT_RED']
  199         hstrs.append(('    {0}Data failed to compile:{1[ENDC]}'
  200                       .format(hcolor, colors)))
  201         for err in data:
  202             if strip_colors:
  203                 err = salt.output.strip_esc_sequence(
  204                     salt.utils.data.decode(err)
  205                 )
  206             hstrs.append(('{0}----------\n    {1}{2[ENDC]}'
  207                           .format(hcolor, err, colors)))
  208     if isinstance(data, dict):
  209         # Verify that the needed data is present
  210         data_tmp = {}
  211         for tname, info in six.iteritems(data):
  212             if isinstance(info, dict) and tname is not 'changes' and info and '__run_num__' not in info:
  213                 err = ('The State execution failed to record the order '
  214                        'in which all states were executed. The state '
  215                        'return missing data is:')
  216                 hstrs.insert(0, pprint.pformat(info))
  217                 hstrs.insert(0, err)
  218             if isinstance(info, dict) and 'result' in info:
  219                 data_tmp[tname] = info
  220         data = data_tmp
  221         # Everything rendered as it should display the output
  222         for tname in sorted(
  223                 data,
  224                 key=lambda k: data[k].get('__run_num__', 0)):
  225             ret = data[tname]
  226             # Increment result counts
  227             rcounts.setdefault(ret['result'], 0)
  228             rcounts[ret['result']] += 1
  229             rduration = ret.get('duration', 0)
  230             try:
  231                 rdurations.append(float(rduration))
  232             except ValueError:
  233                 rduration, _, _ = rduration.partition(' ms')
  234                 try:
  235                     rdurations.append(float(rduration))
  236                 except ValueError:
  237                     log.error('Cannot parse a float from duration %s', ret.get('duration', 0))
  238 
  239             tcolor = colors['GREEN']
  240             if ret.get('name') in ['state.orch', 'state.orchestrate', 'state.sls']:
  241                 nested = output(ret['changes']['return'], indent_level=indent_level+1)
  242                 ctext = re.sub('^', ' ' * 14 * indent_level, '\n'+nested, flags=re.MULTILINE)
  243                 schanged = True
  244                 nchanges += 1
  245             else:
  246                 schanged, ctext = _format_changes(ret['changes'])
  247                 nchanges += 1 if schanged else 0
  248 
  249             # Skip this state if it was successful & diff output was requested
  250             if __opts__.get('state_output_diff', False) and \
  251                ret['result'] and not schanged:
  252                 continue
  253 
  254             # Skip this state if state_verbose is False, the result is True and
  255             # there were no changes made
  256             if not __opts__.get('state_verbose', False) and \
  257                ret['result'] and not schanged:
  258                 continue
  259 
  260             if schanged:
  261                 tcolor = colors['CYAN']
  262             if ret['result'] is False:
  263                 hcolor = colors['RED']
  264                 tcolor = colors['RED']
  265             if ret['result'] is None:
  266                 hcolor = colors['LIGHT_YELLOW']
  267                 tcolor = colors['LIGHT_YELLOW']
  268 
  269             state_output = __opts__.get('state_output', 'full').lower()
  270             comps = tname.split('_|-')
  271 
  272             if state_output.endswith('_id'):
  273                 # Swap in the ID for the name. Refs #35137
  274                 comps[2] = comps[1]
  275 
  276             if state_output.startswith('filter'):
  277                 # By default, full data is shown for all types. However, return
  278                 # data may be excluded by setting state_output_exclude to a
  279                 # comma-separated list of True, False or None, or including the
  280                 # same list with the exclude option on the command line. For
  281                 # now, this option must include a comma. For example:
  282                 #     exclude=True,
  283                 # The same functionality is also available for making return
  284                 # data terse, instead of excluding it.
  285                 cliargs = __opts__.get('arg', [])
  286                 clikwargs = {}
  287                 for item in cliargs:
  288                     if isinstance(item, dict) and '__kwarg__' in item:
  289                         clikwargs = item.copy()
  290 
  291                 exclude = clikwargs.get(
  292                     'exclude', __opts__.get('state_output_exclude', [])
  293                 )
  294                 if isinstance(exclude, six.string_types):
  295                     exclude = six.text_type(exclude).split(',')
  296 
  297                 terse = clikwargs.get(
  298                     'terse', __opts__.get('state_output_terse', [])
  299                 )
  300                 if isinstance(terse, six.string_types):
  301                     terse = six.text_type(terse).split(',')
  302 
  303                 if six.text_type(ret['result']) in terse:
  304                     msg = _format_terse(tcolor, comps, ret, colors, tabular)
  305                     hstrs.append(msg)
  306                     continue
  307                 if six.text_type(ret['result']) in exclude:
  308                     continue
  309 
  310             elif any((
  311                 state_output.startswith('terse'),
  312                 state_output.startswith('mixed') and ret['result'] is not False,  # only non-error'd
  313                 state_output.startswith('changes') and ret['result'] and not schanged  # non-error'd non-changed
  314             )):
  315                 # Print this chunk in a terse way and continue in the loop
  316                 msg = _format_terse(tcolor, comps, ret, colors, tabular)
  317                 hstrs.append(msg)
  318                 continue
  319 
  320             state_lines = [
  321                 '{tcolor}----------{colors[ENDC]}',
  322                 '    {tcolor}      ID: {comps[1]}{colors[ENDC]}',
  323                 '    {tcolor}Function: {comps[0]}.{comps[3]}{colors[ENDC]}',
  324                 '    {tcolor}  Result: {ret[result]!s}{colors[ENDC]}',
  325                 '    {tcolor} Comment: {comment}{colors[ENDC]}',
  326             ]
  327             if __opts__.get('state_output_profile', True) and 'start_time' in ret:
  328                 state_lines.extend([
  329                     '    {tcolor} Started: {ret[start_time]!s}{colors[ENDC]}',
  330                     '    {tcolor}Duration: {ret[duration]!s}{colors[ENDC]}',
  331                 ])
  332             # This isn't the prettiest way of doing this, but it's readable.
  333             if comps[1] != comps[2]:
  334                 state_lines.insert(
  335                     3, '    {tcolor}    Name: {comps[2]}{colors[ENDC]}')
  336             # be sure that ret['comment'] is utf-8 friendly
  337             try:
  338                 if not isinstance(ret['comment'], six.text_type):
  339                     ret['comment'] = six.text_type(ret['comment'])
  340             except UnicodeDecodeError:
  341                 # If we got here, we're on Python 2 and ret['comment'] somehow
  342                 # contained a str type with unicode content.
  343                 ret['comment'] = salt.utils.stringutils.to_unicode(ret['comment'])
  344             try:
  345                 comment = salt.utils.data.decode(ret['comment'])
  346                 comment = comment.strip().replace(
  347                         '\n',
  348                         '\n' + ' ' * 14)
  349             except AttributeError:  # Assume comment is a list
  350                 try:
  351                     comment = ret['comment'].join(' ').replace(
  352                         '\n',
  353                         '\n' + ' ' * 13)
  354                 except AttributeError:
  355                     # Comment isn't a list either, just convert to string
  356                     comment = six.text_type(ret['comment'])
  357                     comment = comment.strip().replace(
  358                         '\n',
  359                         '\n' + ' ' * 14)
  360             # If there is a data attribute, append it to the comment
  361             if 'data' in ret:
  362                 if isinstance(ret['data'], list):
  363                     for item in ret['data']:
  364                         comment = '{0} {1}'.format(comment, item)
  365                 elif isinstance(ret['data'], dict):
  366                     for key, value in ret['data'].items():
  367                         comment = '{0}\n\t\t{1}: {2}'.format(comment, key, value)
  368                 else:
  369                     comment = '{0} {1}'.format(comment, ret['data'])
  370             for detail in ['start_time', 'duration']:
  371                 ret.setdefault(detail, '')
  372             if ret['duration'] != '':
  373                 ret['duration'] = '{0} ms'.format(ret['duration'])
  374             svars = {
  375                 'tcolor': tcolor,
  376                 'comps': comps,
  377                 'ret': ret,
  378                 'comment': salt.utils.data.decode(comment),
  379                 # This nukes any trailing \n and indents the others.
  380                 'colors': colors
  381             }
  382             hstrs.extend([sline.format(**svars) for sline in state_lines])
  383             changes = '     Changes:   ' + ctext
  384             hstrs.append(('{0}{1}{2[ENDC]}'
  385                           .format(tcolor, changes, colors)))
  386 
  387             if 'warnings' in ret:
  388                 rcounts.setdefault('warnings', 0)
  389                 rcounts['warnings'] += 1
  390                 wrapper = textwrap.TextWrapper(
  391                     width=80,
  392                     initial_indent=' ' * 14,
  393                     subsequent_indent=' ' * 14
  394                 )
  395                 hstrs.append(
  396                     '   {colors[LIGHT_RED]} Warnings: {0}{colors[ENDC]}'.format(
  397                         wrapper.fill('\n'.join(ret['warnings'])).lstrip(),
  398                         colors=colors
  399                     )
  400                 )
  401 
  402         # Append result counts to end of output
  403         colorfmt = '{0}{1}{2[ENDC]}'
  404         rlabel = {True: 'Succeeded', False: 'Failed', None: 'Not Run', 'warnings': 'Warnings'}
  405         count_max_len = max([len(six.text_type(x)) for x in six.itervalues(rcounts)] or [0])
  406         label_max_len = max([len(x) for x in six.itervalues(rlabel)] or [0])
  407         line_max_len = label_max_len + count_max_len + 2  # +2 for ': '
  408         hstrs.append(
  409             colorfmt.format(
  410                 colors['CYAN'],
  411                 '\nSummary for {0}\n{1}'.format(host, '-' * line_max_len),
  412                 colors
  413             )
  414         )
  415 
  416         def _counts(label, count):
  417             return '{0}: {1:>{2}}'.format(
  418                 label,
  419                 count,
  420                 line_max_len - (len(label) + 2)
  421             )
  422 
  423         # Successful states
  424         changestats = []
  425         if None in rcounts and rcounts.get(None, 0) > 0:
  426             # test=True states
  427             changestats.append(
  428                 colorfmt.format(
  429                     colors['LIGHT_YELLOW'],
  430                     'unchanged={0}'.format(rcounts.get(None, 0)),
  431                     colors
  432                 )
  433             )
  434         if nchanges > 0:
  435             changestats.append(
  436                 colorfmt.format(
  437                     colors['GREEN'],
  438                     'changed={0}'.format(nchanges),
  439                     colors
  440                 )
  441             )
  442         if changestats:
  443             changestats = ' ({0})'.format(', '.join(changestats))
  444         else:
  445             changestats = ''
  446         hstrs.append(
  447             colorfmt.format(
  448                 colors['GREEN'],
  449                 _counts(
  450                     rlabel[True],
  451                     rcounts.get(True, 0) + rcounts.get(None, 0)
  452                 ),
  453                 colors
  454             ) + changestats
  455         )
  456 
  457         # Failed states
  458         num_failed = rcounts.get(False, 0)
  459         hstrs.append(
  460             colorfmt.format(
  461                 colors['RED'] if num_failed else colors['CYAN'],
  462                 _counts(rlabel[False], num_failed),
  463                 colors
  464             )
  465         )
  466 
  467         num_warnings = rcounts.get('warnings', 0)
  468         if num_warnings:
  469             hstrs.append(
  470                 colorfmt.format(
  471                     colors['LIGHT_RED'],
  472                     _counts(rlabel['warnings'], num_warnings),
  473                     colors
  474                 )
  475             )
  476         totals = '{0}\nTotal states run: {1:>{2}}'.format('-' * line_max_len,
  477                                                sum(six.itervalues(rcounts)) - rcounts.get('warnings', 0),
  478                                                line_max_len - 7)
  479         hstrs.append(colorfmt.format(colors['CYAN'], totals, colors))
  480 
  481         if __opts__.get('state_output_profile', True):
  482             sum_duration = sum(rdurations)
  483             duration_unit = 'ms'
  484             # convert to seconds if duration is 1000ms or more
  485             if sum_duration > 999:
  486                 sum_duration /= 1000
  487                 duration_unit = 's'
  488             total_duration = 'Total run time: {0} {1}'.format(
  489                 '{0:.3f}'.format(sum_duration).rjust(line_max_len - 5),
  490                 duration_unit)
  491             hstrs.append(colorfmt.format(colors['CYAN'], total_duration, colors))
  492 
  493     if strip_colors:
  494         host = salt.output.strip_esc_sequence(host)
  495     hstrs.insert(0, ('{0}{1}:{2[ENDC]}'.format(hcolor, host, colors)))
  496     return '\n'.join(hstrs), nchanges > 0
  497 
  498 
  499 def _nested_changes(changes):
  500     '''
  501     Print the changes data using the nested outputter
  502     '''
  503     ret = '\n'
  504     ret += salt.output.out_format(
  505             changes,
  506             'nested',
  507             __opts__,
  508             nested_indent=14)
  509     return ret
  510 
  511 
  512 def _format_changes(changes, orchestration=False):
  513     '''
  514     Format the changes dict based on what the data is
  515     '''
  516     if not changes:
  517         return False, ''
  518 
  519     if orchestration:
  520         return True, _nested_changes(changes)
  521 
  522     if not isinstance(changes, dict):
  523         return True, 'Invalid Changes data: {0}'.format(changes)
  524 
  525     ret = changes.get('ret')
  526     if ret is not None and changes.get('out') == 'highstate':
  527         ctext = ''
  528         changed = False
  529         for host, hostdata in six.iteritems(ret):
  530             s, c = _format_host(host, hostdata)
  531             ctext += '\n' + '\n'.join((' ' * 14 + l) for l in s.splitlines())
  532             changed = changed or c
  533     else:
  534         changed = True
  535         ctext = _nested_changes(changes)
  536     return changed, ctext
  537 
  538 
  539 def _format_terse(tcolor, comps, ret, colors, tabular):
  540     '''
  541     Terse formatting of a message.
  542     '''
  543     result = 'Clean'
  544     if ret['changes']:
  545         result = 'Changed'
  546     if ret['result'] is False:
  547         result = 'Failed'
  548     elif ret['result'] is None:
  549         result = 'Differs'
  550     if tabular is True:
  551         fmt_string = ''
  552         if 'warnings' in ret:
  553             fmt_string += '{c[LIGHT_RED]}Warnings:\n{w}{c[ENDC]}\n'.format(
  554                 c=colors, w='\n'.join(ret['warnings'])
  555             )
  556         fmt_string += '{0}'
  557         if __opts__.get('state_output_profile', True) and 'start_time' in ret:
  558             fmt_string += '{6[start_time]!s} [{6[duration]!s:>7} ms] '
  559         fmt_string += '{2:>10}.{3:<10} {4:7}   Name: {1}{5}'
  560     elif isinstance(tabular, six.string_types):
  561         fmt_string = tabular
  562     else:
  563         fmt_string = ''
  564         if 'warnings' in ret:
  565             fmt_string += '{c[LIGHT_RED]}Warnings:\n{w}{c[ENDC]}'.format(
  566                 c=colors, w='\n'.join(ret['warnings'])
  567             )
  568         fmt_string += ' {0} Name: {1} - Function: {2}.{3} - Result: {4}'
  569         if __opts__.get('state_output_profile', True) and 'start_time' in ret:
  570             fmt_string += ' Started: - {6[start_time]!s} Duration: {6[duration]!s} ms'
  571         fmt_string += '{5}'
  572 
  573     msg = fmt_string.format(tcolor,
  574                             comps[2],
  575                             comps[0],
  576                             comps[-1],
  577                             result,
  578                             colors['ENDC'],
  579                             ret)
  580     return msg