"Fossies" - the Fresh Open Source Software Archive

Member "salt-2018.3.3/salt/output/highstate.py" (25 Sep 2018, 21732 Bytes) of package /linux/misc/salt-2018.3.3.tar.gz:


As a special service "Fossies" has tried to format the requested source page into HTML format using (guessed) Python source code syntax highlighting (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file. For more information about "highstate.py" see the Fossies "Dox" file reference documentation and the latest Fossies "Diffs" side-by-side code changes report: 2018.3.2_vs_2018.3.3.

    1 # -*- coding: utf-8 -*-
    2 '''
    3 Outputter for displaying results of state runs
    4 ==============================================
    5 
    6 The return data from the Highstate command is a standard data structure
    7 which is parsed by the highstate outputter to deliver a clean and readable
    8 set of information about the HighState run on minions.
    9 
   10 Two configurations can be set to modify the highstate outputter. These values
   11 can be set in the master config to change the output of the ``salt`` command or
   12 set in the minion config to change the output of the ``salt-call`` command.
   13 
   14 state_verbose
   15     By default `state_verbose` is set to `True`, setting this to `False` will
   16     instruct the highstate outputter to omit displaying anything in green, this
   17     means that nothing with a result of True and no changes will not be printed
   18 state_output:
   19     The highstate outputter has six output modes,
   20     ``full``, ``terse``, ``mixed``, ``changes`` and ``filter``
   21 
   22     * The default is set to ``full``, which will display many lines of detailed
   23       information for each executed chunk.
   24 
   25     * If ``terse`` is used, then the output is greatly simplified and shown in
   26       only one line.
   27 
   28     * If ``mixed`` is used, then terse output will be used unless a state
   29       failed, in which case full output will be used.
   30 
   31     * If ``changes`` is used, then terse output will be used if there was no
   32       error and no changes, otherwise full output will be used.
   33 
   34     * If ``filter`` is used, then either or both of two different filters can be
   35       used: ``exclude`` or ``terse``.
   36 
   37         * for ``exclude``, state.highstate expects a list of states to be excluded (or ``None``)
   38           followed by ``True`` for terse output or ``False`` for regular output.
   39           Because of parsing nuances, if only one of these is used, it must still
   40           contain a comma. For instance: `exclude=True,`.
   41 
   42         * for ``terse``, state.highstate expects simply ``True`` or ``False``.
   43 
   44       These can be set as such from the command line, or in the Salt config as
   45       `state_output_exclude` or `state_output_terse`, respectively.
   46 
   47     The output modes have one modifier:
   48 
   49     ``full_id``, ``terse_id``, ``mixed_id``, ``changes_id`` and ``filter_id``
   50     If ``_id`` is used, then the corresponding form will be used, but the value for ``name``
   51     will be drawn from the state ID. This is useful for cases where the name
   52     value might be very long and hard to read.
   53 
   54 state_tabular:
   55     If `state_output` uses the terse output, set this to `True` for an aligned
   56     output format.  If you wish to use a custom format, this can be set to a
   57     string.
   58 
   59 Example usage:
   60 
   61 If ``state_output: filter`` is set in the configuration file:
   62 
   63 .. code-block:: bash
   64 
   65     salt '*' state.highstate exclude=None,True
   66 
   67 
   68 means to exclude no states from the highstate and turn on terse output.
   69 
   70 .. code-block:: bash
   71 
   72     salt twd state.highstate exclude=problemstate1,problemstate2,False
   73 
   74 
   75 means to exclude states ``problemstate1`` and ``problemstate2``
   76 from the highstate, and use regular output.
   77 
   78 Example output for the above highstate call when ``top.sls`` defines only
   79 one other state to apply to minion ``twd``:
   80 
   81 .. code-block:: text
   82 
   83     twd:
   84 
   85     Summary for twd
   86     ------------
   87     Succeeded: 1 (changed=1)
   88     Failed:    0
   89     ------------
   90     Total states run:     1
   91 
   92 
   93 Example output with no special settings in configuration files:
   94 
   95 .. code-block:: text
   96 
   97     myminion:
   98     ----------
   99               ID: test.ping
  100         Function: module.run
  101           Result: True
  102          Comment: Module function test.ping executed
  103          Changes:
  104                   ----------
  105                   ret:
  106                       True
  107 
  108     Summary for myminion
  109     ------------
  110     Succeeded: 1
  111     Failed:    0
  112     ------------
  113     Total:     0
  114 '''
  115 
  116 # Import python libs
  117 from __future__ import absolute_import, print_function, unicode_literals
  118 import pprint
  119 import textwrap
  120 
  121 # Import salt libs
  122 import salt.utils.color
  123 import salt.utils.data
  124 import salt.utils.stringutils
  125 import salt.output
  126 
  127 # Import 3rd-party libs
  128 from salt.ext import six
  129 
  130 import logging
  131 
  132 log = logging.getLogger(__name__)
  133 
  134 
  135 def output(data, **kwargs):  # pylint: disable=unused-argument
  136     '''
  137     The HighState Outputter is only meant to be used with the state.highstate
  138     function, or a function that returns highstate return data.
  139     '''
  140     # Discard retcode in dictionary as present in orchestrate data
  141     local_masters = [key for key in data.keys() if key.endswith('.local_master')]
  142     orchestrator_output = 'retcode' in data.keys() and len(local_masters) == 1
  143 
  144     if orchestrator_output:
  145         del data['retcode']
  146 
  147     # If additional information is passed through via the "data" dictionary to
  148     # the highstate outputter, such as "outputter" or "retcode", discard it.
  149     # We only want the state data that was passed through, if it is wrapped up
  150     # in the "data" key, as the orchestrate runner does. See Issue #31330,
  151     # pull request #27838, and pull request #27175 for more information.
  152     if 'data' in data:
  153         data = data.pop('data')
  154 
  155     ret = [
  156         _format_host(host, hostdata)[0]
  157         for host, hostdata in six.iteritems(data)
  158     ]
  159     if ret:
  160         return "\n".join(ret)
  161     log.error(
  162         'Data passed to highstate outputter is not a valid highstate return: %s',
  163         data
  164     )
  165     # We should not reach here, but if we do return empty string
  166     return ''
  167 
  168 
  169 def _format_host(host, data):
  170     host = salt.utils.data.decode(host)
  171 
  172     colors = salt.utils.color.get_colors(
  173             __opts__.get('color'),
  174             __opts__.get('color_theme'))
  175     tabular = __opts__.get('state_tabular', False)
  176     rcounts = {}
  177     rdurations = []
  178     hcolor = colors['GREEN']
  179     hstrs = []
  180     nchanges = 0
  181     strip_colors = __opts__.get('strip_colors', True)
  182 
  183     if isinstance(data, int) or isinstance(data, six.string_types):
  184         # Data in this format is from saltmod.function,
  185         # so it is always a 'change'
  186         nchanges = 1
  187         hstrs.append(('{0}    {1}{2[ENDC]}'
  188                       .format(hcolor, data, colors)))
  189         hcolor = colors['CYAN']  # Print the minion name in cyan
  190     if isinstance(data, list):
  191         # Errors have been detected, list them in RED!
  192         hcolor = colors['LIGHT_RED']
  193         hstrs.append(('    {0}Data failed to compile:{1[ENDC]}'
  194                       .format(hcolor, colors)))
  195         for err in data:
  196             if strip_colors:
  197                 err = salt.output.strip_esc_sequence(
  198                     salt.utils.data.decode(err)
  199                 )
  200             hstrs.append(('{0}----------\n    {1}{2[ENDC]}'
  201                           .format(hcolor, err, colors)))
  202     if isinstance(data, dict):
  203         # Verify that the needed data is present
  204         data_tmp = {}
  205         for tname, info in six.iteritems(data):
  206             if isinstance(info, dict) and tname is not 'changes' and info and '__run_num__' not in info:
  207                 err = ('The State execution failed to record the order '
  208                        'in which all states were executed. The state '
  209                        'return missing data is:')
  210                 hstrs.insert(0, pprint.pformat(info))
  211                 hstrs.insert(0, err)
  212             if isinstance(info, dict) and 'result' in info:
  213                 data_tmp[tname] = info
  214         data = data_tmp
  215         # Everything rendered as it should display the output
  216         for tname in sorted(
  217                 data,
  218                 key=lambda k: data[k].get('__run_num__', 0)):
  219             ret = data[tname]
  220             # Increment result counts
  221             rcounts.setdefault(ret['result'], 0)
  222             rcounts[ret['result']] += 1
  223             rduration = ret.get('duration', 0)
  224             try:
  225                 rdurations.append(float(rduration))
  226             except ValueError:
  227                 rduration, _, _ = rduration.partition(' ms')
  228                 try:
  229                     rdurations.append(float(rduration))
  230                 except ValueError:
  231                     log.error('Cannot parse a float from duration {0}'
  232                               .format(ret.get('duration', 0)))
  233 
  234             tcolor = colors['GREEN']
  235             orchestration = ret.get('__orchestration__', False)
  236             schanged, ctext = _format_changes(ret['changes'], orchestration)
  237             if not ctext and 'pchanges' in ret:
  238                 schanged, ctext = _format_changes(ret['pchanges'], orchestration)
  239             nchanges += 1 if schanged else 0
  240 
  241             # Skip this state if it was successful & diff output was requested
  242             if __opts__.get('state_output_diff', False) and \
  243                ret['result'] and not schanged:
  244                 continue
  245 
  246             # Skip this state if state_verbose is False, the result is True and
  247             # there were no changes made
  248             if not __opts__.get('state_verbose', False) and \
  249                ret['result'] and not schanged:
  250                 continue
  251 
  252             if schanged:
  253                 tcolor = colors['CYAN']
  254             if ret['result'] is False:
  255                 hcolor = colors['RED']
  256                 tcolor = colors['RED']
  257             if ret['result'] is None:
  258                 hcolor = colors['LIGHT_YELLOW']
  259                 tcolor = colors['LIGHT_YELLOW']
  260 
  261             state_output = __opts__.get('state_output', 'full').lower()
  262             comps = tname.split('_|-')
  263 
  264             if state_output.endswith('_id'):
  265                 # Swap in the ID for the name. Refs #35137
  266                 comps[2] = comps[1]
  267 
  268             if state_output.startswith('filter'):
  269                 # By default, full data is shown for all types. However, return
  270                 # data may be excluded by setting state_output_exclude to a
  271                 # comma-separated list of True, False or None, or including the
  272                 # same list with the exclude option on the command line. For
  273                 # now, this option must include a comma. For example:
  274                 #     exclude=True,
  275                 # The same functionality is also available for making return
  276                 # data terse, instead of excluding it.
  277                 cliargs = __opts__.get('arg', [])
  278                 clikwargs = {}
  279                 for item in cliargs:
  280                     if isinstance(item, dict) and '__kwarg__' in item:
  281                         clikwargs = item.copy()
  282 
  283                 exclude = clikwargs.get(
  284                     'exclude', __opts__.get('state_output_exclude', [])
  285                 )
  286                 if isinstance(exclude, six.string_types):
  287                     exclude = six.text_type(exclude).split(',')
  288 
  289                 terse = clikwargs.get(
  290                     'terse', __opts__.get('state_output_terse', [])
  291                 )
  292                 if isinstance(terse, six.string_types):
  293                     terse = six.text_type(terse).split(',')
  294 
  295                 if six.text_type(ret['result']) in terse:
  296                     msg = _format_terse(tcolor, comps, ret, colors, tabular)
  297                     hstrs.append(msg)
  298                     continue
  299                 if six.text_type(ret['result']) in exclude:
  300                     continue
  301 
  302             elif any((
  303                 state_output.startswith('terse'),
  304                 state_output.startswith('mixed') and ret['result'] is not False,  # only non-error'd
  305                 state_output.startswith('changes') and ret['result'] and not schanged  # non-error'd non-changed
  306             )):
  307                 # Print this chunk in a terse way and continue in the loop
  308                 msg = _format_terse(tcolor, comps, ret, colors, tabular)
  309                 hstrs.append(msg)
  310                 continue
  311 
  312             state_lines = [
  313                 '{tcolor}----------{colors[ENDC]}',
  314                 '    {tcolor}      ID: {comps[1]}{colors[ENDC]}',
  315                 '    {tcolor}Function: {comps[0]}.{comps[3]}{colors[ENDC]}',
  316                 '    {tcolor}  Result: {ret[result]!s}{colors[ENDC]}',
  317                 '    {tcolor} Comment: {comment}{colors[ENDC]}',
  318             ]
  319             if __opts__.get('state_output_profile', True) and 'start_time' in ret:
  320                 state_lines.extend([
  321                     '    {tcolor} Started: {ret[start_time]!s}{colors[ENDC]}',
  322                     '    {tcolor}Duration: {ret[duration]!s}{colors[ENDC]}',
  323                 ])
  324             # This isn't the prettiest way of doing this, but it's readable.
  325             if comps[1] != comps[2]:
  326                 state_lines.insert(
  327                     3, '    {tcolor}    Name: {comps[2]}{colors[ENDC]}')
  328             # be sure that ret['comment'] is utf-8 friendly
  329             try:
  330                 if not isinstance(ret['comment'], six.text_type):
  331                     ret['comment'] = six.text_type(ret['comment'])
  332             except UnicodeDecodeError:
  333                 # If we got here, we're on Python 2 and ret['comment'] somehow
  334                 # contained a str type with unicode content.
  335                 ret['comment'] = salt.utils.stringutils.to_unicode(ret['comment'])
  336             try:
  337                 comment = salt.utils.data.decode(ret['comment'])
  338                 comment = comment.strip().replace(
  339                         '\n',
  340                         '\n' + ' ' * 14)
  341             except AttributeError:  # Assume comment is a list
  342                 try:
  343                     comment = ret['comment'].join(' ').replace(
  344                         '\n',
  345                         '\n' + ' ' * 13)
  346                 except AttributeError:
  347                     # Comment isn't a list either, just convert to string
  348                     comment = six.text_type(ret['comment'])
  349                     comment = comment.strip().replace(
  350                         '\n',
  351                         '\n' + ' ' * 14)
  352             # If there is a data attribute, append it to the comment
  353             if 'data' in ret:
  354                 if isinstance(ret['data'], list):
  355                     for item in ret['data']:
  356                         comment = '{0} {1}'.format(comment, item)
  357                 elif isinstance(ret['data'], dict):
  358                     for key, value in ret['data'].items():
  359                         comment = '{0}\n\t\t{1}: {2}'.format(comment, key, value)
  360                 else:
  361                     comment = '{0} {1}'.format(comment, ret['data'])
  362             for detail in ['start_time', 'duration']:
  363                 ret.setdefault(detail, '')
  364             if ret['duration'] != '':
  365                 ret['duration'] = '{0} ms'.format(ret['duration'])
  366             svars = {
  367                 'tcolor': tcolor,
  368                 'comps': comps,
  369                 'ret': ret,
  370                 'comment': salt.utils.data.decode(comment),
  371                 # This nukes any trailing \n and indents the others.
  372                 'colors': colors
  373             }
  374             hstrs.extend([sline.format(**svars) for sline in state_lines])
  375             changes = '     Changes:   ' + ctext
  376             hstrs.append(('{0}{1}{2[ENDC]}'
  377                           .format(tcolor, changes, colors)))
  378 
  379             if 'warnings' in ret:
  380                 rcounts.setdefault('warnings', 0)
  381                 rcounts['warnings'] += 1
  382                 wrapper = textwrap.TextWrapper(
  383                     width=80,
  384                     initial_indent=' ' * 14,
  385                     subsequent_indent=' ' * 14
  386                 )
  387                 hstrs.append(
  388                     '   {colors[LIGHT_RED]} Warnings: {0}{colors[ENDC]}'.format(
  389                         wrapper.fill('\n'.join(ret['warnings'])).lstrip(),
  390                         colors=colors
  391                     )
  392                 )
  393 
  394         # Append result counts to end of output
  395         colorfmt = '{0}{1}{2[ENDC]}'
  396         rlabel = {True: 'Succeeded', False: 'Failed', None: 'Not Run', 'warnings': 'Warnings'}
  397         count_max_len = max([len(six.text_type(x)) for x in six.itervalues(rcounts)] or [0])
  398         label_max_len = max([len(x) for x in six.itervalues(rlabel)] or [0])
  399         line_max_len = label_max_len + count_max_len + 2  # +2 for ': '
  400         hstrs.append(
  401             colorfmt.format(
  402                 colors['CYAN'],
  403                 '\nSummary for {0}\n{1}'.format(host, '-' * line_max_len),
  404                 colors
  405             )
  406         )
  407 
  408         def _counts(label, count):
  409             return '{0}: {1:>{2}}'.format(
  410                 label,
  411                 count,
  412                 line_max_len - (len(label) + 2)
  413             )
  414 
  415         # Successful states
  416         changestats = []
  417         if None in rcounts and rcounts.get(None, 0) > 0:
  418             # test=True states
  419             changestats.append(
  420                 colorfmt.format(
  421                     colors['LIGHT_YELLOW'],
  422                     'unchanged={0}'.format(rcounts.get(None, 0)),
  423                     colors
  424                 )
  425             )
  426         if nchanges > 0:
  427             changestats.append(
  428                 colorfmt.format(
  429                     colors['GREEN'],
  430                     'changed={0}'.format(nchanges),
  431                     colors
  432                 )
  433             )
  434         if changestats:
  435             changestats = ' ({0})'.format(', '.join(changestats))
  436         else:
  437             changestats = ''
  438         hstrs.append(
  439             colorfmt.format(
  440                 colors['GREEN'],
  441                 _counts(
  442                     rlabel[True],
  443                     rcounts.get(True, 0) + rcounts.get(None, 0)
  444                 ),
  445                 colors
  446             ) + changestats
  447         )
  448 
  449         # Failed states
  450         num_failed = rcounts.get(False, 0)
  451         hstrs.append(
  452             colorfmt.format(
  453                 colors['RED'] if num_failed else colors['CYAN'],
  454                 _counts(rlabel[False], num_failed),
  455                 colors
  456             )
  457         )
  458 
  459         num_warnings = rcounts.get('warnings', 0)
  460         if num_warnings:
  461             hstrs.append(
  462                 colorfmt.format(
  463                     colors['LIGHT_RED'],
  464                     _counts(rlabel['warnings'], num_warnings),
  465                     colors
  466                 )
  467             )
  468         totals = '{0}\nTotal states run: {1:>{2}}'.format('-' * line_max_len,
  469                                                sum(six.itervalues(rcounts)) - rcounts.get('warnings', 0),
  470                                                line_max_len - 7)
  471         hstrs.append(colorfmt.format(colors['CYAN'], totals, colors))
  472 
  473         if __opts__.get('state_output_profile', True):
  474             sum_duration = sum(rdurations)
  475             duration_unit = 'ms'
  476             # convert to seconds if duration is 1000ms or more
  477             if sum_duration > 999:
  478                 sum_duration /= 1000
  479                 duration_unit = 's'
  480             total_duration = 'Total run time: {0} {1}'.format(
  481                 '{0:.3f}'.format(sum_duration).rjust(line_max_len - 5),
  482                 duration_unit)
  483             hstrs.append(colorfmt.format(colors['CYAN'], total_duration, colors))
  484 
  485     if strip_colors:
  486         host = salt.output.strip_esc_sequence(host)
  487     hstrs.insert(0, ('{0}{1}:{2[ENDC]}'.format(hcolor, host, colors)))
  488     return '\n'.join(hstrs), nchanges > 0
  489 
  490 
  491 def _nested_changes(changes):
  492     '''
  493     Print the changes data using the nested outputter
  494     '''
  495     ret = '\n'
  496     ret += salt.output.out_format(
  497             changes,
  498             'nested',
  499             __opts__,
  500             nested_indent=14)
  501     return ret
  502 
  503 
  504 def _format_changes(changes, orchestration=False):
  505     '''
  506     Format the changes dict based on what the data is
  507     '''
  508     if not changes:
  509         return False, ''
  510 
  511     if orchestration:
  512         return True, _nested_changes(changes)
  513 
  514     if not isinstance(changes, dict):
  515         return True, 'Invalid Changes data: {0}'.format(changes)
  516 
  517     ret = changes.get('ret')
  518     if ret is not None and changes.get('out') == 'highstate':
  519         ctext = ''
  520         changed = False
  521         for host, hostdata in six.iteritems(ret):
  522             s, c = _format_host(host, hostdata)
  523             ctext += '\n' + '\n'.join((' ' * 14 + l) for l in s.splitlines())
  524             changed = changed or c
  525     else:
  526         changed = True
  527         ctext = _nested_changes(changes)
  528     return changed, ctext
  529 
  530 
  531 def _format_terse(tcolor, comps, ret, colors, tabular):
  532     '''
  533     Terse formatting of a message.
  534     '''
  535     result = 'Clean'
  536     if ret['changes']:
  537         result = 'Changed'
  538     if ret['result'] is False:
  539         result = 'Failed'
  540     elif ret['result'] is None:
  541         result = 'Differs'
  542     if tabular is True:
  543         fmt_string = ''
  544         if 'warnings' in ret:
  545             fmt_string += '{c[LIGHT_RED]}Warnings:\n{w}{c[ENDC]}\n'.format(
  546                 c=colors, w='\n'.join(ret['warnings'])
  547             )
  548         fmt_string += '{0}'
  549         if __opts__.get('state_output_profile', True) and 'start_time' in ret:
  550             fmt_string += '{6[start_time]!s} [{6[duration]!s:>7} ms] '
  551         fmt_string += '{2:>10}.{3:<10} {4:7}   Name: {1}{5}'
  552     elif isinstance(tabular, six.string_types):
  553         fmt_string = tabular
  554     else:
  555         fmt_string = ''
  556         if 'warnings' in ret:
  557             fmt_string += '{c[LIGHT_RED]}Warnings:\n{w}{c[ENDC]}'.format(
  558                 c=colors, w='\n'.join(ret['warnings'])
  559             )
  560         fmt_string += ' {0} Name: {1} - Function: {2}.{3} - Result: {4}'
  561         if __opts__.get('state_output_profile', True) and 'start_time' in ret:
  562             fmt_string += ' Started: - {6[start_time]!s} Duration: {6[duration]!s} ms'
  563         fmt_string += '{5}'
  564 
  565     msg = fmt_string.format(tcolor,
  566                             comps[2],
  567                             comps[0],
  568                             comps[-1],
  569                             result,
  570                             colors['ENDC'],
  571                             ret)
  572     return msg