"Fossies" - the Fresh Open Source Software Archive

Member "salt-2016.11.3/salt/output/highstate.py" (21 Feb 2017, 21995 Bytes) of archive /linux/misc/salt-2016.11.3.tar.gz:


As a special service "Fossies" has tried to format the requested source page into HTML format using (guessed) Python source code syntax highlighting (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file. For more information about "highstate.py" see the Fossies "Dox" file reference documentation and the latest Fossies "Diffs" side-by-side code changes report: 2016.11.2_vs_2016.11.3.

    1 # -*- coding: utf-8 -*-
    2 '''
    3 Outputter for displaying results of state runs
    4 ==============================================
    5 
    6 The return data from the Highstate command is a standard data structure
    7 which is parsed by the highstate outputter to deliver a clean and readable
    8 set of information about the HighState run on minions.
    9 
   10 Two configurations can be set to modify the highstate outputter. These values
   11 can be set in the master config to change the output of the ``salt`` command or
   12 set in the minion config to change the output of the ``salt-call`` command.
   13 
   14 state_verbose:
   15     By default `state_verbose` is set to `True`, setting this to `False` will
   16     instruct the highstate outputter to omit displaying anything in green, this
   17     means that nothing with a result of True and no changes will not be printed
   18 state_output:
   19     The highstate outputter has six output modes, ``full``, ``terse``,
   20     ``mixed``, ``mixed_id``, ``changes`` and ``filter``.
   21 
   22     * The default is set to ``full``, which will display many lines of detailed
   23       information for each executed chunk.
   24     * If ``terse`` is used, then the output is greatly simplified and shown in
   25       only one line.
   26     * If ``mixed`` is used, then terse output will be used unless a state
   27       failed, in which case full output will be used.
   28     * If ``mixed_id`` is used, then the mixed form will be used, but the value for ``name``
   29       will be drawn from the state ID. This is useful for cases where the name
   30       value might be very long and hard to read.
   31     * If ``changes`` is used, then terse output will be used if there was no
   32       error and no changes, otherwise full output will be used.
   33     * If ``filter`` is used, then either or both of two different filters can be
   34       used: ``exclude`` or ``terse``.
   35       * for ``exclude``, state.highstate expects a list of states to be excluded
   36         (or ``None``)
   37         followed by ``True`` for terse output or ``False`` for regular output.
   38         Because of parsing nuances, if only one of these is used, it must still
   39         contain a comma. For instance: `exclude=True,`.
   40       * for ``terse``, state.highstate expects simply ``True`` or ``False``.
   41       These can be set as such from the command line, or in the Salt config as
   42       `state_output_exclude` or `state_output_terse`, respectively.
   43 state_tabular:
   44     If `state_output` uses the terse output, set this to `True` for an aligned
   45     output format.  If you wish to use a custom format, this can be set to a
   46     string.
   47 
   48 Example usage:
   49 
   50 If ``state_output: filter`` is set in the configuration file:
   51 
   52 .. code-block:: bash
   53 
   54     salt '*' state.highstate exclude=None,True
   55 
   56 
   57 means to exclude no states from the highstate and turn on terse output.
   58 
   59 .. code-block:: bash
   60 
   61     salt twd state.highstate exclude=problemstate1,problemstate2,False
   62 
   63 
   64 means to exclude states ``problemstate1`` and ``problemstate2``
   65 from the highstate, and use regular output.
   66 
   67 Example output for the above highstate call when ``top.sls`` defines only
   68 one other state to apply to minion ``twd``:
   69 
   70 .. code-block:: text
   71 
   72     twd:
   73 
   74     Summary for twd
   75     ------------
   76     Succeeded: 1 (changed=1)
   77     Failed:    0
   78     ------------
   79     Total states run:     1
   80 
   81 
   82 Example output with no special settings in configuration files:
   83 
   84 .. code-block:: text
   85 
   86     myminion:
   87     ----------
   88               ID: test.ping
   89         Function: module.run
   90           Result: True
   91          Comment: Module function test.ping executed
   92          Changes:
   93                   ----------
   94                   ret:
   95                       True
   96 
   97     Summary for myminion
   98     ------------
   99     Succeeded: 1
  100     Failed:    0
  101     ------------
  102     Total:     0
  103 '''
  104 
  105 # Import python libs
  106 from __future__ import absolute_import
  107 import pprint
  108 import textwrap
  109 
  110 # Import salt libs
  111 import salt.utils
  112 import salt.output
  113 from salt.utils.locales import sdecode
  114 
  115 # Import 3rd-party libs
  116 import salt.ext.six as six
  117 
  118 import logging
  119 
  120 log = logging.getLogger(__name__)
  121 
  122 
  123 def output(data, **kwargs):  # pylint: disable=unused-argument
  124     '''
  125     The HighState Outputter is only meant to be used with the state.highstate
  126     function, or a function that returns highstate return data.
  127     '''
  128 
  129     # Discard retcode in dictionary as present in orchestrate data
  130     local_masters = [key for key in data.keys() if key.endswith('.local_master')]
  131     orchestrator_output = 'retcode' in data.keys() and len(local_masters) == 1
  132 
  133     if orchestrator_output:
  134         del data['retcode']
  135 
  136     # If additional information is passed through via the "data" dictionary to
  137     # the highstate outputter, such as "outputter" or "retcode", discard it.
  138     # We only want the state data that was passed through, if it is wrapped up
  139     # in the "data" key, as the orchestrate runner does. See Issue #31330,
  140     # pull request #27838, and pull request #27175 for more information.
  141     if 'data' in data:
  142         data = data.pop('data')
  143 
  144     ret = [
  145         _format_host(host, hostdata)[0]
  146         for host, hostdata in six.iteritems(data)
  147     ]
  148     if ret:
  149         return "\n".join(ret)
  150     log.error(
  151         'Data passed to highstate outputter is not a valid highstate return: %s',
  152         data
  153     )
  154     # We should not reach here, but if we do return empty string
  155     return ''
  156 
  157 
  158 def _format_host(host, data):
  159     host = sdecode(host)
  160 
  161     colors = salt.utils.get_colors(
  162             __opts__.get('color'),
  163             __opts__.get('color_theme'))
  164     tabular = __opts__.get('state_tabular', False)
  165     rcounts = {}
  166     rdurations = []
  167     hcolor = colors['GREEN']
  168     hstrs = []
  169     nchanges = 0
  170     strip_colors = __opts__.get('strip_colors', True)
  171 
  172     if isinstance(data, int) or isinstance(data, str):
  173         # Data in this format is from saltmod.function,
  174         # so it is always a 'change'
  175         nchanges = 1
  176         hstrs.append((u'{0}    {1}{2[ENDC]}'
  177                       .format(hcolor, data, colors)))
  178         hcolor = colors['CYAN']  # Print the minion name in cyan
  179     if isinstance(data, list):
  180         # Errors have been detected, list them in RED!
  181         hcolor = colors['LIGHT_RED']
  182         hstrs.append((u'    {0}Data failed to compile:{1[ENDC]}'
  183                       .format(hcolor, colors)))
  184         for err in data:
  185             if strip_colors:
  186                 err = salt.output.strip_esc_sequence(sdecode(err))
  187             hstrs.append((u'{0}----------\n    {1}{2[ENDC]}'
  188                           .format(hcolor, err, colors)))
  189     if isinstance(data, dict):
  190         # Verify that the needed data is present
  191         data_tmp = {}
  192         for tname, info in six.iteritems(data):
  193             if isinstance(info, dict) and tname is not 'changes' and info and '__run_num__' not in info:
  194                 err = (u'The State execution failed to record the order '
  195                        'in which all states were executed. The state '
  196                        'return missing data is:')
  197                 hstrs.insert(0, pprint.pformat(info))
  198                 hstrs.insert(0, err)
  199             if isinstance(info, dict) and 'result' in info:
  200                 data_tmp[tname] = info
  201         data = data_tmp
  202         # Everything rendered as it should display the output
  203         for tname in sorted(
  204                 data,
  205                 key=lambda k: data[k].get('__run_num__', 0)):
  206             ret = data[tname]
  207             # Increment result counts
  208             rcounts.setdefault(ret['result'], 0)
  209             rcounts[ret['result']] += 1
  210             rduration = ret.get('duration', 0)
  211             try:
  212                 float(rduration)
  213                 rdurations.append(rduration)
  214             except ValueError:
  215                 rduration, _, _ = rduration.partition(' ms')
  216                 try:
  217                     float(rduration)
  218                     rdurations.append(rduration)
  219                 except ValueError:
  220                     log.error('Cannot parse a float from duration {0}'
  221                               .format(ret.get('duration', 0)))
  222 
  223             tcolor = colors['GREEN']
  224             orchestration = ret.get('__orchestration__', False)
  225             schanged, ctext = _format_changes(ret['changes'], orchestration)
  226             nchanges += 1 if schanged else 0
  227 
  228             # Skip this state if it was successful & diff output was requested
  229             if __opts__.get('state_output_diff', False) and \
  230                ret['result'] and not schanged:
  231                 continue
  232 
  233             # Skip this state if state_verbose is False, the result is True and
  234             # there were no changes made
  235             if not __opts__.get('state_verbose', False) and \
  236                ret['result'] and not schanged:
  237                 continue
  238 
  239             if schanged:
  240                 tcolor = colors['CYAN']
  241             if ret['result'] is False:
  242                 hcolor = colors['RED']
  243                 tcolor = colors['RED']
  244             if ret['result'] is None:
  245                 hcolor = colors['LIGHT_YELLOW']
  246                 tcolor = colors['LIGHT_YELLOW']
  247             comps = [sdecode(comp) for comp in tname.split('_|-')]
  248             if __opts__.get('state_output', 'full').lower() == 'filter':
  249                 # By default, full data is shown for all types. However, return
  250                 # data may be excluded by setting state_output_exclude to a
  251                 # comma-separated list of True, False or None, or including the
  252                 # same list with the exclude option on the command line. For
  253                 # now, this option must include a comma. For example:
  254                 #     exclude=True,
  255                 # The same functionality is also available for making return
  256                 # data terse, instead of excluding it.
  257                 cliargs = __opts__.get('arg', [])
  258                 clikwargs = {}
  259                 for item in cliargs:
  260                     if isinstance(item, dict) and '__kwarg__' in item:
  261                         clikwargs = item.copy()
  262 
  263                 exclude = clikwargs.get(
  264                     'exclude', __opts__.get('state_output_exclude', [])
  265                 )
  266                 if isinstance(exclude, six.string_types):
  267                     exclude = str(exclude).split(',')
  268 
  269                 terse = clikwargs.get(
  270                     'terse', __opts__.get('state_output_terse', [])
  271                 )
  272                 if isinstance(terse, six.string_types):
  273                     terse = str(terse).split(',')
  274 
  275                 if str(ret['result']) in terse:
  276                     msg = _format_terse(tcolor, comps, ret, colors, tabular)
  277                     hstrs.append(msg)
  278                     continue
  279                 if str(ret['result']) in exclude:
  280                     continue
  281             elif __opts__.get('state_output', 'full').lower() == 'terse':
  282                 # Print this chunk in a terse way and continue in the
  283                 # loop
  284                 msg = _format_terse(tcolor, comps, ret, colors, tabular)
  285                 hstrs.append(msg)
  286                 continue
  287             elif __opts__.get('state_output', 'full').lower().startswith('mixed'):
  288                 if __opts__['state_output'] == 'mixed_id':
  289                     # Swap in the ID for the name. Refs #35137
  290                     comps[2] = comps[1]
  291                 # Print terse unless it failed
  292                 if ret['result'] is not False:
  293                     msg = _format_terse(tcolor, comps, ret, colors, tabular)
  294                     hstrs.append(msg)
  295                     continue
  296             elif __opts__.get('state_output', 'full').lower() == 'changes':
  297                 # Print terse if no error and no changes, otherwise, be
  298                 # verbose
  299                 if ret['result'] and not schanged:
  300                     msg = _format_terse(tcolor, comps, ret, colors, tabular)
  301                     hstrs.append(msg)
  302                     continue
  303             state_lines = [
  304                 u'{tcolor}----------{colors[ENDC]}',
  305                 u'    {tcolor}      ID: {comps[1]}{colors[ENDC]}',
  306                 u'    {tcolor}Function: {comps[0]}.{comps[3]}{colors[ENDC]}',
  307                 u'    {tcolor}  Result: {ret[result]!s}{colors[ENDC]}',
  308                 u'    {tcolor} Comment: {comment}{colors[ENDC]}',
  309             ]
  310             if __opts__.get('state_output_profile', True) and 'start_time' in ret:
  311                 state_lines.extend([
  312                     u'    {tcolor} Started: {ret[start_time]!s}{colors[ENDC]}',
  313                     u'    {tcolor}Duration: {ret[duration]!s}{colors[ENDC]}',
  314                 ])
  315             # This isn't the prettiest way of doing this, but it's readable.
  316             if comps[1] != comps[2]:
  317                 state_lines.insert(
  318                     3, u'    {tcolor}    Name: {comps[2]}{colors[ENDC]}')
  319             # be sure that ret['comment'] is utf-8 friendly
  320             try:
  321                 if not isinstance(ret['comment'], six.text_type):
  322                     ret['comment'] = str(ret['comment']).decode('utf-8')
  323             except UnicodeDecodeError:
  324                 # but try to continue on errors
  325                 pass
  326             try:
  327                 comment = sdecode(ret['comment'])
  328                 comment = comment.strip().replace(
  329                         u'\n',
  330                         u'\n' + u' ' * 14)
  331             except AttributeError:  # Assume comment is a list
  332                 try:
  333                     comment = ret['comment'].join(' ').replace(
  334                         u'\n',
  335                         u'\n' + u' ' * 13)
  336                 except AttributeError:
  337                     # Comment isn't a list either, just convert to string
  338                     comment = str(ret['comment'])
  339                     comment = comment.strip().replace(
  340                         u'\n',
  341                         u'\n' + u' ' * 14)
  342             # If there is a data attribute, append it to the comment
  343             if 'data' in ret:
  344                 if isinstance(ret['data'], list):
  345                     for item in ret['data']:
  346                         comment = '{0} {1}'.format(comment, item)
  347                 elif isinstance(ret['data'], dict):
  348                     for key, value in ret['data'].items():
  349                         comment = '{0}\n\t\t{1}: {2}'.format(comment, key, value)
  350                 else:
  351                     comment = '{0} {1}'.format(comment, ret['data'])
  352             for detail in ['start_time', 'duration']:
  353                 ret.setdefault(detail, u'')
  354             if ret['duration'] != '':
  355                 ret['duration'] = u'{0} ms'.format(ret['duration'])
  356             svars = {
  357                 'tcolor': tcolor,
  358                 'comps': comps,
  359                 'ret': ret,
  360                 'comment': sdecode(comment),
  361                 # This nukes any trailing \n and indents the others.
  362                 'colors': colors
  363             }
  364             hstrs.extend([sline.format(**svars) for sline in state_lines])
  365             changes = u'     Changes:   ' + ctext
  366             hstrs.append((u'{0}{1}{2[ENDC]}'
  367                           .format(tcolor, changes, colors)))
  368 
  369             if 'warnings' in ret:
  370                 rcounts.setdefault('warnings', 0)
  371                 rcounts['warnings'] += 1
  372                 wrapper = textwrap.TextWrapper(
  373                     width=80,
  374                     initial_indent=u' ' * 14,
  375                     subsequent_indent=u' ' * 14
  376                 )
  377                 hstrs.append(
  378                     u'   {colors[LIGHT_RED]} Warnings: {0}{colors[ENDC]}'.format(
  379                         wrapper.fill('\n'.join(ret['warnings'])).lstrip(),
  380                         colors=colors
  381                     )
  382                 )
  383 
  384         # Append result counts to end of output
  385         colorfmt = u'{0}{1}{2[ENDC]}'
  386         rlabel = {True: u'Succeeded', False: u'Failed', None: u'Not Run', 'warnings': u'Warnings'}
  387         count_max_len = max([len(str(x)) for x in six.itervalues(rcounts)] or [0])
  388         label_max_len = max([len(x) for x in six.itervalues(rlabel)] or [0])
  389         line_max_len = label_max_len + count_max_len + 2  # +2 for ': '
  390         hstrs.append(
  391             colorfmt.format(
  392                 colors['CYAN'],
  393                 u'\nSummary for {0}\n{1}'.format(host, '-' * line_max_len),
  394                 colors
  395             )
  396         )
  397 
  398         def _counts(label, count):
  399             return u'{0}: {1:>{2}}'.format(
  400                 label,
  401                 count,
  402                 line_max_len - (len(label) + 2)
  403             )
  404 
  405         # Successful states
  406         changestats = []
  407         if None in rcounts and rcounts.get(None, 0) > 0:
  408             # test=True states
  409             changestats.append(
  410                 colorfmt.format(
  411                     colors['LIGHT_YELLOW'],
  412                     u'unchanged={0}'.format(rcounts.get(None, 0)),
  413                     colors
  414                 )
  415             )
  416         if nchanges > 0:
  417             changestats.append(
  418                 colorfmt.format(
  419                     colors['GREEN'],
  420                     u'changed={0}'.format(nchanges),
  421                     colors
  422                 )
  423             )
  424         if changestats:
  425             changestats = u' ({0})'.format(', '.join(changestats))
  426         else:
  427             changestats = u''
  428         hstrs.append(
  429             colorfmt.format(
  430                 colors['GREEN'],
  431                 _counts(
  432                     rlabel[True],
  433                     rcounts.get(True, 0) + rcounts.get(None, 0)
  434                 ),
  435                 colors
  436             ) + changestats
  437         )
  438 
  439         # Failed states
  440         num_failed = rcounts.get(False, 0)
  441         hstrs.append(
  442             colorfmt.format(
  443                 colors['RED'] if num_failed else colors['CYAN'],
  444                 _counts(rlabel[False], num_failed),
  445                 colors
  446             )
  447         )
  448 
  449         num_warnings = rcounts.get('warnings', 0)
  450         if num_warnings:
  451             hstrs.append(
  452                 colorfmt.format(
  453                     colors['LIGHT_RED'],
  454                     _counts(rlabel['warnings'], num_warnings),
  455                     colors
  456                 )
  457             )
  458         totals = u'{0}\nTotal states run: {1:>{2}}'.format('-' * line_max_len,
  459                                                sum(six.itervalues(rcounts)) - rcounts.get('warnings', 0),
  460                                                line_max_len - 7)
  461         hstrs.append(colorfmt.format(colors['CYAN'], totals, colors))
  462 
  463         if __opts__.get('state_output_profile', True):
  464             sum_duration = sum(rdurations)
  465             duration_unit = 'ms'
  466             # convert to seconds if duration is 1000ms or more
  467             if sum_duration > 999:
  468                 sum_duration /= 1000
  469                 duration_unit = 's'
  470             total_duration = u'Total run time: {0} {1}'.format(
  471                 '{0:.3f}'.format(sum_duration).rjust(line_max_len - 5),
  472                 duration_unit)
  473             hstrs.append(colorfmt.format(colors['CYAN'], total_duration, colors))
  474 
  475     if strip_colors:
  476         host = salt.output.strip_esc_sequence(host)
  477     hstrs.insert(0, (u'{0}{1}:{2[ENDC]}'.format(hcolor, host, colors)))
  478     return u'\n'.join(hstrs), nchanges > 0
  479 
  480 
  481 def _nested_changes(changes):
  482     '''
  483     Print the changes data using the nested outputter
  484     '''
  485     global __opts__  # pylint: disable=W0601
  486 
  487     opts = __opts__.copy()
  488     # Pass the __opts__ dict. The loader will splat this modules __opts__ dict
  489     # anyway so have to restore it after the other outputter is done
  490     if __opts__['color']:
  491         __opts__['color'] = u'CYAN'
  492     ret = u'\n'
  493     ret += salt.output.out_format(
  494             changes,
  495             'nested',
  496             __opts__,
  497             nested_indent=14)
  498     __opts__ = opts
  499     return ret
  500 
  501 
  502 def _format_changes(changes, orchestration=False):
  503     '''
  504     Format the changes dict based on what the data is
  505     '''
  506     if not changes:
  507         return False, u''
  508 
  509     if orchestration:
  510         return True, _nested_changes(changes)
  511 
  512     if not isinstance(changes, dict):
  513         return True, u'Invalid Changes data: {0}'.format(changes)
  514 
  515     ret = changes.get('ret')
  516     if ret is not None and changes.get('out') == 'highstate':
  517         ctext = u''
  518         changed = False
  519         for host, hostdata in six.iteritems(ret):
  520             s, c = _format_host(host, hostdata)
  521             ctext += u'\n' + u'\n'.join((u' ' * 14 + l) for l in s.splitlines())
  522             changed = changed or c
  523     else:
  524         changed = True
  525         ctext = _nested_changes(changes)
  526     return changed, ctext
  527 
  528 
  529 def _format_terse(tcolor, comps, ret, colors, tabular):
  530     '''
  531     Terse formatting of a message.
  532     '''
  533     result = u'Clean'
  534     if ret['changes']:
  535         result = u'Changed'
  536     if ret['result'] is False:
  537         result = u'Failed'
  538     elif ret['result'] is None:
  539         result = u'Differs'
  540     if tabular is True:
  541         fmt_string = ''
  542         if 'warnings' in ret:
  543             fmt_string += u'{c[LIGHT_RED]}Warnings:\n{w}{c[ENDC]}\n'.format(
  544                 c=colors, w='\n'.join(ret['warnings'])
  545             )
  546         fmt_string += u'{0}'
  547         if __opts__.get('state_output_profile', True) and 'start_time' in ret:
  548             fmt_string += u'{6[start_time]!s} [{6[duration]!s} ms] '
  549         fmt_string += u'{2:>10}.{3:<10} {4:7}   Name: {1}{5}'
  550     elif isinstance(tabular, str):
  551         fmt_string = tabular
  552     else:
  553         fmt_string = ''
  554         if 'warnings' in ret:
  555             fmt_string += u'{c[LIGHT_RED]}Warnings:\n{w}{c[ENDC]}'.format(
  556                 c=colors, w='\n'.join(ret['warnings'])
  557             )
  558         fmt_string += u' {0} Name: {1} - Function: {2}.{3} - Result: {4}'
  559         if __opts__.get('state_output_profile', True) and 'start_time' in ret:
  560             fmt_string += u' Started: - {6[start_time]!s} Duration: {6[duration]!s} ms'
  561         fmt_string += u'{5}'
  562 
  563     msg = fmt_string.format(tcolor,
  564                             comps[2],
  565                             comps[0],
  566                             comps[-1],
  567                             result,
  568                             colors['ENDC'],
  569                             ret)
  570     return msg