"Fossies" - the Fresh Open Source Software Archive

Member "salt-3002.2/salt/pillar/file_tree.py" (18 Nov 2020, 18416 Bytes) of package /linux/misc/salt-3002.2.tar.gz:


As a special service "Fossies" has tried to format the requested source page into HTML format using (guessed) Python source code syntax highlighting (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file. For more information about "file_tree.py" see the Fossies "Dox" file reference documentation and the latest Fossies "Diffs" side-by-side code changes report: 3002.1_vs_3002.2.

    1 """
    2 The ``file_tree`` external pillar allows values from all files in a directory
    3 tree to be imported as Pillar data.
    4 
    5 .. note::
    6 
    7     This is an external pillar and is subject to the :ref:`rules and
    8     constraints <external-pillars>` governing external pillars.
    9 
   10 .. versionadded:: 2015.5.0
   11 
   12 In this pillar, data is organized by either Minion ID or Nodegroup name.  To
   13 setup pillar data for a specific Minion, place it in
   14 ``<root_dir>/hosts/<minion_id>``.  To setup pillar data for an entire
   15 Nodegroup, place it in ``<root_dir>/nodegroups/<node_group>`` where
   16 ``<node_group>`` is the Nodegroup's name.
   17 
   18 Example ``file_tree`` Pillar
   19 ============================
   20 
   21 Master Configuration
   22 --------------------
   23 
   24 .. code-block:: yaml
   25 
   26     ext_pillar:
   27       - file_tree:
   28           root_dir: /srv/ext_pillar
   29           follow_dir_links: False
   30           keep_newline: True
   31 
   32 The ``root_dir`` parameter is required and points to the directory where files
   33 for each host are stored. The ``follow_dir_links`` parameter is optional and
   34 defaults to False. If ``follow_dir_links`` is set to True, this external pillar
   35 will follow symbolic links to other directories.
   36 
   37 .. warning::
   38     Be careful when using ``follow_dir_links``, as a recursive symlink chain
   39     will result in unexpected results.
   40 
   41 .. versionchanged:: 2018.3.0
   42     If ``root_dir`` is a relative path, it will be treated as relative to the
   43     :conf_master:`pillar_roots` of the environment specified by
   44     :conf_minion:`pillarenv`. If an environment specifies multiple
   45     roots, this module will search for files relative to all of them, in order,
   46     merging the results.
   47 
   48 If ``keep_newline`` is set to ``True``, then the pillar values for files ending
   49 in newlines will keep that newline. The default behavior is to remove the
   50 end-of-file newline. ``keep_newline`` should be turned on if the pillar data is
   51 intended to be used to deploy a file using ``contents_pillar`` with a
   52 :py:func:`file.managed <salt.states.file.managed>` state.
   53 
   54 .. versionchanged:: 2015.8.4
   55     The ``raw_data`` parameter has been renamed to ``keep_newline``. In earlier
   56     releases, ``raw_data`` must be used. Also, this parameter can now be a list
   57     of globs, allowing for more granular control over which pillar values keep
   58     their end-of-file newline. The globs match paths relative to the
   59     directories named for minion IDs and nodegroups underneath the ``root_dir``
   60     (see the layout examples in the below sections).
   61 
   62     .. code-block:: yaml
   63 
   64         ext_pillar:
   65           - file_tree:
   66               root_dir: /path/to/root/directory
   67               keep_newline:
   68                 - files/testdir/*
   69 
   70 .. note::
   71     In earlier releases, this documentation incorrectly stated that binary
   72     files would not affected by the ``keep_newline`` configuration.  However,
   73     this module does not actually distinguish between binary and text files.
   74 
   75 .. versionchanged:: 2017.7.0
   76     Templating/rendering has been added. You can now specify a default render
   77     pipeline and a black- and whitelist of (dis)allowed renderers.
   78 
   79     ``template`` must be set to ``True`` for templating to happen.
   80 
   81     .. code-block:: yaml
   82 
   83         ext_pillar:
   84           - file_tree:
   85             root_dir: /path/to/root/directory
   86             render_default: jinja|yaml
   87             renderer_blacklist:
   88               - gpg
   89             renderer_whitelist:
   90               - jinja
   91               - yaml
   92             template: True
   93 
   94 Assigning Pillar Data to Individual Hosts
   95 -----------------------------------------
   96 
   97 To configure pillar data for each host, this external pillar will recursively
   98 iterate over ``root_dir``/hosts/``id`` (where ``id`` is a minion ID), and
   99 compile pillar data with each subdirectory as a dictionary key and each file
  100 as a value.
  101 
  102 For example, the following ``root_dir`` tree:
  103 
  104 .. code-block:: text
  105 
  106     ./hosts/
  107     ./hosts/test-host/
  108     ./hosts/test-host/files/
  109     ./hosts/test-host/files/testdir/
  110     ./hosts/test-host/files/testdir/file1.txt
  111     ./hosts/test-host/files/testdir/file2.txt
  112     ./hosts/test-host/files/another-testdir/
  113     ./hosts/test-host/files/another-testdir/symlink-to-file1.txt
  114 
  115 will result in the following pillar tree for minion with ID ``test-host``:
  116 
  117 .. code-block:: text
  118 
  119     test-host:
  120         ----------
  121         apache:
  122             ----------
  123             config.d:
  124                 ----------
  125                 00_important.conf:
  126                     <important_config important_setting="yes" />
  127                 20_bob_extra.conf:
  128                     <bob_specific_cfg has_freeze_ray="yes" />
  129         corporate_app:
  130             ----------
  131             settings:
  132                 ----------
  133                 common_settings:
  134                     // This is the main settings file for the corporate
  135                     // internal web app
  136                     main_setting: probably
  137                 bob_settings:
  138                     role: bob
  139 
  140 .. note::
  141 
  142     The leaf data in the example shown is the contents of the pillar files.
  143 """
  144 
  145 import fnmatch
  146 import logging
  147 import os
  148 
  149 import salt.loader
  150 import salt.template
  151 import salt.utils.dictupdate
  152 import salt.utils.files
  153 import salt.utils.minions
  154 import salt.utils.path
  155 import salt.utils.stringio
  156 import salt.utils.stringutils
  157 
  158 # Set up logging
  159 log = logging.getLogger(__name__)
  160 
  161 
  162 def _on_walk_error(err):
  163     """
  164     Log salt.utils.path.os_walk() error.
  165     """
  166     log.error("%s: %s", err.filename, err.strerror)
  167 
  168 
  169 def _check_newline(prefix, file_name, keep_newline):
  170     """
  171     Return a boolean stating whether or not a file's trailing newline should be
  172     removed. To figure this out, first check if keep_newline is a boolean and
  173     if so, return its opposite. Otherwise, iterate over keep_newline and check
  174     if any of the patterns match the file path. If a match is found, return
  175     False, otherwise return True.
  176     """
  177     if isinstance(keep_newline, bool):
  178         return not keep_newline
  179     full_path = os.path.join(prefix, file_name)
  180     for pattern in keep_newline:
  181         try:
  182             if fnmatch.fnmatch(full_path, pattern):
  183                 return False
  184         except TypeError:
  185             if fnmatch.fnmatch(full_path, str(pattern)):
  186                 return False
  187     return True
  188 
  189 
  190 def _construct_pillar(
  191     top_dir,
  192     follow_dir_links,
  193     keep_newline=False,
  194     render_default=None,
  195     renderer_blacklist=None,
  196     renderer_whitelist=None,
  197     template=False,
  198 ):
  199     """
  200     Construct pillar from file tree.
  201     """
  202     pillar = {}
  203     renderers = salt.loader.render(__opts__, __salt__)
  204 
  205     norm_top_dir = os.path.normpath(top_dir)
  206     for dir_path, dir_names, file_names in salt.utils.path.os_walk(
  207         top_dir, topdown=True, onerror=_on_walk_error, followlinks=follow_dir_links
  208     ):
  209         # Find current path in pillar tree
  210         pillar_node = pillar
  211         norm_dir_path = os.path.normpath(dir_path)
  212         prefix = os.path.relpath(norm_dir_path, norm_top_dir)
  213         if norm_dir_path != norm_top_dir:
  214             path_parts = []
  215             head = prefix
  216             while head:
  217                 head, tail = os.path.split(head)
  218                 path_parts.insert(0, tail)
  219             while path_parts:
  220                 pillar_node = pillar_node[path_parts.pop(0)]
  221 
  222         # Create dicts for subdirectories
  223         for dir_name in dir_names:
  224             pillar_node[dir_name] = {}
  225 
  226         # Add files
  227         for file_name in file_names:
  228             file_path = os.path.join(dir_path, file_name)
  229             if not os.path.isfile(file_path):
  230                 log.error("file_tree: %s: not a regular file", file_path)
  231                 continue
  232 
  233             contents = b""
  234             try:
  235                 with salt.utils.files.fopen(file_path, "rb") as fhr:
  236                     buf = fhr.read(__opts__["file_buffer_size"])
  237                     while buf:
  238                         contents += buf
  239                         buf = fhr.read(__opts__["file_buffer_size"])
  240                     if contents.endswith(b"\n") and _check_newline(
  241                         prefix, file_name, keep_newline
  242                     ):
  243                         contents = contents[:-1]
  244             except OSError as exc:
  245                 log.error("file_tree: Error reading %s: %s", file_path, exc.strerror)
  246             else:
  247                 data = contents
  248                 if template is True:
  249                     data = salt.template.compile_template_str(
  250                         template=salt.utils.stringutils.to_unicode(contents),
  251                         renderers=renderers,
  252                         default=render_default,
  253                         blacklist=renderer_blacklist,
  254                         whitelist=renderer_whitelist,
  255                     )
  256                 if salt.utils.stringio.is_readable(data):
  257                     pillar_node[file_name] = data.getvalue()
  258                 else:
  259                     pillar_node[file_name] = data
  260 
  261     return pillar
  262 
  263 
  264 def ext_pillar(
  265     minion_id,
  266     pillar,
  267     root_dir=None,
  268     follow_dir_links=False,
  269     debug=False,
  270     keep_newline=False,
  271     render_default=None,
  272     renderer_blacklist=None,
  273     renderer_whitelist=None,
  274     template=False,
  275 ):
  276     """
  277     Compile pillar data from the given ``root_dir`` specific to Nodegroup names
  278     and Minion IDs.
  279 
  280     If a Minion's ID is not found at ``<root_dir>/host/<minion_id>`` or if it
  281     is not included in any Nodegroups named at
  282     ``<root_dir>/nodegroups/<node_group>``, no pillar data provided by this
  283     pillar module will be available for that Minion.
  284 
  285     .. versionchanged:: 2017.7.0
  286         Templating/rendering has been added. You can now specify a default
  287         render pipeline and a black- and whitelist of (dis)allowed renderers.
  288 
  289         ``template`` must be set to ``True`` for templating to happen.
  290 
  291         .. code-block:: yaml
  292 
  293             ext_pillar:
  294               - file_tree:
  295                 root_dir: /path/to/root/directory
  296                 render_default: jinja|yaml
  297                 renderer_blacklist:
  298                   - gpg
  299                 renderer_whitelist:
  300                   - jinja
  301                   - yaml
  302                 template: True
  303 
  304     :param minion_id:
  305         The ID of the Minion whose pillar data is to be collected
  306 
  307     :param pillar:
  308         Unused by the ``file_tree`` pillar module
  309 
  310     :param root_dir:
  311         Filesystem directory used as the root for pillar data (e.g.
  312         ``/srv/ext_pillar``)
  313 
  314         .. versionchanged:: 2018.3.0
  315             If ``root_dir`` is a relative path, it will be treated as relative to the
  316             :conf_master:`pillar_roots` of the environment specified by
  317             :conf_minion:`pillarenv`. If an environment specifies multiple
  318             roots, this module will search for files relative to all of them, in order,
  319             merging the results.
  320 
  321     :param follow_dir_links:
  322         Follow symbolic links to directories while collecting pillar files.
  323         Defaults to ``False``.
  324 
  325         .. warning::
  326 
  327             Care should be exercised when enabling this option as it will
  328             follow links that point outside of ``root_dir``.
  329 
  330         .. warning::
  331 
  332             Symbolic links that lead to infinite recursion are not filtered.
  333 
  334     :param debug:
  335         Enable debug information at log level ``debug``.  Defaults to
  336         ``False``.  This option may be useful to help debug errors when setting
  337         up the ``file_tree`` pillar module.
  338 
  339     :param keep_newline:
  340         Preserve the end-of-file newline in files.  Defaults to ``False``.
  341         This option may either be a boolean or a list of file globs (as defined
  342         by the `Python fnmatch package
  343         <https://docs.python.org/library/fnmatch.html>`_) for which end-of-file
  344         newlines are to be kept.
  345 
  346         ``keep_newline`` should be turned on if the pillar data is intended to
  347         be used to deploy a file using ``contents_pillar`` with a
  348         :py:func:`file.managed <salt.states.file.managed>` state.
  349 
  350         .. versionchanged:: 2015.8.4
  351             The ``raw_data`` parameter has been renamed to ``keep_newline``. In
  352             earlier releases, ``raw_data`` must be used. Also, this parameter
  353             can now be a list of globs, allowing for more granular control over
  354             which pillar values keep their end-of-file newline. The globs match
  355             paths relative to the directories named for Minion IDs and
  356             Nodegroup namess underneath the ``root_dir``.
  357 
  358             .. code-block:: yaml
  359 
  360                 ext_pillar:
  361                   - file_tree:
  362                       root_dir: /srv/ext_pillar
  363                       keep_newline:
  364                         - apache/config.d/*
  365                         - corporate_app/settings/*
  366 
  367         .. note::
  368             In earlier releases, this documentation incorrectly stated that
  369             binary files would not affected by the ``keep_newline``.  However,
  370             this module does not actually distinguish between binary and text
  371             files.
  372 
  373 
  374     :param render_default:
  375         Override Salt's :conf_master:`default global renderer <renderer>` for
  376         the ``file_tree`` pillar.
  377 
  378         .. code-block:: yaml
  379 
  380             render_default: jinja
  381 
  382     :param renderer_blacklist:
  383         Disallow renderers for pillar files.
  384 
  385         .. code-block:: yaml
  386 
  387             renderer_blacklist:
  388               - json
  389 
  390     :param renderer_whitelist:
  391         Allow renderers for pillar files.
  392 
  393         .. code-block:: yaml
  394 
  395             renderer_whitelist:
  396               - yaml
  397               - jinja
  398 
  399     :param template:
  400         Enable templating of pillar files.  Defaults to ``False``.
  401     """
  402     # Not used
  403     del pillar
  404 
  405     if not root_dir:
  406         log.error("file_tree: no root_dir specified")
  407         return {}
  408 
  409     if not os.path.isabs(root_dir):
  410         pillarenv = __opts__["pillarenv"]
  411         if pillarenv is None:
  412             log.error("file_tree: root_dir is relative but pillarenv is not set")
  413             return {}
  414         log.debug("file_tree: pillarenv = %s", pillarenv)
  415 
  416         env_roots = __opts__["pillar_roots"].get(pillarenv, None)
  417         if env_roots is None:
  418             log.error(
  419                 "file_tree: root_dir is relative but no pillar_roots are specified "
  420                 " for pillarenv %s",
  421                 pillarenv,
  422             )
  423             return {}
  424 
  425         env_dirs = []
  426         for env_root in env_roots:
  427             env_dir = os.path.normpath(os.path.join(env_root, root_dir))
  428             # don't redundantly load consecutively, but preserve any expected precedence
  429             if env_dir not in env_dirs or env_dir != env_dirs[-1]:
  430                 env_dirs.append(env_dir)
  431         dirs = env_dirs
  432     else:
  433         dirs = [root_dir]
  434 
  435     result_pillar = {}
  436     for root in dirs:
  437         dir_pillar = _ext_pillar(
  438             minion_id,
  439             root,
  440             follow_dir_links,
  441             debug,
  442             keep_newline,
  443             render_default,
  444             renderer_blacklist,
  445             renderer_whitelist,
  446             template,
  447         )
  448         result_pillar = salt.utils.dictupdate.merge(
  449             result_pillar, dir_pillar, strategy="recurse"
  450         )
  451     return result_pillar
  452 
  453 
  454 def _ext_pillar(
  455     minion_id,
  456     root_dir,
  457     follow_dir_links,
  458     debug,
  459     keep_newline,
  460     render_default,
  461     renderer_blacklist,
  462     renderer_whitelist,
  463     template,
  464 ):
  465     """
  466     Compile pillar data for a single root_dir for the specified minion ID
  467     """
  468     log.debug("file_tree: reading %s", root_dir)
  469 
  470     if not os.path.isdir(root_dir):
  471         log.error(
  472             "file_tree: root_dir %s does not exist or is not a directory", root_dir
  473         )
  474         return {}
  475 
  476     if not isinstance(keep_newline, (bool, list)):
  477         log.error(
  478             "file_tree: keep_newline must be either True/False or a list "
  479             "of file globs. Skipping this ext_pillar for root_dir %s",
  480             root_dir,
  481         )
  482         return {}
  483 
  484     ngroup_pillar = {}
  485     nodegroups_dir = os.path.join(root_dir, "nodegroups")
  486     if os.path.exists(nodegroups_dir) and len(__opts__.get("nodegroups", ())) > 0:
  487         master_ngroups = __opts__["nodegroups"]
  488         ext_pillar_dirs = os.listdir(nodegroups_dir)
  489         if len(ext_pillar_dirs) > 0:
  490             for nodegroup in ext_pillar_dirs:
  491                 if os.path.isdir(nodegroups_dir) and nodegroup in master_ngroups:
  492                     ckminions = salt.utils.minions.CkMinions(__opts__)
  493                     _res = ckminions.check_minions(
  494                         master_ngroups[nodegroup], "compound"
  495                     )
  496                     match = _res["minions"]
  497                     if minion_id in match:
  498                         ngroup_dir = os.path.join(nodegroups_dir, str(nodegroup))
  499                         ngroup_pillar = salt.utils.dictupdate.merge(
  500                             ngroup_pillar,
  501                             _construct_pillar(
  502                                 ngroup_dir,
  503                                 follow_dir_links,
  504                                 keep_newline,
  505                                 render_default,
  506                                 renderer_blacklist,
  507                                 renderer_whitelist,
  508                                 template,
  509                             ),
  510                             strategy="recurse",
  511                         )
  512         else:
  513             if debug is True:
  514                 log.debug(
  515                     "file_tree: no nodegroups found in file tree directory %s, skipping...",
  516                     ext_pillar_dirs,
  517                 )
  518     else:
  519         if debug is True:
  520             log.debug("file_tree: no nodegroups found in master configuration")
  521 
  522     host_dir = os.path.join(root_dir, "hosts", minion_id)
  523     if not os.path.exists(host_dir):
  524         if debug is True:
  525             log.debug(
  526                 "file_tree: no pillar data for minion %s found in file tree directory %s",
  527                 minion_id,
  528                 host_dir,
  529             )
  530         return ngroup_pillar
  531 
  532     if not os.path.isdir(host_dir):
  533         log.error("file_tree: %s exists, but is not a directory", host_dir)
  534         return ngroup_pillar
  535 
  536     host_pillar = _construct_pillar(
  537         host_dir,
  538         follow_dir_links,
  539         keep_newline,
  540         render_default,
  541         renderer_blacklist,
  542         renderer_whitelist,
  543         template,
  544     )
  545     return salt.utils.dictupdate.merge(ngroup_pillar, host_pillar, strategy="recurse")