"Fossies" - the Fresh Open Source Software Archive

Member "salt-3002.2/salt/states/file.py" (18 Nov 2020, 309936 Bytes) of package /linux/misc/salt-3002.2.tar.gz:


As a special service "Fossies" has tried to format the requested source page into HTML format using (guessed) Python source code syntax highlighting (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file. For more information about "file.py" see the Fossies "Dox" file reference documentation and the latest Fossies "Diffs" side-by-side code changes report: 3002.1_vs_3002.2.

    1 """
    2 Operations on regular files, special files, directories, and symlinks
    3 =====================================================================
    4 
    5 Salt States can aggressively manipulate files on a system. There are a number
    6 of ways in which files can be managed.
    7 
    8 Regular files can be enforced with the :mod:`file.managed
    9 <salt.states.file.managed>` state. This state downloads files from the salt
   10 master and places them on the target system. Managed files can be rendered as a
   11 jinja, mako, or wempy template, adding a dynamic component to file management.
   12 An example of :mod:`file.managed <salt.states.file.managed>` which makes use of
   13 the jinja templating system would look like this:
   14 
   15 .. code-block:: jinja
   16 
   17     /etc/http/conf/http.conf:
   18       file.managed:
   19         - source: salt://apache/http.conf
   20         - user: root
   21         - group: root
   22         - mode: 644
   23         - attrs: ai
   24         - template: jinja
   25         - defaults:
   26             custom_var: "default value"
   27             other_var: 123
   28     {% if grains['os'] == 'Ubuntu' %}
   29         - context:
   30             custom_var: "override"
   31     {% endif %}
   32 
   33 It is also possible to use the :mod:`py renderer <salt.renderers.py>` as a
   34 templating option. The template would be a Python script which would need to
   35 contain a function called ``run()``, which returns a string. All arguments
   36 to the state will be made available to the Python script as globals. The
   37 returned string will be the contents of the managed file. For example:
   38 
   39 .. code-block:: python
   40 
   41     def run():
   42         lines = ['foo', 'bar', 'baz']
   43         lines.extend([source, name, user, context])  # Arguments as globals
   44         return '\\n\\n'.join(lines)
   45 
   46 .. note::
   47 
   48     The ``defaults`` and ``context`` arguments require extra indentation (four
   49     spaces instead of the normal two) in order to create a nested dictionary.
   50     :ref:`More information <nested-dict-indentation>`.
   51 
   52 If using a template, any user-defined template variables in the file defined in
   53 ``source`` must be passed in using the ``defaults`` and/or ``context``
   54 arguments. The general best practice is to place default values in
   55 ``defaults``, with conditional overrides going into ``context``, as seen above.
   56 
   57 The template will receive a variable ``custom_var``, which would be accessed in
   58 the template using ``{{ custom_var }}``. If the operating system is Ubuntu, the
   59 value of the variable ``custom_var`` would be *override*, otherwise it is the
   60 default *default value*
   61 
   62 The ``source`` parameter can be specified as a list. If this is done, then the
   63 first file to be matched will be the one that is used. This allows you to have
   64 a default file on which to fall back if the desired file does not exist on the
   65 salt fileserver. Here's an example:
   66 
   67 .. code-block:: jinja
   68 
   69     /etc/foo.conf:
   70       file.managed:
   71         - source:
   72           - salt://foo.conf.{{ grains['fqdn'] }}
   73           - salt://foo.conf.fallback
   74         - user: foo
   75         - group: users
   76         - mode: 644
   77         - attrs: i
   78         - backup: minion
   79 
   80 .. note::
   81 
   82     Salt supports backing up managed files via the backup option. For more
   83     details on this functionality please review the
   84     :ref:`backup_mode documentation <file-state-backups>`.
   85 
   86 The ``source`` parameter can also specify a file in another Salt environment.
   87 In this example ``foo.conf`` in the ``dev`` environment will be used instead.
   88 
   89 .. code-block:: yaml
   90 
   91     /etc/foo.conf:
   92       file.managed:
   93         - source:
   94           - 'salt://foo.conf?saltenv=dev'
   95         - user: foo
   96         - group: users
   97         - mode: '0644'
   98         - attrs: i
   99 
  100 .. warning::
  101 
  102     When using a mode that includes a leading zero you must wrap the
  103     value in single quotes. If the value is not wrapped in quotes it
  104     will be read by YAML as an integer and evaluated as an octal.
  105 
  106 The ``names`` parameter, which is part of the state compiler, can be used to
  107 expand the contents of a single state declaration into multiple, single state
  108 declarations. Each item in the ``names`` list receives its own individual state
  109 ``name`` and is converted into its own low-data structure. This is a convenient
  110 way to manage several files with similar attributes.
  111 
  112 .. code-block:: yaml
  113 
  114     salt_master_conf:
  115       file.managed:
  116         - user: root
  117         - group: root
  118         - mode: '0644'
  119         - names:
  120           - /etc/salt/master.d/master.conf:
  121             - source: salt://saltmaster/master.conf
  122           - /etc/salt/minion.d/minion-99.conf:
  123             - source: salt://saltmaster/minion.conf
  124 
  125 .. note::
  126 
  127     There is more documentation about this feature in the :ref:`Names declaration
  128     <names-declaration>` section of the :ref:`Highstate docs <states-highstate>`.
  129 
  130 Special files can be managed via the ``mknod`` function. This function will
  131 create and enforce the permissions on a special file. The function supports the
  132 creation of character devices, block devices, and FIFO pipes. The function will
  133 create the directory structure up to the special file if it is needed on the
  134 minion. The function will not overwrite or operate on (change major/minor
  135 numbers) existing special files with the exception of user, group, and
  136 permissions. In most cases the creation of some special files require root
  137 permissions on the minion. This would require that the minion to be run as the
  138 root user. Here is an example of a character device:
  139 
  140 .. code-block:: yaml
  141 
  142     /var/named/chroot/dev/random:
  143       file.mknod:
  144         - ntype: c
  145         - major: 1
  146         - minor: 8
  147         - user: named
  148         - group: named
  149         - mode: 660
  150 
  151 Here is an example of a block device:
  152 
  153 .. code-block:: yaml
  154 
  155     /var/named/chroot/dev/loop0:
  156       file.mknod:
  157         - ntype: b
  158         - major: 7
  159         - minor: 0
  160         - user: named
  161         - group: named
  162         - mode: 660
  163 
  164 Here is an example of a fifo pipe:
  165 
  166 .. code-block:: yaml
  167 
  168     /var/named/chroot/var/log/logfifo:
  169       file.mknod:
  170         - ntype: p
  171         - user: named
  172         - group: named
  173         - mode: 660
  174 
  175 Directories can be managed via the ``directory`` function. This function can
  176 create and enforce the permissions on a directory. A directory statement will
  177 look like this:
  178 
  179 .. code-block:: yaml
  180 
  181     /srv/stuff/substuf:
  182       file.directory:
  183         - user: fred
  184         - group: users
  185         - mode: 755
  186         - makedirs: True
  187 
  188 If you need to enforce user and/or group ownership or permissions recursively
  189 on the directory's contents, you can do so by adding a ``recurse`` directive:
  190 
  191 .. code-block:: yaml
  192 
  193     /srv/stuff/substuf:
  194       file.directory:
  195         - user: fred
  196         - group: users
  197         - mode: 755
  198         - makedirs: True
  199         - recurse:
  200           - user
  201           - group
  202           - mode
  203 
  204 As a default, ``mode`` will resolve to ``dir_mode`` and ``file_mode``, to
  205 specify both directory and file permissions, use this form:
  206 
  207 .. code-block:: yaml
  208 
  209     /srv/stuff/substuf:
  210       file.directory:
  211         - user: fred
  212         - group: users
  213         - file_mode: 744
  214         - dir_mode: 755
  215         - makedirs: True
  216         - recurse:
  217           - user
  218           - group
  219           - mode
  220 
  221 Symlinks can be easily created; the symlink function is very simple and only
  222 takes a few arguments:
  223 
  224 .. code-block:: yaml
  225 
  226     /etc/grub.conf:
  227       file.symlink:
  228         - target: /boot/grub/grub.conf
  229 
  230 Recursive directory management can also be set via the ``recurse``
  231 function. Recursive directory management allows for a directory on the salt
  232 master to be recursively copied down to the minion. This is a great tool for
  233 deploying large code and configuration systems. A state using ``recurse``
  234 would look something like this:
  235 
  236 .. code-block:: yaml
  237 
  238     /opt/code/flask:
  239       file.recurse:
  240         - source: salt://code/flask
  241         - include_empty: True
  242 
  243 A more complex ``recurse`` example:
  244 
  245 .. code-block:: jinja
  246 
  247     {% set site_user = 'testuser' %}
  248     {% set site_name = 'test_site' %}
  249     {% set project_name = 'test_proj' %}
  250     {% set sites_dir = 'test_dir' %}
  251 
  252     django-project:
  253       file.recurse:
  254         - name: {{ sites_dir }}/{{ site_name }}/{{ project_name }}
  255         - user: {{ site_user }}
  256         - dir_mode: 2775
  257         - file_mode: '0644'
  258         - template: jinja
  259         - source: salt://project/templates_dir
  260         - include_empty: True
  261 
  262 Retention scheduling can be applied to manage contents of backup directories.
  263 For example:
  264 
  265 .. code-block:: yaml
  266 
  267     /var/backups/example_directory:
  268       file.retention_schedule:
  269         - strptime_format: example_name_%Y%m%dT%H%M%S.tar.bz2
  270         - retain:
  271             most_recent: 5
  272             first_of_hour: 4
  273             first_of_day: 14
  274             first_of_week: 6
  275             first_of_month: 6
  276             first_of_year: all
  277 
  278 """
  279 
  280 
  281 import copy
  282 import difflib
  283 import itertools
  284 import logging
  285 import os
  286 import posixpath
  287 import re
  288 import shutil
  289 import sys
  290 import time
  291 import traceback
  292 from collections import defaultdict
  293 from collections.abc import Iterable, Mapping
  294 from datetime import date, datetime  # python3 problem in the making?
  295 
  296 import salt.loader
  297 import salt.payload
  298 import salt.utils.data
  299 import salt.utils.dateutils
  300 import salt.utils.dictupdate
  301 import salt.utils.files
  302 import salt.utils.hashutils
  303 import salt.utils.path
  304 import salt.utils.platform
  305 import salt.utils.stringutils
  306 import salt.utils.templates
  307 import salt.utils.url
  308 import salt.utils.versions
  309 from salt.exceptions import CommandExecutionError
  310 from salt.ext.six.moves import zip_longest
  311 from salt.ext.six.moves.urllib.parse import urlparse as _urlparse
  312 from salt.serializers import DeserializationError
  313 from salt.state import get_accumulator_dir as _get_accumulator_dir
  314 
  315 if salt.utils.platform.is_windows():
  316     import salt.utils.win_dacl
  317     import salt.utils.win_functions
  318     import salt.utils.winapi
  319 
  320 if salt.utils.platform.is_windows():
  321     import pywintypes
  322     import win32com.client
  323 
  324 log = logging.getLogger(__name__)
  325 
  326 COMMENT_REGEX = r"^([[:space:]]*){0}[[:space:]]?"
  327 __NOT_FOUND = object()
  328 
  329 __func_alias__ = {
  330     "copy_": "copy",
  331 }
  332 
  333 
  334 def _get_accumulator_filepath():
  335     """
  336     Return accumulator data path.
  337     """
  338     return os.path.join(_get_accumulator_dir(__opts__["cachedir"]), __instance_id__)
  339 
  340 
  341 def _load_accumulators():
  342     def _deserialize(path):
  343         serial = salt.payload.Serial(__opts__)
  344         ret = {"accumulators": {}, "accumulators_deps": {}}
  345         try:
  346             with salt.utils.files.fopen(path, "rb") as f:
  347                 loaded = serial.load(f)
  348                 return loaded if loaded else ret
  349         except (OSError, NameError):
  350             # NameError is a msgpack error from salt-ssh
  351             return ret
  352 
  353     loaded = _deserialize(_get_accumulator_filepath())
  354 
  355     return loaded["accumulators"], loaded["accumulators_deps"]
  356 
  357 
  358 def _persist_accummulators(accumulators, accumulators_deps):
  359     accumm_data = {"accumulators": accumulators, "accumulators_deps": accumulators_deps}
  360 
  361     serial = salt.payload.Serial(__opts__)
  362     try:
  363         with salt.utils.files.fopen(_get_accumulator_filepath(), "w+b") as f:
  364             serial.dump(accumm_data, f)
  365     except NameError:
  366         # msgpack error from salt-ssh
  367         pass
  368 
  369 
  370 def _check_user(user, group):
  371     """
  372     Checks if the named user and group are present on the minion
  373     """
  374     err = ""
  375     if user:
  376         uid = __salt__["file.user_to_uid"](user)
  377         if uid == "":
  378             err += "User {} is not available ".format(user)
  379     if group:
  380         gid = __salt__["file.group_to_gid"](group)
  381         if gid == "":
  382             err += "Group {} is not available".format(group)
  383     return err
  384 
  385 
  386 def _is_valid_relpath(relpath, maxdepth=None):
  387     """
  388     Performs basic sanity checks on a relative path.
  389 
  390     Requires POSIX-compatible paths (i.e. the kind obtained through
  391     cp.list_master or other such calls).
  392 
  393     Ensures that the path does not contain directory transversal, and
  394     that it does not exceed a stated maximum depth (if specified).
  395     """
  396     # Check relpath surrounded by slashes, so that `..` can be caught as
  397     # a path component at the start, end, and in the middle of the path.
  398     sep, pardir = posixpath.sep, posixpath.pardir
  399     if sep + pardir + sep in sep + relpath + sep:
  400         return False
  401 
  402     # Check that the relative path's depth does not exceed maxdepth
  403     if maxdepth is not None:
  404         path_depth = relpath.strip(sep).count(sep)
  405         if path_depth > maxdepth:
  406             return False
  407 
  408     return True
  409 
  410 
  411 def _salt_to_os_path(path):
  412     """
  413     Converts a path from the form received via salt master to the OS's native
  414     path format.
  415     """
  416     return os.path.normpath(path.replace(posixpath.sep, os.path.sep))
  417 
  418 
  419 def _gen_recurse_managed_files(
  420     name,
  421     source,
  422     keep_symlinks=False,
  423     include_pat=None,
  424     exclude_pat=None,
  425     maxdepth=None,
  426     include_empty=False,
  427     **kwargs
  428 ):
  429     """
  430     Generate the list of files managed by a recurse state
  431     """
  432 
  433     # Convert a relative path generated from salt master paths to an OS path
  434     # using "name" as the base directory
  435     def full_path(master_relpath):
  436         return os.path.join(name, _salt_to_os_path(master_relpath))
  437 
  438     # Process symlinks and return the updated filenames list
  439     def process_symlinks(filenames, symlinks):
  440         for lname, ltarget in symlinks.items():
  441             srelpath = posixpath.relpath(lname, srcpath)
  442             if not _is_valid_relpath(srelpath, maxdepth=maxdepth):
  443                 continue
  444             if not salt.utils.stringutils.check_include_exclude(
  445                 srelpath, include_pat, exclude_pat
  446             ):
  447                 continue
  448             # Check for all paths that begin with the symlink
  449             # and axe it leaving only the dirs/files below it.
  450             # This needs to use list() otherwise they reference
  451             # the same list.
  452             _filenames = list(filenames)
  453             for filename in _filenames:
  454                 if filename.startswith(lname):
  455                     log.debug(
  456                         "** skipping file ** {}, it intersects a "
  457                         "symlink".format(filename)
  458                     )
  459                     filenames.remove(filename)
  460             # Create the symlink along with the necessary dirs.
  461             # The dir perms/ownership will be adjusted later
  462             # if needed
  463             managed_symlinks.add((srelpath, ltarget))
  464 
  465             # Add the path to the keep set in case clean is set to True
  466             keep.add(full_path(srelpath))
  467         vdir.update(keep)
  468         return filenames
  469 
  470     managed_files = set()
  471     managed_directories = set()
  472     managed_symlinks = set()
  473     keep = set()
  474     vdir = set()
  475 
  476     srcpath, senv = salt.utils.url.parse(source)
  477     if senv is None:
  478         senv = __env__
  479     if not srcpath.endswith(posixpath.sep):
  480         # we're searching for things that start with this *directory*.
  481         srcpath = srcpath + posixpath.sep
  482     fns_ = __salt__["cp.list_master"](senv, srcpath)
  483 
  484     # If we are instructed to keep symlinks, then process them.
  485     if keep_symlinks:
  486         # Make this global so that emptydirs can use it if needed.
  487         symlinks = __salt__["cp.list_master_symlinks"](senv, srcpath)
  488         fns_ = process_symlinks(fns_, symlinks)
  489 
  490     for fn_ in fns_:
  491         if not fn_.strip():
  492             continue
  493 
  494         # fn_ here is the absolute (from file_roots) source path of
  495         # the file to copy from; it is either a normal file or an
  496         # empty dir(if include_empty==true).
  497 
  498         relname = salt.utils.data.decode(posixpath.relpath(fn_, srcpath))
  499         if not _is_valid_relpath(relname, maxdepth=maxdepth):
  500             continue
  501 
  502         # Check if it is to be excluded. Match only part of the path
  503         # relative to the target directory
  504         if not salt.utils.stringutils.check_include_exclude(
  505             relname, include_pat, exclude_pat
  506         ):
  507             continue
  508         dest = full_path(relname)
  509         dirname = os.path.dirname(dest)
  510         keep.add(dest)
  511 
  512         if dirname not in vdir:
  513             # verify the directory perms if they are set
  514             managed_directories.add(dirname)
  515             vdir.add(dirname)
  516 
  517         src = salt.utils.url.create(fn_, saltenv=senv)
  518         managed_files.add((dest, src))
  519 
  520     if include_empty:
  521         mdirs = __salt__["cp.list_master_dirs"](senv, srcpath)
  522         for mdir in mdirs:
  523             relname = posixpath.relpath(mdir, srcpath)
  524             if not _is_valid_relpath(relname, maxdepth=maxdepth):
  525                 continue
  526             if not salt.utils.stringutils.check_include_exclude(
  527                 relname, include_pat, exclude_pat
  528             ):
  529                 continue
  530             mdest = full_path(relname)
  531             # Check for symlinks that happen to point to an empty dir.
  532             if keep_symlinks:
  533                 islink = False
  534                 for link in symlinks:
  535                     if mdir.startswith(link, 0):
  536                         log.debug(
  537                             "** skipping empty dir ** {}, it intersects"
  538                             " a symlink".format(mdir)
  539                         )
  540                         islink = True
  541                         break
  542                 if islink:
  543                     continue
  544 
  545             managed_directories.add(mdest)
  546             keep.add(mdest)
  547 
  548     return managed_files, managed_directories, managed_symlinks, keep
  549 
  550 
  551 def _gen_keep_files(name, require, walk_d=None):
  552     """
  553     Generate the list of files that need to be kept when a dir based function
  554     like directory or recurse has a clean.
  555     """
  556 
  557     def _is_child(path, directory):
  558         """
  559         Check whether ``path`` is child of ``directory``
  560         """
  561         path = os.path.abspath(path)
  562         directory = os.path.abspath(directory)
  563 
  564         relative = os.path.relpath(path, directory)
  565 
  566         return not relative.startswith(os.pardir)
  567 
  568     def _add_current_path(path):
  569         _ret = set()
  570         if os.path.isdir(path):
  571             dirs, files = walk_d.get(path, ((), ()))
  572             _ret.add(path)
  573             for _name in files:
  574                 _ret.add(os.path.join(path, _name))
  575             for _name in dirs:
  576                 _ret.add(os.path.join(path, _name))
  577         return _ret
  578 
  579     def _process_by_walk_d(name, ret):
  580         if os.path.isdir(name):
  581             walk_ret.update(_add_current_path(name))
  582             dirs, _ = walk_d.get(name, ((), ()))
  583             for _d in dirs:
  584                 p = os.path.join(name, _d)
  585                 walk_ret.update(_add_current_path(p))
  586                 _process_by_walk_d(p, ret)
  587 
  588     def _process(name):
  589         ret = set()
  590         if os.path.isdir(name):
  591             for root, dirs, files in salt.utils.path.os_walk(name):
  592                 ret.add(name)
  593                 for name in files:
  594                     ret.add(os.path.join(root, name))
  595                 for name in dirs:
  596                     ret.add(os.path.join(root, name))
  597         return ret
  598 
  599     keep = set()
  600     if isinstance(require, list):
  601         required_files = [comp for comp in require if "file" in comp]
  602         for comp in required_files:
  603             for low in __lowstate__:
  604                 # A requirement should match either the ID and the name of
  605                 # another state.
  606                 if low["name"] == comp["file"] or low["__id__"] == comp["file"]:
  607                     fn = low["name"]
  608                     fun = low["fun"]
  609                     if os.path.isdir(fn):
  610                         if _is_child(fn, name):
  611                             if fun == "recurse":
  612                                 fkeep = _gen_recurse_managed_files(**low)[3]
  613                                 log.debug("Keep from {}: {}".format(fn, fkeep))
  614                                 keep.update(fkeep)
  615                             elif walk_d:
  616                                 walk_ret = set()
  617                                 _process_by_walk_d(fn, walk_ret)
  618                                 keep.update(walk_ret)
  619                             else:
  620                                 keep.update(_process(fn))
  621                     else:
  622                         keep.add(fn)
  623     log.debug("Files to keep from required states: {}".format(list(keep)))
  624     return list(keep)
  625 
  626 
  627 def _check_file(name):
  628     ret = True
  629     msg = ""
  630 
  631     if not os.path.isabs(name):
  632         ret = False
  633         msg = "Specified file {} is not an absolute path".format(name)
  634     elif not os.path.exists(name):
  635         ret = False
  636         msg = "{}: file not found".format(name)
  637 
  638     return ret, msg
  639 
  640 
  641 def _find_keep_files(root, keep):
  642     """
  643     Compile a list of valid keep files (and directories).
  644     Used by _clean_dir()
  645     """
  646     real_keep = set()
  647     real_keep.add(root)
  648     if isinstance(keep, list):
  649         for fn_ in keep:
  650             if not os.path.isabs(fn_):
  651                 continue
  652             fn_ = os.path.normcase(os.path.abspath(fn_))
  653             real_keep.add(fn_)
  654             while True:
  655                 fn_ = os.path.abspath(os.path.dirname(fn_))
  656                 real_keep.add(fn_)
  657                 drive, path = os.path.splitdrive(fn_)
  658                 if not path.lstrip(os.sep):
  659                     break
  660     return real_keep
  661 
  662 
  663 def _clean_dir(root, keep, exclude_pat):
  664     """
  665     Clean out all of the files and directories in a directory (root) while
  666     preserving the files in a list (keep) and part of exclude_pat
  667     """
  668     root = os.path.normcase(root)
  669     real_keep = _find_keep_files(root, keep)
  670     removed = set()
  671 
  672     def _delete_not_kept(nfn):
  673         if nfn not in real_keep:
  674             # -- check if this is a part of exclude_pat(only). No need to
  675             # check include_pat
  676             if not salt.utils.stringutils.check_include_exclude(
  677                 os.path.relpath(nfn, root), None, exclude_pat
  678             ):
  679                 return
  680             removed.add(nfn)
  681             if not __opts__["test"]:
  682                 try:
  683                     os.remove(nfn)
  684                 except OSError:
  685                     __salt__["file.remove"](nfn)
  686 
  687     for roots, dirs, files in salt.utils.path.os_walk(root):
  688         for name in itertools.chain(dirs, files):
  689             _delete_not_kept(os.path.join(roots, name))
  690     return list(removed)
  691 
  692 
  693 def _error(ret, err_msg):
  694     ret["result"] = False
  695     ret["comment"] = err_msg
  696     return ret
  697 
  698 
  699 def _check_directory(
  700     name,
  701     user=None,
  702     group=None,
  703     recurse=False,
  704     dir_mode=None,
  705     file_mode=None,
  706     clean=False,
  707     require=False,
  708     exclude_pat=None,
  709     max_depth=None,
  710     follow_symlinks=False,
  711 ):
  712     """
  713     Check what changes need to be made on a directory
  714     """
  715     changes = {}
  716     if recurse or clean:
  717         assert max_depth is None or not clean
  718         # walk path only once and store the result
  719         walk_l = list(_depth_limited_walk(name, max_depth))
  720         # root: (dirs, files) structure, compatible for python2.6
  721         walk_d = {}
  722         for i in walk_l:
  723             walk_d[i[0]] = (i[1], i[2])
  724 
  725     if recurse:
  726         try:
  727             recurse_set = _get_recurse_set(recurse)
  728         except (TypeError, ValueError) as exc:
  729             return False, "{}".format(exc), changes
  730         if "user" not in recurse_set:
  731             user = None
  732         if "group" not in recurse_set:
  733             group = None
  734         if "mode" not in recurse_set:
  735             dir_mode = None
  736             file_mode = None
  737 
  738         check_files = "ignore_files" not in recurse_set
  739         check_dirs = "ignore_dirs" not in recurse_set
  740         for root, dirs, files in walk_l:
  741             if check_files:
  742                 for fname in files:
  743                     fchange = {}
  744                     path = os.path.join(root, fname)
  745                     stats = __salt__["file.stats"](path, None, follow_symlinks)
  746                     if user is not None and user != stats.get("user"):
  747                         fchange["user"] = user
  748                     if group is not None and group != stats.get("group"):
  749                         fchange["group"] = group
  750                     smode = salt.utils.files.normalize_mode(stats.get("mode"))
  751                     file_mode = salt.utils.files.normalize_mode(file_mode)
  752                     if (
  753                         file_mode is not None
  754                         and file_mode != smode
  755                         and (
  756                             # Ignore mode for symlinks on linux based systems where we can not
  757                             # change symlink file permissions
  758                             follow_symlinks
  759                             or stats.get("type") != "link"
  760                             or not salt.utils.platform.is_linux()
  761                         )
  762                     ):
  763                         fchange["mode"] = file_mode
  764                     if fchange:
  765                         changes[path] = fchange
  766             if check_dirs:
  767                 for name_ in dirs:
  768                     path = os.path.join(root, name_)
  769                     fchange = _check_dir_meta(
  770                         path, user, group, dir_mode, follow_symlinks
  771                     )
  772                     if fchange:
  773                         changes[path] = fchange
  774     # Recurse skips root (we always do dirs, not root), so always check root:
  775     fchange = _check_dir_meta(name, user, group, dir_mode, follow_symlinks)
  776     if fchange:
  777         changes[name] = fchange
  778     if clean:
  779         keep = _gen_keep_files(name, require, walk_d)
  780 
  781         def _check_changes(fname):
  782             path = os.path.join(root, fname)
  783             if path in keep:
  784                 return {}
  785             else:
  786                 if not salt.utils.stringutils.check_include_exclude(
  787                     os.path.relpath(path, name), None, exclude_pat
  788                 ):
  789                     return {}
  790                 else:
  791                     return {path: {"removed": "Removed due to clean"}}
  792 
  793         for root, dirs, files in walk_l:
  794             for fname in files:
  795                 changes.update(_check_changes(fname))
  796             for name_ in dirs:
  797                 changes.update(_check_changes(name_))
  798 
  799     if not os.path.isdir(name):
  800         changes[name] = {"directory": "new"}
  801     if changes:
  802         comments = ["The following files will be changed:\n"]
  803         for fn_ in changes:
  804             for key, val in changes[fn_].items():
  805                 comments.append("{}: {} - {}\n".format(fn_, key, val))
  806         return None, "".join(comments), changes
  807     return True, "The directory {} is in the correct state".format(name), changes
  808 
  809 
  810 def _check_directory_win(
  811     name,
  812     win_owner=None,
  813     win_perms=None,
  814     win_deny_perms=None,
  815     win_inheritance=None,
  816     win_perms_reset=None,
  817 ):
  818     """
  819     Check what changes need to be made on a directory
  820     """
  821     changes = {}
  822 
  823     if not os.path.isdir(name):
  824         changes = {name: {"directory": "new"}}
  825     else:
  826         # Check owner by SID
  827         if win_owner is not None:
  828             current_owner = salt.utils.win_dacl.get_owner(name)
  829             current_owner_sid = salt.utils.win_functions.get_sid_from_name(
  830                 current_owner
  831             )
  832             expected_owner_sid = salt.utils.win_functions.get_sid_from_name(win_owner)
  833             if not current_owner_sid == expected_owner_sid:
  834                 changes["owner"] = win_owner
  835 
  836         # Check perms
  837         perms = salt.utils.win_dacl.get_permissions(name)
  838 
  839         # Verify Permissions
  840         if win_perms is not None:
  841             for user in win_perms:
  842                 # Check that user exists:
  843                 try:
  844                     salt.utils.win_dacl.get_name(user)
  845                 except CommandExecutionError:
  846                     continue
  847 
  848                 grant_perms = []
  849                 # Check for permissions
  850                 if isinstance(win_perms[user]["perms"], str):
  851                     if not salt.utils.win_dacl.has_permission(
  852                         name, user, win_perms[user]["perms"]
  853                     ):
  854                         grant_perms = win_perms[user]["perms"]
  855                 else:
  856                     for perm in win_perms[user]["perms"]:
  857                         if not salt.utils.win_dacl.has_permission(
  858                             name, user, perm, exact=False
  859                         ):
  860                             grant_perms.append(win_perms[user]["perms"])
  861                 if grant_perms:
  862                     if "grant_perms" not in changes:
  863                         changes["grant_perms"] = {}
  864                     if user not in changes["grant_perms"]:
  865                         changes["grant_perms"][user] = {}
  866                     changes["grant_perms"][user]["perms"] = grant_perms
  867 
  868                 # Check Applies to
  869                 if "applies_to" not in win_perms[user]:
  870                     applies_to = "this_folder_subfolders_files"
  871                 else:
  872                     applies_to = win_perms[user]["applies_to"]
  873 
  874                 if user in perms:
  875                     user = salt.utils.win_dacl.get_name(user)
  876 
  877                     # Get the proper applies_to text
  878                     at_flag = salt.utils.win_dacl.flags().ace_prop["file"][applies_to]
  879                     applies_to_text = salt.utils.win_dacl.flags().ace_prop["file"][
  880                         at_flag
  881                     ]
  882 
  883                     if "grant" in perms[user]:
  884                         if not perms[user]["grant"]["applies to"] == applies_to_text:
  885                             if "grant_perms" not in changes:
  886                                 changes["grant_perms"] = {}
  887                             if user not in changes["grant_perms"]:
  888                                 changes["grant_perms"][user] = {}
  889                             changes["grant_perms"][user]["applies_to"] = applies_to
  890 
  891         # Verify Deny Permissions
  892         if win_deny_perms is not None:
  893             for user in win_deny_perms:
  894                 # Check that user exists:
  895                 try:
  896                     salt.utils.win_dacl.get_name(user)
  897                 except CommandExecutionError:
  898                     continue
  899 
  900                 deny_perms = []
  901                 # Check for permissions
  902                 if isinstance(win_deny_perms[user]["perms"], str):
  903                     if not salt.utils.win_dacl.has_permission(
  904                         name, user, win_deny_perms[user]["perms"], "deny"
  905                     ):
  906                         deny_perms = win_deny_perms[user]["perms"]
  907                 else:
  908                     for perm in win_deny_perms[user]["perms"]:
  909                         if not salt.utils.win_dacl.has_permission(
  910                             name, user, perm, "deny", exact=False
  911                         ):
  912                             deny_perms.append(win_deny_perms[user]["perms"])
  913                 if deny_perms:
  914                     if "deny_perms" not in changes:
  915                         changes["deny_perms"] = {}
  916                     if user not in changes["deny_perms"]:
  917                         changes["deny_perms"][user] = {}
  918                     changes["deny_perms"][user]["perms"] = deny_perms
  919 
  920                 # Check Applies to
  921                 if "applies_to" not in win_deny_perms[user]:
  922                     applies_to = "this_folder_subfolders_files"
  923                 else:
  924                     applies_to = win_deny_perms[user]["applies_to"]
  925 
  926                 if user in perms:
  927                     user = salt.utils.win_dacl.get_name(user)
  928 
  929                     # Get the proper applies_to text
  930                     at_flag = salt.utils.win_dacl.flags().ace_prop["file"][applies_to]
  931                     applies_to_text = salt.utils.win_dacl.flags().ace_prop["file"][
  932                         at_flag
  933                     ]
  934 
  935                     if "deny" in perms[user]:
  936                         if not perms[user]["deny"]["applies to"] == applies_to_text:
  937                             if "deny_perms" not in changes:
  938                                 changes["deny_perms"] = {}
  939                             if user not in changes["deny_perms"]:
  940                                 changes["deny_perms"][user] = {}
  941                             changes["deny_perms"][user]["applies_to"] = applies_to
  942 
  943         # Check inheritance
  944         if win_inheritance is not None:
  945             if not win_inheritance == salt.utils.win_dacl.get_inheritance(name):
  946                 changes["inheritance"] = win_inheritance
  947 
  948         # Check reset
  949         if win_perms_reset:
  950             for user_name in perms:
  951                 if user_name not in win_perms:
  952                     if (
  953                         "grant" in perms[user_name]
  954                         and not perms[user_name]["grant"]["inherited"]
  955                     ):
  956                         if "remove_perms" not in changes:
  957                             changes["remove_perms"] = {}
  958                         changes["remove_perms"].update({user_name: perms[user_name]})
  959                 if user_name not in win_deny_perms:
  960                     if (
  961                         "deny" in perms[user_name]
  962                         and not perms[user_name]["deny"]["inherited"]
  963                     ):
  964                         if "remove_perms" not in changes:
  965                             changes["remove_perms"] = {}
  966                         changes["remove_perms"].update({user_name: perms[user_name]})
  967 
  968     if changes:
  969         return None, 'The directory "{}" will be changed'.format(name), changes
  970 
  971     return True, "The directory {} is in the correct state".format(name), changes
  972 
  973 
  974 def _check_dir_meta(name, user, group, mode, follow_symlinks=False):
  975     """
  976     Check the changes in directory metadata
  977     """
  978     try:
  979         stats = __salt__["file.stats"](name, None, follow_symlinks)
  980     except CommandExecutionError:
  981         stats = {}
  982 
  983     changes = {}
  984     if not stats:
  985         changes["directory"] = "new"
  986         return changes
  987     if user is not None and user != stats["user"] and user != stats.get("uid"):
  988         changes["user"] = user
  989     if group is not None and group != stats["group"] and group != stats.get("gid"):
  990         changes["group"] = group
  991     # Normalize the dir mode
  992     smode = salt.utils.files.normalize_mode(stats["mode"])
  993     mode = salt.utils.files.normalize_mode(mode)
  994     if (
  995         mode is not None
  996         and mode != smode
  997         and (
  998             # Ignore mode for symlinks on linux based systems where we can not
  999             # change symlink file permissions
 1000             follow_symlinks
 1001             or stats.get("type") != "link"
 1002             or not salt.utils.platform.is_linux()
 1003         )
 1004     ):
 1005         changes["mode"] = mode
 1006     return changes
 1007 
 1008 
 1009 def _check_touch(name, atime, mtime):
 1010     """
 1011     Check to see if a file needs to be updated or created
 1012     """
 1013     ret = {
 1014         "result": None,
 1015         "comment": "",
 1016         "changes": {"new": name},
 1017     }
 1018     if not os.path.exists(name):
 1019         ret["comment"] = "File {} is set to be created".format(name)
 1020     else:
 1021         stats = __salt__["file.stats"](name, follow_symlinks=False)
 1022         if (atime is not None and str(atime) != str(stats["atime"])) or (
 1023             mtime is not None and str(mtime) != str(stats["mtime"])
 1024         ):
 1025             ret["comment"] = "Times set to be updated on file {}".format(name)
 1026             ret["changes"] = {"touched": name}
 1027         else:
 1028             ret["result"] = True
 1029             ret["comment"] = "File {} exists and has the correct times".format(name)
 1030     return ret
 1031 
 1032 
 1033 def _get_symlink_ownership(path):
 1034     if salt.utils.platform.is_windows():
 1035         owner = salt.utils.win_dacl.get_owner(path)
 1036         return owner, owner
 1037     else:
 1038         return (
 1039             __salt__["file.get_user"](path, follow_symlinks=False),
 1040             __salt__["file.get_group"](path, follow_symlinks=False),
 1041         )
 1042 
 1043 
 1044 def _check_symlink_ownership(path, user, group, win_owner):
 1045     """
 1046     Check if the symlink ownership matches the specified user and group
 1047     """
 1048     cur_user, cur_group = _get_symlink_ownership(path)
 1049     if salt.utils.platform.is_windows():
 1050         return win_owner == cur_user
 1051     else:
 1052         return (cur_user == user) and (cur_group == group)
 1053 
 1054 
 1055 def _set_symlink_ownership(path, user, group, win_owner):
 1056     """
 1057     Set the ownership of a symlink and return a boolean indicating
 1058     success/failure
 1059     """
 1060     if salt.utils.platform.is_windows():
 1061         try:
 1062             salt.utils.win_dacl.set_owner(path, win_owner)
 1063         except CommandExecutionError:
 1064             pass
 1065     else:
 1066         try:
 1067             __salt__["file.lchown"](path, user, group)
 1068         except OSError:
 1069             pass
 1070     return _check_symlink_ownership(path, user, group, win_owner)
 1071 
 1072 
 1073 def _symlink_check(name, target, force, user, group, win_owner):
 1074     """
 1075     Check the symlink function
 1076     """
 1077     changes = {}
 1078     if not os.path.exists(name) and not __salt__["file.is_link"](name):
 1079         changes["new"] = name
 1080         return (
 1081             None,
 1082             "Symlink {} to {} is set for creation".format(name, target),
 1083             changes,
 1084         )
 1085     if __salt__["file.is_link"](name):
 1086         if __salt__["file.readlink"](name) != target:
 1087             changes["change"] = name
 1088             return (
 1089                 None,
 1090                 "Link {} target is set to be changed to {}".format(name, target),
 1091                 changes,
 1092             )
 1093         else:
 1094             result = True
 1095             msg = "The symlink {} is present".format(name)
 1096             if not _check_symlink_ownership(name, user, group, win_owner):
 1097                 result = None
 1098                 changes["ownership"] = "{}:{}".format(*_get_symlink_ownership(name))
 1099                 msg += (
 1100                     ", but the ownership of the symlink would be changed "
 1101                     "from {2}:{3} to {0}:{1}"
 1102                 ).format(user, group, *_get_symlink_ownership(name))
 1103             return result, msg, changes
 1104     else:
 1105         if force:
 1106             return (
 1107                 None,
 1108                 (
 1109                     "The file or directory {} is set for removal to "
 1110                     "make way for a new symlink targeting {}".format(name, target)
 1111                 ),
 1112                 changes,
 1113             )
 1114         return (
 1115             False,
 1116             (
 1117                 "File or directory exists where the symlink {} "
 1118                 "should be. Did you mean to use force?".format(name)
 1119             ),
 1120             changes,
 1121         )
 1122 
 1123 
 1124 def _hardlink_same(name, target):
 1125     """
 1126     Check to see if the inodes match for the name and the target
 1127     """
 1128     res = __salt__["file.stats"](name, None, follow_symlinks=False)
 1129     if "inode" not in res:
 1130         return False
 1131     name_i = res["inode"]
 1132 
 1133     res = __salt__["file.stats"](target, None, follow_symlinks=False)
 1134     if "inode" not in res:
 1135         return False
 1136     target_i = res["inode"]
 1137 
 1138     return name_i == target_i
 1139 
 1140 
 1141 def _hardlink_check(name, target, force):
 1142     """
 1143     Check the hardlink function
 1144     """
 1145     changes = {}
 1146     if not os.path.exists(target):
 1147         msg = "Target {} for hard link does not exist".format(target)
 1148         return False, msg, changes
 1149 
 1150     elif os.path.isdir(target):
 1151         msg = "Unable to hard link from directory {}".format(target)
 1152         return False, msg, changes
 1153 
 1154     if os.path.isdir(name):
 1155         msg = "Unable to hard link to directory {}".format(name)
 1156         return False, msg, changes
 1157 
 1158     elif not os.path.exists(name):
 1159         msg = "Hard link {} to {} is set for creation".format(name, target)
 1160         changes["new"] = name
 1161         return None, msg, changes
 1162 
 1163     elif __salt__["file.is_hardlink"](name):
 1164         if _hardlink_same(name, target):
 1165             msg = "The hard link {} is presently targetting {}".format(name, target)
 1166             return True, msg, changes
 1167 
 1168         msg = "Link {} target is set to be changed to {}".format(name, target)
 1169         changes["change"] = name
 1170         return None, msg, changes
 1171 
 1172     if force:
 1173         msg = (
 1174             "The file or directory {} is set for removal to "
 1175             "make way for a new hard link targeting {}".format(name, target)
 1176         )
 1177         return None, msg, changes
 1178 
 1179     msg = (
 1180         "File or directory exists where the hard link {} "
 1181         "should be. Did you mean to use force?".format(name)
 1182     )
 1183     return False, msg, changes
 1184 
 1185 
 1186 def _test_owner(kwargs, user=None):
 1187     """
 1188     Convert owner to user, since other config management tools use owner,
 1189     no need to punish people coming from other systems.
 1190     PLEASE DO NOT DOCUMENT THIS! WE USE USER, NOT OWNER!!!!
 1191     """
 1192     if user:
 1193         return user
 1194     if "owner" in kwargs:
 1195         log.warning(
 1196             'Use of argument owner found, "owner" is invalid, please ' 'use "user"'
 1197         )
 1198         return kwargs["owner"]
 1199 
 1200     return user
 1201 
 1202 
 1203 def _unify_sources_and_hashes(
 1204     source=None, source_hash=None, sources=None, source_hashes=None
 1205 ):
 1206     """
 1207     Silly little function to give us a standard tuple list for sources and
 1208     source_hashes
 1209     """
 1210     if sources is None:
 1211         sources = []
 1212 
 1213     if source_hashes is None:
 1214         source_hashes = []
 1215 
 1216     if source and sources:
 1217         return (False, "source and sources are mutually exclusive", [])
 1218 
 1219     if source_hash and source_hashes:
 1220         return (False, "source_hash and source_hashes are mutually exclusive", [])
 1221 
 1222     if source:
 1223         return (True, "", [(source, source_hash)])
 1224 
 1225     # Make a nice neat list of tuples exactly len(sources) long..
 1226     return True, "", list(zip_longest(sources, source_hashes[: len(sources)]))
 1227 
 1228 
 1229 def _get_template_texts(
 1230     source_list=None, template="jinja", defaults=None, context=None, **kwargs
 1231 ):
 1232     """
 1233     Iterate a list of sources and process them as templates.
 1234     Returns a list of 'chunks' containing the rendered templates.
 1235     """
 1236 
 1237     ret = {
 1238         "name": "_get_template_texts",
 1239         "changes": {},
 1240         "result": True,
 1241         "comment": "",
 1242         "data": [],
 1243     }
 1244 
 1245     if source_list is None:
 1246         return _error(ret, "_get_template_texts called with empty source_list")
 1247 
 1248     txtl = []
 1249 
 1250     for (source, source_hash) in source_list:
 1251 
 1252         tmpctx = defaults if defaults else {}
 1253         if context:
 1254             tmpctx.update(context)
 1255         rndrd_templ_fn = __salt__["cp.get_template"](
 1256             source, "", template=template, saltenv=__env__, context=tmpctx, **kwargs
 1257         )
 1258         msg = "cp.get_template returned {0} (Called with: {1})"
 1259         log.debug(msg.format(rndrd_templ_fn, source))
 1260         if rndrd_templ_fn:
 1261             tmplines = None
 1262             with salt.utils.files.fopen(rndrd_templ_fn, "rb") as fp_:
 1263                 tmplines = fp_.read()
 1264                 tmplines = salt.utils.stringutils.to_unicode(tmplines)
 1265                 tmplines = tmplines.splitlines(True)
 1266             if not tmplines:
 1267                 msg = "Failed to read rendered template file {0} ({1})"
 1268                 log.debug(msg.format(rndrd_templ_fn, source))
 1269                 ret["name"] = source
 1270                 return _error(ret, msg.format(rndrd_templ_fn, source))
 1271             txtl.append("".join(tmplines))
 1272         else:
 1273             msg = "Failed to load template file {}".format(source)
 1274             log.debug(msg)
 1275             ret["name"] = source
 1276             return _error(ret, msg)
 1277 
 1278     ret["data"] = txtl
 1279     return ret
 1280 
 1281 
 1282 def _validate_str_list(arg, encoding=None):
 1283     """
 1284     ensure ``arg`` is a list of strings
 1285     """
 1286     if isinstance(arg, bytes):
 1287         ret = [salt.utils.stringutils.to_unicode(arg, encoding=encoding)]
 1288     elif isinstance(arg, str):
 1289         ret = [arg]
 1290     elif isinstance(arg, Iterable) and not isinstance(arg, Mapping):
 1291         ret = []
 1292         for item in arg:
 1293             if isinstance(item, str):
 1294                 ret.append(item)
 1295             else:
 1296                 ret.append(str(item))
 1297     else:
 1298         ret = [str(arg)]
 1299     return ret
 1300 
 1301 
 1302 def _get_shortcut_ownership(path):
 1303     return __salt__["file.get_user"](path, follow_symlinks=False)
 1304 
 1305 
 1306 def _check_shortcut_ownership(path, user):
 1307     """
 1308     Check if the shortcut ownership matches the specified user
 1309     """
 1310     cur_user = _get_shortcut_ownership(path)
 1311     return cur_user == user
 1312 
 1313 
 1314 def _set_shortcut_ownership(path, user):
 1315     """
 1316     Set the ownership of a shortcut and return a boolean indicating
 1317     success/failure
 1318     """
 1319     try:
 1320         __salt__["file.lchown"](path, user)
 1321     except OSError:
 1322         pass
 1323     return _check_shortcut_ownership(path, user)
 1324 
 1325 
 1326 def _shortcut_check(
 1327     name, target, arguments, working_dir, description, icon_location, force, user
 1328 ):
 1329     """
 1330     Check the shortcut function
 1331     """
 1332     changes = {}
 1333     if not os.path.exists(name):
 1334         changes["new"] = name
 1335         return (
 1336             None,
 1337             'Shortcut "{}" to "{}" is set for creation'.format(name, target),
 1338             changes,
 1339         )
 1340 
 1341     if os.path.isfile(name):
 1342         with salt.utils.winapi.Com():
 1343             shell = win32com.client.Dispatch("WScript.Shell")
 1344             scut = shell.CreateShortcut(name)
 1345             state_checks = [scut.TargetPath.lower() == target.lower()]
 1346             if arguments is not None:
 1347                 state_checks.append(scut.Arguments == arguments)
 1348             if working_dir is not None:
 1349                 state_checks.append(
 1350                     scut.WorkingDirectory.lower() == working_dir.lower()
 1351                 )
 1352             if description is not None:
 1353                 state_checks.append(scut.Description == description)
 1354             if icon_location is not None:
 1355                 state_checks.append(scut.IconLocation.lower() == icon_location.lower())
 1356 
 1357         if not all(state_checks):
 1358             changes["change"] = name
 1359             return (
 1360                 None,
 1361                 'Shortcut "{}" target is set to be changed to "{}"'.format(
 1362                     name, target
 1363                 ),
 1364                 changes,
 1365             )
 1366         else:
 1367             result = True
 1368             msg = 'The shortcut "{}" is present'.format(name)
 1369             if not _check_shortcut_ownership(name, user):
 1370                 result = None
 1371                 changes["ownership"] = "{}".format(_get_shortcut_ownership(name))
 1372                 msg += (
 1373                     ", but the ownership of the shortcut would be changed "
 1374                     "from {1} to {0}"
 1375                 ).format(user, _get_shortcut_ownership(name))
 1376             return result, msg, changes
 1377     else:
 1378         if force:
 1379             return (
 1380                 None,
 1381                 (
 1382                     'The link or directory "{}" is set for removal to '
 1383                     'make way for a new shortcut targeting "{}"'.format(name, target)
 1384                 ),
 1385                 changes,
 1386             )
 1387         return (
 1388             False,
 1389             (
 1390                 'Link or directory exists where the shortcut "{}" '
 1391                 "should be. Did you mean to use force?".format(name)
 1392             ),
 1393             changes,
 1394         )
 1395 
 1396 
 1397 def _makedirs(
 1398     name,
 1399     user=None,
 1400     group=None,
 1401     dir_mode=None,
 1402     win_owner=None,
 1403     win_perms=None,
 1404     win_deny_perms=None,
 1405     win_inheritance=None,
 1406 ):
 1407     """
 1408     Helper function for creating directories when the ``makedirs`` option is set
 1409     to ``True``. Handles Unix and Windows based systems
 1410 
 1411     .. versionadded:: 2017.7.8
 1412 
 1413     Args:
 1414         name (str): The directory path to create
 1415         user (str): The linux user to own the directory
 1416         group (str): The linux group to own the directory
 1417         dir_mode (str): The linux mode to apply to the directory
 1418         win_owner (str): The Windows user to own the directory
 1419         win_perms (dict): A dictionary of grant permissions for Windows
 1420         win_deny_perms (dict): A dictionary of deny permissions for Windows
 1421         win_inheritance (bool): True to inherit permissions on Windows
 1422 
 1423     Returns:
 1424         bool: True if successful, otherwise False on Windows
 1425         str: Error messages on failure on Linux
 1426         None: On successful creation on Linux
 1427 
 1428     Raises:
 1429         CommandExecutionError: If the drive is not mounted on Windows
 1430     """
 1431     if salt.utils.platform.is_windows():
 1432         # Make sure the drive is mapped before trying to create the
 1433         # path in windows
 1434         drive, path = os.path.splitdrive(name)
 1435         if not os.path.isdir(drive):
 1436             raise CommandExecutionError(drive)
 1437         win_owner = win_owner if win_owner else user
 1438         return __salt__["file.makedirs"](
 1439             path=name,
 1440             owner=win_owner,
 1441             grant_perms=win_perms,
 1442             deny_perms=win_deny_perms,
 1443             inheritance=win_inheritance,
 1444         )
 1445     else:
 1446         return __salt__["file.makedirs"](
 1447             path=name, user=user, group=group, mode=dir_mode
 1448         )
 1449 
 1450 
 1451 def hardlink(
 1452     name,
 1453     target,
 1454     force=False,
 1455     makedirs=False,
 1456     user=None,
 1457     group=None,
 1458     dir_mode=None,
 1459     **kwargs
 1460 ):
 1461     """
 1462     Create a hard link
 1463     If the file already exists and is a hard link pointing to any location other
 1464     than the specified target, the hard link will be replaced. If the hard link
 1465     is a regular file or directory then the state will return False. If the
 1466     regular file is desired to be replaced with a hard link pass force: True
 1467 
 1468     name
 1469         The location of the hard link to create
 1470     target
 1471         The location that the hard link points to
 1472     force
 1473         If the name of the hard link exists and force is set to False, the
 1474         state will fail. If force is set to True, the file or directory in the
 1475         way of the hard link file will be deleted to make room for the hard
 1476         link, unless backupname is set, when it will be renamed
 1477     makedirs
 1478         If the location of the hard link does not already have a parent directory
 1479         then the state will fail, setting makedirs to True will allow Salt to
 1480         create the parent directory
 1481     user
 1482         The user to own any directories made if makedirs is set to true. This
 1483         defaults to the user salt is running as on the minion
 1484     group
 1485         The group ownership set on any directories made if makedirs is set to
 1486         true. This defaults to the group salt is running as on the minion. On
 1487         Windows, this is ignored
 1488     dir_mode
 1489         If directories are to be created, passing this option specifies the
 1490         permissions for those directories.
 1491     """
 1492     name = os.path.expanduser(name)
 1493 
 1494     # Make sure that leading zeros stripped by YAML loader are added back
 1495     dir_mode = salt.utils.files.normalize_mode(dir_mode)
 1496 
 1497     user = _test_owner(kwargs, user=user)
 1498     ret = {"name": name, "changes": {}, "result": True, "comment": ""}
 1499     if not name:
 1500         return _error(ret, "Must provide name to file.hardlink")
 1501 
 1502     if user is None:
 1503         user = __opts__["user"]
 1504 
 1505     if salt.utils.platform.is_windows():
 1506         if group is not None:
 1507             log.warning(
 1508                 "The group argument for {} has been ignored as this "
 1509                 "is a Windows system.".format(name)
 1510             )
 1511         group = user
 1512 
 1513     if group is None:
 1514         group = __salt__["file.gid_to_group"](__salt__["user.info"](user).get("gid", 0))
 1515 
 1516     preflight_errors = []
 1517     uid = __salt__["file.user_to_uid"](user)
 1518     gid = __salt__["file.group_to_gid"](group)
 1519 
 1520     if uid == "":
 1521         preflight_errors.append("User {} does not exist".format(user))
 1522 
 1523     if gid == "":
 1524         preflight_errors.append("Group {} does not exist".format(group))
 1525 
 1526     if not os.path.isabs(name):
 1527         preflight_errors.append(
 1528             "Specified file {} is not an absolute path".format(name)
 1529         )
 1530 
 1531     if not os.path.isabs(target):
 1532         preflight_errors.append(
 1533             "Specified target {} is not an absolute path".format(target)
 1534         )
 1535 
 1536     if preflight_errors:
 1537         msg = ". ".join(preflight_errors)
 1538         if len(preflight_errors) > 1:
 1539             msg += "."
 1540         return _error(ret, msg)
 1541 
 1542     if __opts__["test"]:
 1543         tresult, tcomment, tchanges = _hardlink_check(name, target, force)
 1544         ret["result"] = tresult
 1545         ret["comment"] = tcomment
 1546         ret["changes"] = tchanges
 1547         return ret
 1548 
 1549     # We use zip_longest here because there's a number of issues in pylint's
 1550     # tracker that complains about not linking the zip builtin.
 1551     for direction, item in zip_longest(["to", "from"], [name, target]):
 1552         if os.path.isdir(item):
 1553             msg = "Unable to hard link {} directory {}".format(direction, item)
 1554             return _error(ret, msg)
 1555 
 1556     if not os.path.exists(target):
 1557         msg = "Target {} for hard link does not exist".format(target)
 1558         return _error(ret, msg)
 1559 
 1560     # Check that the directory to write the hard link to exists
 1561     if not os.path.isdir(os.path.dirname(name)):
 1562         if makedirs:
 1563             __salt__["file.makedirs"](name, user=user, group=group, mode=dir_mode)
 1564 
 1565         else:
 1566             return _error(
 1567                 ret,
 1568                 "Directory {} for hard link is not present".format(
 1569                     os.path.dirname(name)
 1570                 ),
 1571             )
 1572 
 1573     # If file is not a hard link and we're actually overwriting it, then verify
 1574     # that this was forced.
 1575     if os.path.isfile(name) and not __salt__["file.is_hardlink"](name):
 1576 
 1577         # Remove whatever is in the way. This should then hit the else case
 1578         # of the file.is_hardlink check below
 1579         if force:
 1580             os.remove(name)
 1581             ret["changes"]["forced"] = "File for hard link was forcibly replaced"
 1582 
 1583         # Otherwise throw an error
 1584         else:
 1585             return _error(
 1586                 ret, ("File exists where the hard link {} should be".format(name))
 1587             )
 1588 
 1589     # If the file is a hard link, then we can simply rewrite its target since
 1590     # nothing is really being lost here.
 1591     if __salt__["file.is_hardlink"](name):
 1592 
 1593         # If the inodes point to the same thing, then there's nothing to do
 1594         # except for let the user know that this has already happened.
 1595         if _hardlink_same(name, target):
 1596             ret["result"] = True
 1597             ret["comment"] = (
 1598                 "Target of hard link {} is already pointing "
 1599                 "to {}".format(name, target)
 1600             )
 1601             return ret
 1602 
 1603         # First remove the old hard link since a reference to it already exists
 1604         os.remove(name)
 1605 
 1606         # Now we can remake it
 1607         try:
 1608             __salt__["file.link"](target, name)
 1609 
 1610         # Or not...
 1611         except CommandExecutionError as E:
 1612             ret["result"] = False
 1613             ret["comment"] = "Unable to set target of hard link {} -> " "{}: {}".format(
 1614                 name, target, E
 1615             )
 1616             return ret
 1617 
 1618         # Good to go
 1619         ret["result"] = True
 1620         ret["comment"] = "Set target of hard link {} -> {}".format(name, target)
 1621         ret["changes"]["new"] = name
 1622 
 1623     # The link is not present, so simply make it
 1624     elif not os.path.exists(name):
 1625         try:
 1626             __salt__["file.link"](target, name)
 1627 
 1628         # Or not...
 1629         except CommandExecutionError as E:
 1630             ret["result"] = False
 1631             ret["comment"] = "Unable to create new hard link {} -> " "{}: {}".format(
 1632                 name, target, E
 1633             )
 1634             return ret
 1635 
 1636         # Made a new hard link, things are ok
 1637         ret["result"] = True
 1638         ret["comment"] = "Created new hard link {} -> {}".format(name, target)
 1639         ret["changes"]["new"] = name
 1640 
 1641     return ret
 1642 
 1643 
 1644 def symlink(
 1645     name,
 1646     target,
 1647     force=False,
 1648     backupname=None,
 1649     makedirs=False,
 1650     user=None,
 1651     group=None,
 1652     mode=None,
 1653     win_owner=None,
 1654     win_perms=None,
 1655     win_deny_perms=None,
 1656     win_inheritance=None,
 1657     **kwargs
 1658 ):
 1659     """
 1660     Create a symbolic link (symlink, soft link)
 1661 
 1662     If the file already exists and is a symlink pointing to any location other
 1663     than the specified target, the symlink will be replaced. If an entry with
 1664     the same name exists then the state will return False. If the existing
 1665     entry is desired to be replaced with a symlink pass force: True, if it is
 1666     to be renamed, pass a backupname.
 1667 
 1668     name
 1669         The location of the symlink to create
 1670 
 1671     target
 1672         The location that the symlink points to
 1673 
 1674     force
 1675         If the name of the symlink exists and is not a symlink and
 1676         force is set to False, the state will fail. If force is set to
 1677         True, the existing entry in the way of the symlink file
 1678         will be deleted to make room for the symlink, unless
 1679         backupname is set, when it will be renamed
 1680 
 1681         .. versionchanged:: Neon
 1682             Force will now remove all types of existing file system entries,
 1683             not just files, directories and symlinks.
 1684 
 1685     backupname
 1686         If the name of the symlink exists and is not a symlink, it will be
 1687         renamed to the backupname. If the backupname already
 1688         exists and force is False, the state will fail. Otherwise, the
 1689         backupname will be removed first.
 1690         An absolute path OR a basename file/directory name must be provided.
 1691         The latter will be placed relative to the symlink destination's parent
 1692         directory.
 1693 
 1694     makedirs
 1695         If the location of the symlink does not already have a parent directory
 1696         then the state will fail, setting makedirs to True will allow Salt to
 1697         create the parent directory
 1698 
 1699     user
 1700         The user to own the file, this defaults to the user salt is running as
 1701         on the minion
 1702 
 1703     group
 1704         The group ownership set for the file, this defaults to the group salt
 1705         is running as on the minion. On Windows, this is ignored
 1706 
 1707     mode
 1708         The permissions to set on this file, aka 644, 0775, 4664. Not supported
 1709         on Windows.
 1710 
 1711         The default mode for new files and directories corresponds umask of salt
 1712         process. For existing files and directories it's not enforced.
 1713 
 1714     win_owner : None
 1715         The owner of the symlink and directories if ``makedirs`` is True. If
 1716         this is not passed, ``user`` will be used. If ``user`` is not passed,
 1717         the account under which Salt is running will be used.
 1718 
 1719         .. versionadded:: 2017.7.7
 1720 
 1721     win_perms : None
 1722         A dictionary containing permissions to grant
 1723 
 1724         .. versionadded:: 2017.7.7
 1725 
 1726     win_deny_perms : None
 1727         A dictionary containing permissions to deny
 1728 
 1729         .. versionadded:: 2017.7.7
 1730 
 1731     win_inheritance : None
 1732         True to inherit permissions from parent, otherwise False
 1733 
 1734         .. versionadded:: 2017.7.7
 1735     """
 1736     name = os.path.expanduser(name)
 1737 
 1738     # Make sure that leading zeros stripped by YAML loader are added back
 1739     mode = salt.utils.files.normalize_mode(mode)
 1740 
 1741     user = _test_owner(kwargs, user=user)
 1742     ret = {"name": name, "changes": {}, "result": True, "comment": ""}
 1743     if not name:
 1744         return _error(ret, "Must provide name to file.symlink")
 1745 
 1746     if user is None:
 1747         user = __opts__["user"]
 1748 
 1749     if salt.utils.platform.is_windows():
 1750 
 1751         # Make sure the user exists in Windows
 1752         # Salt default is 'root'
 1753         if not __salt__["user.info"](user):
 1754             # User not found, use the account salt is running under
 1755             # If username not found, use System
 1756             user = __salt__["user.current"]()
 1757             if not user:
 1758                 user = "SYSTEM"
 1759 
 1760         # If win_owner is not passed, use user
 1761         if win_owner is None:
 1762             win_owner = user if user else None
 1763 
 1764         # Group isn't relevant to Windows, use win_perms/win_deny_perms
 1765         if group is not None:
 1766             log.warning(
 1767                 "The group argument for {} has been ignored as this "
 1768                 "is a Windows system. Please use the `win_*` parameters to set "
 1769                 "permissions in Windows.".format(name)
 1770             )
 1771         group = user
 1772 
 1773     if group is None:
 1774         group = __salt__["file.gid_to_group"](__salt__["user.info"](user).get("gid", 0))
 1775 
 1776     preflight_errors = []
 1777     if salt.utils.platform.is_windows():
 1778         # Make sure the passed owner exists
 1779         try:
 1780             salt.utils.win_functions.get_sid_from_name(win_owner)
 1781         except CommandExecutionError as exc:
 1782             preflight_errors.append("User {} does not exist".format(win_owner))
 1783 
 1784         # Make sure users passed in win_perms exist
 1785         if win_perms:
 1786             for name_check in win_perms:
 1787                 try:
 1788                     salt.utils.win_functions.get_sid_from_name(name_check)
 1789                 except CommandExecutionError as exc:
 1790                     preflight_errors.append("User {} does not exist".format(name_check))
 1791 
 1792         # Make sure users passed in win_deny_perms exist
 1793         if win_deny_perms:
 1794             for name_check in win_deny_perms:
 1795                 try:
 1796                     salt.utils.win_functions.get_sid_from_name(name_check)
 1797                 except CommandExecutionError as exc:
 1798                     preflight_errors.append("User {} does not exist".format(name_check))
 1799     else:
 1800         uid = __salt__["file.user_to_uid"](user)
 1801         gid = __salt__["file.group_to_gid"](group)
 1802 
 1803         if uid == "":
 1804             preflight_errors.append("User {} does not exist".format(user))
 1805 
 1806         if gid == "":
 1807             preflight_errors.append("Group {} does not exist".format(group))
 1808 
 1809     if not os.path.isabs(name):
 1810         preflight_errors.append(
 1811             "Specified file {} is not an absolute path".format(name)
 1812         )
 1813 
 1814     if preflight_errors:
 1815         msg = ". ".join(preflight_errors)
 1816         if len(preflight_errors) > 1:
 1817             msg += "."
 1818         return _error(ret, msg)
 1819 
 1820     tresult, tcomment, tchanges = _symlink_check(
 1821         name, target, force, user, group, win_owner
 1822     )
 1823 
 1824     if not os.path.isdir(os.path.dirname(name)):
 1825         if makedirs:
 1826             if __opts__["test"]:
 1827                 tcomment += "\n{} will be created".format(os.path.dirname(name))
 1828             else:
 1829                 try:
 1830                     _makedirs(
 1831                         name=name,
 1832                         user=user,
 1833                         group=group,
 1834                         dir_mode=mode,
 1835                         win_owner=win_owner,
 1836                         win_perms=win_perms,
 1837                         win_deny_perms=win_deny_perms,
 1838                         win_inheritance=win_inheritance,
 1839                     )
 1840                 except CommandExecutionError as exc:
 1841                     return _error(ret, "Drive {} is not mapped".format(exc.message))
 1842         else:
 1843             if __opts__["test"]:
 1844                 tcomment += "\nDirectory {} for symlink is not present" "".format(
 1845                     os.path.dirname(name)
 1846                 )
 1847             else:
 1848                 return _error(
 1849                     ret,
 1850                     "Directory {} for symlink is not present".format(
 1851                         os.path.dirname(name)
 1852                     ),
 1853                 )
 1854 
 1855     if __opts__["test"]:
 1856         ret["result"] = tresult
 1857         ret["comment"] = tcomment
 1858         ret["changes"] = tchanges
 1859         return ret
 1860 
 1861     if __salt__["file.is_link"](name):
 1862         # The link exists, verify that it matches the target
 1863         if os.path.normpath(__salt__["file.readlink"](name)) != os.path.normpath(
 1864             target
 1865         ):
 1866             # The target is wrong, delete the link
 1867             os.remove(name)
 1868         else:
 1869             if _check_symlink_ownership(name, user, group, win_owner):
 1870                 # The link looks good!
 1871                 if salt.utils.platform.is_windows():
 1872                     ret["comment"] = "Symlink {} is present and owned by {}" "".format(
 1873                         name, win_owner
 1874                     )
 1875                 else:
 1876                     ret["comment"] = (
 1877                         "Symlink {} is present and owned by "
 1878                         "{}:{}".format(name, user, group)
 1879                     )
 1880             else:
 1881                 if _set_symlink_ownership(name, user, group, win_owner):
 1882                     if salt.utils.platform.is_windows():
 1883                         ret["comment"] = "Set ownership of symlink {} to " "{}".format(
 1884                             name, win_owner
 1885                         )
 1886                         ret["changes"]["ownership"] = win_owner
 1887                     else:
 1888                         ret["comment"] = (
 1889                             "Set ownership of symlink {} to "
 1890                             "{}:{}".format(name, user, group)
 1891                         )
 1892                         ret["changes"]["ownership"] = "{}:{}".format(user, group)
 1893                 else:
 1894                     ret["result"] = False
 1895                     if salt.utils.platform.is_windows():
 1896                         ret["comment"] += (
 1897                             "Failed to set ownership of symlink "
 1898                             "{} to {}".format(name, win_owner)
 1899                         )
 1900                     else:
 1901                         ret["comment"] += (
 1902                             "Failed to set ownership of symlink {} to "
 1903                             "{}:{}".format(name, user, group)
 1904                         )
 1905             return ret
 1906 
 1907     elif os.path.exists(name):
 1908         # It is not a link, but a file, dir, socket, FIFO etc.
 1909         if backupname is not None:
 1910             if not os.path.isabs(backupname):
 1911                 if backupname == os.path.basename(backupname):
 1912                     backupname = os.path.join(
 1913                         os.path.dirname(os.path.normpath(name)), backupname
 1914                     )
 1915                 else:
 1916                     return _error(
 1917                         ret,
 1918                         (
 1919                             (
 1920                                 "Backupname must be an absolute path "
 1921                                 "or a file name: {}"
 1922                             ).format(backupname)
 1923                         ),
 1924                     )
 1925             # Make a backup first
 1926             if os.path.lexists(backupname):
 1927                 if not force:
 1928                     return _error(
 1929                         ret,
 1930                         (
 1931                             (
 1932                                 "Symlink & backup dest exists and Force not set."
 1933                                 " {} -> {} - backup: {}"
 1934                             ).format(name, target, backupname)
 1935                         ),
 1936                     )
 1937                 else:
 1938                     __salt__["file.remove"](backupname)
 1939             try:
 1940                 __salt__["file.move"](name, backupname)
 1941             except Exception as exc:  # pylint: disable=broad-except
 1942                 ret["changes"] = {}
 1943                 log.debug(
 1944                     "Encountered error renaming %s to %s",
 1945                     name,
 1946                     backupname,
 1947                     exc_info=True,
 1948                 )
 1949                 return _error(
 1950                     ret,
 1951                     (
 1952                         "Unable to rename {} to backup {} -> "
 1953                         ": {}".format(name, backupname, exc)
 1954                     ),
 1955                 )
 1956         elif force:
 1957             # Remove whatever is in the way
 1958             if __salt__["file.is_link"](name):
 1959                 __salt__["file.remove"](name)
 1960                 ret["changes"]["forced"] = "Symlink was forcibly replaced"
 1961             else:
 1962                 __salt__["file.remove"](name)
 1963         else:
 1964             # Otherwise throw an error
 1965             fs_entry_type = (
 1966                 "File"
 1967                 if os.path.isfile(name)
 1968                 else "Directory"
 1969                 if os.path.isdir(name)
 1970                 else "File system entry"
 1971             )
 1972             return _error(
 1973                 ret,
 1974                 (
 1975                     "{} exists where the symlink {} should be".format(
 1976                         fs_entry_type, name
 1977                     )
 1978                 ),
 1979             )
 1980 
 1981     if not os.path.exists(name):
 1982         # The link is not present, make it
 1983         try:
 1984             __salt__["file.symlink"](target, name)
 1985         except OSError as exc:
 1986             ret["result"] = False
 1987             ret["comment"] = "Unable to create new symlink {} -> " "{}: {}".format(
 1988                 name, target, exc
 1989             )
 1990             return ret
 1991         else:
 1992             ret["comment"] = "Created new symlink {} -> " "{}".format(name, target)
 1993             ret["changes"]["new"] = name
 1994 
 1995         if not _check_symlink_ownership(name, user, group, win_owner):
 1996             if not _set_symlink_ownership(name, user, group, win_owner):
 1997                 ret["result"] = False
 1998                 ret[
 1999                     "comment"
 2000                 ] += ", but was unable to set ownership to " "{}:{}".format(user, group)
 2001     return ret
 2002 
 2003 
 2004 def absent(name, **kwargs):
 2005     """
 2006     Make sure that the named file or directory is absent. If it exists, it will
 2007     be deleted. This will work to reverse any of the functions in the file
 2008     state module. If a directory is supplied, it will be recursively deleted.
 2009 
 2010     name
 2011         The path which should be deleted
 2012     """
 2013     name = os.path.expanduser(name)
 2014 
 2015     ret = {"name": name, "changes": {}, "result": True, "comment": ""}
 2016     if not name:
 2017         return _error(ret, "Must provide name to file.absent")
 2018     if not os.path.isabs(name):
 2019         return _error(ret, "Specified file {} is not an absolute path".format(name))
 2020     if name == "/":
 2021         return _error(ret, 'Refusing to make "/" absent')
 2022     if os.path.isfile(name) or os.path.islink(name):
 2023         if __opts__["test"]:
 2024             ret["result"] = None
 2025             ret["changes"]["removed"] = name
 2026             ret["comment"] = "File {} is set for removal".format(name)
 2027             return ret
 2028         try:
 2029             if salt.utils.platform.is_windows():
 2030                 __salt__["file.remove"](name, force=True)
 2031             else:
 2032                 __salt__["file.remove"](name)
 2033             ret["comment"] = "Removed file {}".format(name)
 2034             ret["changes"]["removed"] = name
 2035             return ret
 2036         except CommandExecutionError as exc:
 2037             return _error(ret, "{}".format(exc))
 2038 
 2039     elif os.path.isdir(name):
 2040         if __opts__["test"]:
 2041             ret["result"] = None
 2042             ret["changes"]["removed"] = name
 2043             ret["comment"] = "Directory {} is set for removal".format(name)
 2044             return ret
 2045         try:
 2046             if salt.utils.platform.is_windows():
 2047                 __salt__["file.remove"](name, force=True)
 2048             else:
 2049                 __salt__["file.remove"](name)
 2050             ret["comment"] = "Removed directory {}".format(name)
 2051             ret["changes"]["removed"] = name
 2052             return ret
 2053         except OSError:
 2054             return _error(ret, "Failed to remove directory {}".format(name))
 2055 
 2056     ret["comment"] = "File {} is not present".format(name)
 2057     return ret
 2058 
 2059 
 2060 def tidied(name, age=0, matches=None, rmdirs=False, size=0, **kwargs):
 2061     """
 2062     Remove unwanted files based on specific criteria. Multiple criteria
 2063     are OR’d together, so a file that is too large but is not old enough
 2064     will still get tidied.
 2065 
 2066     If neither age nor size is given all files which match a pattern in
 2067     matches will be removed.
 2068 
 2069     name
 2070         The directory tree that should be tidied
 2071 
 2072     age
 2073         Maximum age in days after which files are considered for removal
 2074 
 2075     matches
 2076         List of regular expressions to restrict what gets removed.  Default: ['.*']
 2077 
 2078     rmdirs
 2079         Whether or not it's allowed to remove directories
 2080 
 2081     size
 2082         Maximum allowed file size. Files greater or equal to this size are
 2083         removed. Doesn't apply to directories or symbolic links
 2084 
 2085     .. code-block:: yaml
 2086 
 2087         cleanup:
 2088           file.tidied:
 2089             - name: /tmp/salt_test
 2090             - rmdirs: True
 2091             - matches:
 2092               - foo
 2093               - b.*r
 2094     """
 2095     name = os.path.expanduser(name)
 2096 
 2097     ret = {"name": name, "changes": {}, "result": True, "comment": ""}
 2098 
 2099     # Check preconditions
 2100     if not os.path.isabs(name):
 2101         return _error(ret, "Specified file {} is not an absolute path".format(name))
 2102     if not os.path.isdir(name):
 2103         return _error(ret, "{} does not exist or is not a directory.".format(name))
 2104 
 2105     # Define some variables
 2106     todelete = []
 2107     today = date.today()
 2108 
 2109     # Compile regular expressions
 2110     if matches is None:
 2111         matches = [".*"]
 2112     progs = []
 2113     for regex in matches:
 2114         progs.append(re.compile(regex))
 2115 
 2116     # Helper to match a given name against one or more pre-compiled regular
 2117     # expressions
 2118     def _matches(name):
 2119         for prog in progs:
 2120             if prog.match(name):
 2121                 return True
 2122         return False
 2123 
 2124     # Iterate over given directory tree, depth-first
 2125     for root, dirs, files in os.walk(top=name, topdown=False):
 2126         # Check criteria for the found files and directories
 2127         for elem in files + dirs:
 2128             myage = 0
 2129             mysize = 0
 2130             deleteme = True
 2131             path = os.path.join(root, elem)
 2132             if os.path.islink(path):
 2133                 # Get age of symlink (not symlinked file)
 2134                 myage = abs(today - date.fromtimestamp(os.lstat(path).st_atime))
 2135             elif elem in dirs:
 2136                 # Get age of directory, check if directories should be deleted at all
 2137                 myage = abs(today - date.fromtimestamp(os.path.getatime(path)))
 2138                 deleteme = rmdirs
 2139             else:
 2140                 # Get age and size of regular file
 2141                 myage = abs(today - date.fromtimestamp(os.path.getatime(path)))
 2142                 mysize = os.path.getsize(path)
 2143             # Verify against given criteria, collect all elements that should be removed
 2144             if (
 2145                 (mysize >= size or myage.days >= age)
 2146                 and _matches(name=elem)
 2147                 and deleteme
 2148             ):
 2149                 todelete.append(path)
 2150 
 2151     # Now delete the stuff
 2152     if todelete:
 2153         if __opts__["test"]:
 2154             ret["result"] = None
 2155             ret["comment"] = "{} is set for tidy".format(name)
 2156             ret["changes"] = {"removed": todelete}
 2157             return ret
 2158         ret["changes"]["removed"] = []
 2159         # Iterate over collected items
 2160         try:
 2161             for path in todelete:
 2162                 if salt.utils.platform.is_windows():
 2163                     __salt__["file.remove"](path, force=True)
 2164                 else:
 2165                     __salt__["file.remove"](path)
 2166                 # Remember what we've removed, will appear in the summary
 2167                 ret["changes"]["removed"].append(path)
 2168         except CommandExecutionError as exc:
 2169             return _error(ret, "{}".format(exc))
 2170         # Set comment for the summary
 2171         ret["comment"] = "Removed {} files or directories from directory {}".format(
 2172             len(todelete), name
 2173         )
 2174     else:
 2175         # Set comment in case there was nothing to remove
 2176         ret["comment"] = "Nothing to remove from directory {}".format(name)
 2177     return ret
 2178 
 2179 
 2180 def exists(name, **kwargs):
 2181     """
 2182     Verify that the named file or directory is present or exists.
 2183     Ensures pre-requisites outside of Salt's purview
 2184     (e.g., keytabs, private keys, etc.) have been previously satisfied before
 2185     deployment.
 2186 
 2187     This function does not create the file if it doesn't exist, it will return
 2188     an error.
 2189 
 2190     name
 2191         Absolute path which must exist
 2192     """
 2193     name = os.path.expanduser(name)
 2194 
 2195     ret = {"name": name, "changes": {}, "result": True, "comment": ""}
 2196     if not name:
 2197         return _error(ret, "Must provide name to file.exists")
 2198     if not os.path.exists(name):
 2199         return _error(ret, "Specified path {} does not exist".format(name))
 2200 
 2201     ret["comment"] = "Path {} exists".format(name)
 2202     return ret
 2203 
 2204 
 2205 def missing(name, **kwargs):
 2206     """
 2207     Verify that the named file or directory is missing, this returns True only
 2208     if the named file is missing but does not remove the file if it is present.
 2209 
 2210     name
 2211         Absolute path which must NOT exist
 2212     """
 2213     name = os.path.expanduser(name)
 2214 
 2215     ret = {"name": name, "changes": {}, "result": True, "comment": ""}
 2216     if not name:
 2217         return _error(ret, "Must provide name to file.missing")
 2218     if os.path.exists(name):
 2219         return _error(ret, "Specified path {} exists".format(name))
 2220 
 2221     ret["comment"] = "Path {} is missing".format(name)
 2222     return ret
 2223 
 2224 
 2225 def managed(
 2226     name,
 2227     source=None,
 2228     source_hash="",
 2229     source_hash_name=None,
 2230     keep_source=True,
 2231     user=None,
 2232     group=None,
 2233     mode=None,
 2234     attrs=None,
 2235     template=None,
 2236     makedirs=False,
 2237     dir_mode=None,
 2238     context=None,
 2239     replace=True,
 2240     defaults=None,
 2241     backup="",
 2242     show_changes=True,
 2243     create=True,
 2244     contents=None,
 2245     tmp_dir="",
 2246     tmp_ext="",
 2247     contents_pillar=None,
 2248     contents_grains=None,
 2249     contents_newline=True,
 2250     contents_delimiter=":",
 2251     encoding=None,
 2252     encoding_errors="strict",
 2253     allow_empty=True,
 2254     follow_symlinks=True,
 2255     check_cmd=None,
 2256     skip_verify=False,
 2257     selinux=None,
 2258     win_owner=None,
 2259     win_perms=None,
 2260     win_deny_perms=None,
 2261     win_inheritance=True,
 2262     win_perms_reset=False,
 2263     verify_ssl=True,
 2264     **kwargs
 2265 ):
 2266     r"""
 2267     Manage a given file, this function allows for a file to be downloaded from
 2268     the salt master and potentially run through a templating system.
 2269 
 2270     name
 2271         The location of the file to manage, as an absolute path.
 2272 
 2273     source
 2274         The source file to download to the minion, this source file can be
 2275         hosted on either the salt master server (``salt://``), the salt minion
 2276         local file system (``/``), or on an HTTP or FTP server (``http(s)://``,
 2277         ``ftp://``).
 2278 
 2279         Both HTTPS and HTTP are supported as well as downloading directly
 2280         from Amazon S3 compatible URLs with both pre-configured and automatic
 2281         IAM credentials. (see s3.get state documentation)
 2282         File retrieval from Openstack Swift object storage is supported via
 2283         swift://container/object_path URLs, see swift.get documentation.
 2284         For files hosted on the salt file server, if the file is located on
 2285         the master in the directory named spam, and is called eggs, the source
 2286         string is salt://spam/eggs. If source is left blank or None
 2287         (use ~ in YAML), the file will be created as an empty file and
 2288         the content will not be managed. This is also the case when a file
 2289         already exists and the source is undefined; the contents of the file
 2290         will not be changed or managed. If source is left blank or None, please
 2291         also set replaced to False to make your intention explicit.
 2292 
 2293 
 2294         If the file is hosted on a HTTP or FTP server then the source_hash
 2295         argument is also required.
 2296 
 2297         A list of sources can also be passed in to provide a default source and
 2298         a set of fallbacks. The first source in the list that is found to exist
 2299         will be used and subsequent entries in the list will be ignored. Source
 2300         list functionality only supports local files and remote files hosted on
 2301         the salt master server or retrievable via HTTP, HTTPS, or FTP.
 2302 
 2303         .. code-block:: yaml
 2304 
 2305             file_override_example:
 2306               file.managed:
 2307                 - source:
 2308                   - salt://file_that_does_not_exist
 2309                   - salt://file_that_exists
 2310 
 2311     source_hash
 2312         This can be one of the following:
 2313             1. a source hash string
 2314             2. the URI of a file that contains source hash strings
 2315 
 2316         The function accepts the first encountered long unbroken alphanumeric
 2317         string of correct length as a valid hash, in order from most secure to
 2318         least secure:
 2319 
 2320         .. code-block:: text
 2321 
 2322             Type    Length
 2323             ======  ======
 2324             sha512     128
 2325             sha384      96
 2326             sha256      64
 2327             sha224      56
 2328             sha1        40
 2329             md5         32
 2330 
 2331         **Using a Source Hash File**
 2332             The file can contain several checksums for several files. Each line
 2333             must contain both the file name and the hash.  If no file name is
 2334             matched, the first hash encountered will be used, otherwise the most
 2335             secure hash with the correct source file name will be used.
 2336 
 2337             When using a source hash file the source_hash argument needs to be a
 2338             url, the standard download urls are supported, ftp, http, salt etc:
 2339 
 2340             Example:
 2341 
 2342             .. code-block:: yaml
 2343 
 2344                 tomdroid-src-0.7.3.tar.gz:
 2345                   file.managed:
 2346                     - name: /tmp/tomdroid-src-0.7.3.tar.gz
 2347                     - source: https://launchpad.net/tomdroid/beta/0.7.3/+download/tomdroid-src-0.7.3.tar.gz
 2348                     - source_hash: https://launchpad.net/tomdroid/beta/0.7.3/+download/tomdroid-src-0.7.3.hash
 2349 
 2350             The following lines are all supported formats:
 2351 
 2352             .. code-block:: text
 2353 
 2354                 /etc/rc.conf ef6e82e4006dee563d98ada2a2a80a27
 2355                 sha254c8525aee419eb649f0233be91c151178b30f0dff8ebbdcc8de71b1d5c8bcc06a  /etc/resolv.conf
 2356                 ead48423703509d37c4a90e6a0d53e143b6fc268
 2357 
 2358             Debian file type ``*.dsc`` files are also supported.
 2359 
 2360         **Inserting the Source Hash in the SLS Data**
 2361 
 2362         The source_hash can be specified as a simple checksum, like so:
 2363 
 2364         .. code-block:: yaml
 2365 
 2366             tomdroid-src-0.7.3.tar.gz:
 2367               file.managed:
 2368                 - name: /tmp/tomdroid-src-0.7.3.tar.gz
 2369                 - source: https://launchpad.net/tomdroid/beta/0.7.3/+download/tomdroid-src-0.7.3.tar.gz
 2370                 - source_hash: 79eef25f9b0b2c642c62b7f737d4f53f
 2371 
 2372         .. note::
 2373             Releases prior to 2016.11.0 must also include the hash type, like
 2374             in the below example:
 2375 
 2376             .. code-block:: yaml
 2377 
 2378                 tomdroid-src-0.7.3.tar.gz:
 2379                   file.managed:
 2380                     - name: /tmp/tomdroid-src-0.7.3.tar.gz
 2381                     - source: https://launchpad.net/tomdroid/beta/0.7.3/+download/tomdroid-src-0.7.3.tar.gz
 2382                     - source_hash: md5=79eef25f9b0b2c642c62b7f737d4f53f
 2383 
 2384         Known issues:
 2385             If the remote server URL has the hash file as an apparent
 2386             sub-directory of the source file, the module will discover that it
 2387             has already cached a directory where a file should be cached. For
 2388             example:
 2389 
 2390             .. code-block:: yaml
 2391 
 2392                 tomdroid-src-0.7.3.tar.gz:
 2393                   file.managed:
 2394                     - name: /tmp/tomdroid-src-0.7.3.tar.gz
 2395                     - source: https://launchpad.net/tomdroid/beta/0.7.3/+download/tomdroid-src-0.7.3.tar.gz
 2396                     - source_hash: https://launchpad.net/tomdroid/beta/0.7.3/+download/tomdroid-src-0.7.3.tar.gz/+md5
 2397 
 2398     source_hash_name
 2399         When ``source_hash`` refers to a hash file, Salt will try to find the
 2400         correct hash by matching the filename/URI associated with that hash. By
 2401         default, Salt will look for the filename being managed. When managing a
 2402         file at path ``/tmp/foo.txt``, then the following line in a hash file
 2403         would match:
 2404 
 2405         .. code-block:: text
 2406 
 2407             acbd18db4cc2f85cedef654fccc4a4d8    foo.txt
 2408 
 2409         However, sometimes a hash file will include multiple similar paths:
 2410 
 2411         .. code-block:: text
 2412 
 2413             37b51d194a7513e45b56f6524f2d51f2    ./dir1/foo.txt
 2414             acbd18db4cc2f85cedef654fccc4a4d8    ./dir2/foo.txt
 2415             73feffa4b7f6bb68e44cf984c85f6e88    ./dir3/foo.txt
 2416 
 2417         In cases like this, Salt may match the incorrect hash. This argument
 2418         can be used to tell Salt which filename to match, to ensure that the
 2419         correct hash is identified. For example:
 2420 
 2421         .. code-block:: yaml
 2422 
 2423             /tmp/foo.txt:
 2424               file.managed:
 2425                 - source: https://mydomain.tld/dir2/foo.txt
 2426                 - source_hash: https://mydomain.tld/hashes
 2427                 - source_hash_name: ./dir2/foo.txt
 2428 
 2429         .. note::
 2430             This argument must contain the full filename entry from the
 2431             checksum file, as this argument is meant to disambiguate matches
 2432             for multiple files that have the same basename. So, in the
 2433             example above, simply using ``foo.txt`` would not match.
 2434 
 2435         .. versionadded:: 2016.3.5
 2436 
 2437     keep_source : True
 2438         Set to ``False`` to discard the cached copy of the source file once the
 2439         state completes. This can be useful for larger files to keep them from
 2440         taking up space in minion cache. However, keep in mind that discarding
 2441         the source file will result in the state needing to re-download the
 2442         source file if the state is run again.
 2443 
 2444         .. versionadded:: 2017.7.3
 2445 
 2446     user
 2447         The user to own the file, this defaults to the user salt is running as
 2448         on the minion
 2449 
 2450     group
 2451         The group ownership set for the file, this defaults to the group salt
 2452         is running as on the minion. On Windows, this is ignored
 2453 
 2454     mode
 2455         The permissions to set on this file, e.g. ``644``, ``0775``, or
 2456         ``4664``.
 2457 
 2458         The default mode for new files and directories corresponds to the
 2459         umask of the salt process. The mode of existing files and directories
 2460         will only be changed if ``mode`` is specified.
 2461 
 2462         .. note::
 2463             This option is **not** supported on Windows.
 2464 
 2465         .. versionchanged:: 2016.11.0
 2466             This option can be set to ``keep``, and Salt will keep the mode
 2467             from the Salt fileserver. This is only supported when the
 2468             ``source`` URL begins with ``salt://``, or for files local to the
 2469             minion. Because the ``source`` option cannot be used with any of
 2470             the ``contents`` options, setting the ``mode`` to ``keep`` is also
 2471             incompatible with the ``contents`` options.
 2472 
 2473         .. note:: keep does not work with salt-ssh.
 2474 
 2475             As a consequence of how the files are transferred to the minion, and
 2476             the inability to connect back to the master with salt-ssh, salt is
 2477             unable to stat the file as it exists on the fileserver and thus
 2478             cannot mirror the mode on the salt-ssh minion
 2479 
 2480     attrs
 2481         The attributes to have on this file, e.g. ``a``, ``i``. The attributes
 2482         can be any or a combination of the following characters:
 2483         ``aAcCdDeijPsStTu``.
 2484 
 2485         .. note::
 2486             This option is **not** supported on Windows.
 2487 
 2488         .. versionadded:: 2018.3.0
 2489 
 2490     template
 2491         If this setting is applied, the named templating engine will be used to
 2492         render the downloaded file. The following templates are supported:
 2493 
 2494         - :mod:`cheetah<salt.renderers.cheetah>`
 2495         - :mod:`genshi<salt.renderers.genshi>`
 2496         - :mod:`jinja<salt.renderers.jinja>`
 2497         - :mod:`mako<salt.renderers.mako>`
 2498         - :mod:`py<salt.renderers.py>`
 2499         - :mod:`wempy<salt.renderers.wempy>`
 2500 
 2501     makedirs : False
 2502         If set to ``True``, then the parent directories will be created to
 2503         facilitate the creation of the named file. If ``False``, and the parent
 2504         directory of the destination file doesn't exist, the state will fail.
 2505 
 2506     dir_mode
 2507         If directories are to be created, passing this option specifies the
 2508         permissions for those directories. If this is not set, directories
 2509         will be assigned permissions by adding the execute bit to the mode of
 2510         the files.
 2511 
 2512         The default mode for new files and directories corresponds umask of salt
 2513         process. For existing files and directories it's not enforced.
 2514 
 2515     replace : True
 2516         If set to ``False`` and the file already exists, the file will not be
 2517         modified even if changes would otherwise be made. Permissions and
 2518         ownership will still be enforced, however.
 2519 
 2520     context
 2521         Overrides default context variables passed to the template.
 2522 
 2523     defaults
 2524         Default context passed to the template.
 2525 
 2526     backup
 2527         Overrides the default backup mode for this specific file. See
 2528         :ref:`backup_mode documentation <file-state-backups>` for more details.
 2529 
 2530     show_changes
 2531         Output a unified diff of the old file and the new file. If ``False``
 2532         return a boolean if any changes were made.
 2533 
 2534     create : True
 2535         If set to ``False``, then the file will only be managed if the file
 2536         already exists on the system.
 2537 
 2538     contents
 2539         Specify the contents of the file. Cannot be used in combination with
 2540         ``source``. Ignores hashes and does not use a templating engine.
 2541 
 2542         This value can be either a single string, a multiline YAML string or a
 2543         list of strings.  If a list of strings, then the strings will be joined
 2544         together with newlines in the resulting file. For example, the below
 2545         two example states would result in identical file contents:
 2546 
 2547         .. code-block:: yaml
 2548 
 2549             /path/to/file1:
 2550               file.managed:
 2551                 - contents:
 2552                   - This is line 1
 2553                   - This is line 2
 2554 
 2555             /path/to/file2:
 2556               file.managed:
 2557                 - contents: |
 2558                     This is line 1
 2559                     This is line 2
 2560 
 2561 
 2562     contents_pillar
 2563         .. versionadded:: 0.17.0
 2564         .. versionchanged:: 2016.11.0
 2565             contents_pillar can also be a list, and the pillars will be
 2566             concatenated together to form one file.
 2567 
 2568 
 2569         Operates like ``contents``, but draws from a value stored in pillar,
 2570         using the pillar path syntax used in :mod:`pillar.get
 2571         <salt.modules.pillar.get>`. This is useful when the pillar value
 2572         contains newlines, as referencing a pillar variable using a jinja/mako
 2573         template can result in YAML formatting issues due to the newlines
 2574         causing indentation mismatches.
 2575 
 2576         For example, the following could be used to deploy an SSH private key:
 2577 
 2578         .. code-block:: yaml
 2579 
 2580             /home/deployer/.ssh/id_rsa:
 2581               file.managed:
 2582                 - user: deployer
 2583                 - group: deployer
 2584                 - mode: 600
 2585                 - attrs: a
 2586                 - contents_pillar: userdata:deployer:id_rsa
 2587 
 2588         This would populate ``/home/deployer/.ssh/id_rsa`` with the contents of
 2589         ``pillar['userdata']['deployer']['id_rsa']``. An example of this pillar
 2590         setup would be like so:
 2591 
 2592         .. code-block:: yaml
 2593 
 2594             userdata:
 2595               deployer:
 2596                 id_rsa: |
 2597                     -----BEGIN RSA PRIVATE KEY-----
 2598                     MIIEowIBAAKCAQEAoQiwO3JhBquPAalQF9qP1lLZNXVjYMIswrMe2HcWUVBgh+vY
 2599                     U7sCwx/dH6+VvNwmCoqmNnP+8gTPKGl1vgAObJAnMT623dMXjVKwnEagZPRJIxDy
 2600                     B/HaAre9euNiY3LvIzBTWRSeMfT+rWvIKVBpvwlgGrfgz70m0pqxu+UyFbAGLin+
 2601                     GpxzZAMaFpZw4sSbIlRuissXZj/sHpQb8p9M5IeO4Z3rjkCP1cxI
 2602                     -----END RSA PRIVATE KEY-----
 2603 
 2604         .. note::
 2605             The private key above is shortened to keep the example brief, but
 2606             shows how to do multiline string in YAML. The key is followed by a
 2607             pipe character, and the multiline string is indented two more
 2608             spaces.
 2609 
 2610             To avoid the hassle of creating an indented multiline YAML string,
 2611             the :mod:`file_tree external pillar <salt.pillar.file_tree>` can
 2612             be used instead. However, this will not work for binary files in
 2613             Salt releases before 2015.8.4.
 2614 
 2615     contents_grains
 2616         .. versionadded:: 2014.7.0
 2617 
 2618         Operates like ``contents``, but draws from a value stored in grains,
 2619         using the grains path syntax used in :mod:`grains.get
 2620         <salt.modules.grains.get>`. This functionality works similarly to
 2621         ``contents_pillar``, but with grains.
 2622 
 2623         For example, the following could be used to deploy a "message of the day"
 2624         file:
 2625 
 2626         .. code-block:: yaml
 2627 
 2628             write_motd:
 2629               file.managed:
 2630                 - name: /etc/motd
 2631                 - contents_grains: motd
 2632 
 2633         This would populate ``/etc/motd`` file with the contents of the ``motd``
 2634         grain. The ``motd`` grain is not a default grain, and would need to be
 2635         set prior to running the state:
 2636 
 2637         .. code-block:: bash
 2638 
 2639             salt '*' grains.set motd 'Welcome! This system is managed by Salt.'
 2640 
 2641     contents_newline : True
 2642         .. versionadded:: 2014.7.0
 2643         .. versionchanged:: 2015.8.4
 2644             This option is now ignored if the contents being deployed contain
 2645             binary data.
 2646 
 2647         If ``True``, files managed using ``contents``, ``contents_pillar``, or
 2648         ``contents_grains`` will have a newline added to the end of the file if
 2649         one is not present. Setting this option to ``False`` will ensure the
 2650         final line, or entry, does not contain a new line. If the last line, or
 2651         entry in the file does contain a new line already, this option will not
 2652         remove it.
 2653 
 2654     contents_delimiter
 2655         .. versionadded:: 2015.8.4
 2656 
 2657         Can be used to specify an alternate delimiter for ``contents_pillar``
 2658         or ``contents_grains``. This delimiter will be passed through to
 2659         :py:func:`pillar.get <salt.modules.pillar.get>` or :py:func:`grains.get
 2660         <salt.modules.grains.get>` when retrieving the contents.
 2661 
 2662     encoding
 2663         If specified, then the specified encoding will be used. Otherwise, the
 2664         file will be encoded using the system locale (usually UTF-8). See
 2665         https://docs.python.org/3/library/codecs.html#standard-encodings for
 2666         the list of available encodings.
 2667 
 2668         .. versionadded:: 2017.7.0
 2669 
 2670     encoding_errors : 'strict'
 2671         Error encoding scheme. Default is ```'strict'```.
 2672         See https://docs.python.org/2/library/codecs.html#codec-base-classes
 2673         for the list of available schemes.
 2674 
 2675         .. versionadded:: 2017.7.0
 2676 
 2677     allow_empty : True
 2678         .. versionadded:: 2015.8.4
 2679 
 2680         If set to ``False``, then the state will fail if the contents specified
 2681         by ``contents_pillar`` or ``contents_grains`` are empty.
 2682 
 2683     follow_symlinks : True
 2684         .. versionadded:: 2014.7.0
 2685 
 2686         If the desired path is a symlink follow it and make changes to the
 2687         file to which the symlink points.
 2688 
 2689     check_cmd
 2690         .. versionadded:: 2014.7.0
 2691 
 2692         The specified command will be run with an appended argument of a
 2693         *temporary* file containing the new managed contents.  If the command
 2694         exits with a zero status the new managed contents will be written to
 2695         the managed destination. If the command exits with a nonzero exit
 2696         code, the state will fail and no changes will be made to the file.
 2697 
 2698         For example, the following could be used to verify sudoers before making
 2699         changes:
 2700 
 2701         .. code-block:: yaml
 2702 
 2703             /etc/sudoers:
 2704               file.managed:
 2705                 - user: root
 2706                 - group: root
 2707                 - mode: 0440
 2708                 - attrs: i
 2709                 - source: salt://sudoers/files/sudoers.jinja
 2710                 - template: jinja
 2711                 - check_cmd: /usr/sbin/visudo -c -f
 2712 
 2713         **NOTE**: This ``check_cmd`` functions differently than the requisite
 2714         ``check_cmd``.
 2715 
 2716     tmp_dir
 2717         Directory for temp file created by ``check_cmd``. Useful for checkers
 2718         dependent on config file location (e.g. daemons restricted to their
 2719         own config directories by an apparmor profile).
 2720 
 2721         .. code-block:: yaml
 2722 
 2723             /etc/dhcp/dhcpd.conf:
 2724               file.managed:
 2725                 - user: root
 2726                 - group: root
 2727                 - mode: 0755
 2728                 - tmp_dir: '/etc/dhcp'
 2729                 - contents: "# Managed by Salt"
 2730                 - check_cmd: dhcpd -t -cf
 2731 
 2732     tmp_ext
 2733         Suffix for temp file created by ``check_cmd``. Useful for checkers
 2734         dependent on config file extension (e.g. the init-checkconf upstart
 2735         config checker).
 2736 
 2737         .. code-block:: yaml
 2738 
 2739             /etc/init/test.conf:
 2740               file.managed:
 2741                 - user: root
 2742                 - group: root
 2743                 - mode: 0440
 2744                 - tmp_ext: '.conf'
 2745                 - contents:
 2746                   - 'description "Salt Minion"'
 2747                   - 'start on started mountall'
 2748                   - 'stop on shutdown'
 2749                   - 'respawn'
 2750                   - 'exec salt-minion'
 2751                 - check_cmd: init-checkconf -f
 2752 
 2753     skip_verify : False
 2754         If ``True``, hash verification of remote file sources (``http://``,
 2755         ``https://``, ``ftp://``) will be skipped, and the ``source_hash``
 2756         argument will be ignored.
 2757 
 2758         .. versionadded:: 2016.3.0
 2759 
 2760     selinux : None
 2761         Allows setting the selinux user, role, type, and range of a managed file
 2762 
 2763         .. code-block:: yaml
 2764 
 2765             /tmp/selinux.test
 2766               file.managed:
 2767                 - user: root
 2768                 - selinux:
 2769                     seuser: system_u
 2770                     serole: object_r
 2771                     setype: system_conf_t
 2772                     seranage: s0
 2773 
 2774         .. versionadded:: Neon
 2775 
 2776     win_owner : None
 2777         The owner of the directory. If this is not passed, user will be used. If
 2778         user is not passed, the account under which Salt is running will be
 2779         used.
 2780 
 2781         .. versionadded:: 2017.7.0
 2782 
 2783     win_perms : None
 2784         A dictionary containing permissions to grant and their propagation. For
 2785         example: ``{'Administrators': {'perms': 'full_control'}}`` Can be a
 2786         single basic perm or a list of advanced perms. ``perms`` must be
 2787         specified. ``applies_to`` does not apply to file objects.
 2788 
 2789         .. versionadded:: 2017.7.0
 2790 
 2791     win_deny_perms : None
 2792         A dictionary containing permissions to deny and their propagation. For
 2793         example: ``{'Administrators': {'perms': 'full_control'}}`` Can be a
 2794         single basic perm or a list of advanced perms. ``perms`` must be
 2795         specified. ``applies_to`` does not apply to file objects.
 2796 
 2797         .. versionadded:: 2017.7.0
 2798 
 2799     win_inheritance : True
 2800         True to inherit permissions from the parent directory, False not to
 2801         inherit permission.
 2802 
 2803         .. versionadded:: 2017.7.0
 2804 
 2805     win_perms_reset : False
 2806         If ``True`` the existing DACL will be cleared and replaced with the
 2807         settings defined in this function. If ``False``, new entries will be
 2808         appended to the existing DACL. Default is ``False``.
 2809 
 2810         .. versionadded:: 2018.3.0
 2811 
 2812     Here's an example using the above ``win_*`` parameters:
 2813 
 2814     .. code-block:: yaml
 2815 
 2816         create_config_file:
 2817           file.managed:
 2818             - name: C:\config\settings.cfg
 2819             - source: salt://settings.cfg
 2820             - win_owner: Administrators
 2821             - win_perms:
 2822                 # Basic Permissions
 2823                 dev_ops:
 2824                   perms: full_control
 2825                 # List of advanced permissions
 2826                 appuser:
 2827                   perms:
 2828                     - read_attributes
 2829                     - read_ea
 2830                     - create_folders
 2831                     - read_permissions
 2832                 joe_snuffy:
 2833                   perms: read
 2834             - win_deny_perms:
 2835                 fred_snuffy:
 2836                   perms: full_control
 2837             - win_inheritance: False
 2838 
 2839     verify_ssl
 2840         If ``False``, remote https file sources (``https://``) and source_hash
 2841         will not attempt to validate the servers certificate. Default is True.
 2842 
 2843         .. versionadded:: 3002
 2844     """
 2845     if "env" in kwargs:
 2846         # "env" is not supported; Use "saltenv".
 2847         kwargs.pop("env")
 2848 
 2849     name = os.path.expanduser(name)
 2850 
 2851     ret = {"changes": {}, "comment": "", "name": name, "result": True}
 2852 
 2853     if not name:
 2854         return _error(ret, "Destination file name is required")
 2855 
 2856     if mode is not None and salt.utils.platform.is_windows():
 2857         return _error(ret, "The 'mode' option is not supported on Windows")
 2858 
 2859     if attrs is not None and salt.utils.platform.is_windows():
 2860         return _error(ret, "The 'attrs' option is not supported on Windows")
 2861 
 2862     if selinux is not None and not salt.utils.platform.is_linux():
 2863         return _error(ret, "The 'selinux' option is only supported on Linux")
 2864 
 2865     if selinux:
 2866         seuser = selinux.get("seuser", None)
 2867         serole = selinux.get("serole", None)
 2868         setype = selinux.get("setype", None)
 2869         serange = selinux.get("serange", None)
 2870     else:
 2871         seuser = serole = setype = serange = None
 2872 
 2873     try:
 2874         keep_mode = mode.lower() == "keep"
 2875         if keep_mode:
 2876             # We're not hard-coding the mode, so set it to None
 2877             mode = None
 2878     except AttributeError:
 2879         keep_mode = False
 2880 
 2881     # Make sure that any leading zeros stripped by YAML loader are added back
 2882     mode = salt.utils.files.normalize_mode(mode)
 2883 
 2884     contents_count = len(
 2885         [x for x in (contents, contents_pillar, contents_grains) if x is not None]
 2886     )
 2887 
 2888     if source and contents_count > 0:
 2889         return _error(
 2890             ret,
 2891             "'source' cannot be used in combination with 'contents', "
 2892             "'contents_pillar', or 'contents_grains'",
 2893         )
 2894     elif keep_mode and contents_count > 0:
 2895         return _error(
 2896             ret,
 2897             "Mode preservation cannot be used in combination with 'contents', "
 2898             "'contents_pillar', or 'contents_grains'",
 2899         )
 2900     elif contents_count > 1:
 2901         return _error(
 2902             ret,
 2903             "Only one of 'contents', 'contents_pillar', and "
 2904             "'contents_grains' is permitted",
 2905         )
 2906 
 2907     # If no source is specified, set replace to False, as there is nothing
 2908     # with which to replace the file.
 2909     if not source and contents_count == 0 and replace:
 2910         replace = False
 2911         log.warning(
 2912             "State for file: {} - Neither 'source' nor 'contents' nor "
 2913             "'contents_pillar' nor 'contents_grains' was defined, yet "
 2914             "'replace' was set to 'True'. As there is no source to "
 2915             "replace the file with, 'replace' has been set to 'False' to "
 2916             "avoid reading the file unnecessarily.".format(name)
 2917         )
 2918 
 2919     if "file_mode" in kwargs:
 2920         ret.setdefault("warnings", []).append(
 2921             "The 'file_mode' argument will be ignored.  "
 2922             "Please use 'mode' instead to set file permissions."
 2923         )
 2924 
 2925     # Use this below to avoid multiple '\0' checks and save some CPU cycles
 2926     if contents_pillar is not None:
 2927         if isinstance(contents_pillar, list):
 2928             list_contents = []
 2929             for nextp in contents_pillar:
 2930                 nextc = __salt__["pillar.get"](
 2931                     nextp, __NOT_FOUND, delimiter=contents_delimiter
 2932                 )
 2933                 if nextc is __NOT_FOUND:
 2934                     return _error(ret, "Pillar {} does not exist".format(nextp))
 2935                 list_contents.append(nextc)
 2936             use_contents = os.linesep.join(list_contents)
 2937         else:
 2938             use_contents = __salt__["pillar.get"](
 2939                 contents_pillar, __NOT_FOUND, delimiter=contents_delimiter
 2940             )
 2941             if use_contents is __NOT_FOUND:
 2942                 return _error(ret, "Pillar {} does not exist".format(contents_pillar))
 2943 
 2944     elif contents_grains is not None:
 2945         if isinstance(contents_grains, list):
 2946             list_contents = []
 2947             for nextg in contents_grains:
 2948                 nextc = __salt__["grains.get"](
 2949                     nextg, __NOT_FOUND, delimiter=contents_delimiter
 2950                 )
 2951                 if nextc is __NOT_FOUND:
 2952                     return _error(ret, "Grain {} does not exist".format(nextc))
 2953                 list_contents.append(nextc)
 2954             use_contents = os.linesep.join(list_contents)
 2955         else:
 2956             use_contents = __salt__["grains.get"](
 2957                 contents_grains, __NOT_FOUND, delimiter=contents_delimiter
 2958             )
 2959             if use_contents is __NOT_FOUND:
 2960                 return _error(ret, "Grain {} does not exist".format(contents_grains))
 2961 
 2962     elif contents is not None:
 2963         use_contents = contents
 2964 
 2965     else:
 2966         use_contents = None
 2967 
 2968     if use_contents is not None:
 2969         if not allow_empty and not use_contents:
 2970             if contents_pillar:
 2971                 contents_id = "contents_pillar {}".format(contents_pillar)
 2972             elif contents_grains:
 2973                 contents_id = "contents_grains {}".format(contents_grains)
 2974             else:
 2975                 contents_id = "'contents'"
 2976             return _error(
 2977                 ret,
 2978                 "{} value would result in empty contents. Set allow_empty "
 2979                 "to True to allow the managed file to be empty.".format(contents_id),
 2980             )
 2981 
 2982         try:
 2983             validated_contents = _validate_str_list(use_contents, encoding=encoding)
 2984             if not validated_contents:
 2985                 return _error(
 2986                     ret,
 2987                     "Contents specified by contents/contents_pillar/"
 2988                     "contents_grains is not a string or list of strings, and "
 2989                     "is not binary data. SLS is likely malformed.",
 2990                 )
 2991             contents = ""
 2992             for part in validated_contents:
 2993                 for line in part.splitlines():
 2994                     contents += line.rstrip("\n").rstrip("\r") + os.linesep
 2995             if not contents_newline:
 2996                 # If contents newline is set to False, strip out the newline
 2997                 # character and carriage return character
 2998                 contents = contents.rstrip("\n").rstrip("\r")
 2999 
 3000         except UnicodeDecodeError:
 3001             # Either something terrible happened, or we have binary data.
 3002             if template:
 3003                 return _error(
 3004                     ret,
 3005                     "Contents specified by contents/contents_pillar/"
 3006                     "contents_grains appears to be binary data, and"
 3007                     " as will not be able to be treated as a Jinja"
 3008                     " template.",
 3009                 )
 3010             contents = use_contents
 3011         if template:
 3012             contents = __salt__["file.apply_template_on_contents"](
 3013                 contents,
 3014                 template=template,
 3015                 context=context,
 3016                 defaults=defaults,
 3017                 saltenv=__env__,
 3018             )
 3019             if not isinstance(contents, str):
 3020                 if "result" in contents:
 3021                     ret["result"] = contents["result"]
 3022                 else:
 3023                     ret["result"] = False
 3024                 if "comment" in contents:
 3025                     ret["comment"] = contents["comment"]
 3026                 else:
 3027                     ret["comment"] = "Error while applying template on contents"
 3028                 return ret
 3029 
 3030     user = _test_owner(kwargs, user=user)
 3031     if salt.utils.platform.is_windows():
 3032 
 3033         # If win_owner not passed, use user
 3034         if win_owner is None:
 3035             win_owner = user if user else None
 3036 
 3037         # Group isn't relevant to Windows, use win_perms/win_deny_perms
 3038         if group is not None:
 3039             log.warning(
 3040                 "The group argument for {} has been ignored as this is "
 3041                 "a Windows system. Please use the `win_*` parameters to set "
 3042                 "permissions in Windows.".format(name)
 3043             )
 3044         group = user
 3045 
 3046     if not create:
 3047         if not os.path.isfile(name):
 3048             # Don't create a file that is not already present
 3049             ret["comment"] = (
 3050                 "File {} is not present and is not set for " "creation"
 3051             ).format(name)
 3052             return ret
 3053     u_check = _check_user(user, group)
 3054     if u_check:
 3055         # The specified user or group do not exist
 3056         return _error(ret, u_check)
 3057     if not os.path.isabs(name):
 3058         return _error(ret, "Specified file {} is not an absolute path".format(name))
 3059 
 3060     if os.path.isdir(name):
 3061         ret["comment"] = "Specified target {} is a directory".format(name)
 3062         ret["result"] = False
 3063         return ret
 3064 
 3065     if context is None:
 3066         context = {}
 3067     elif not isinstance(context, dict):
 3068         return _error(ret, "Context must be formed as a dict")
 3069     if defaults and not isinstance(defaults, dict):
 3070         return _error(ret, "Defaults must be formed as a dict")
 3071 
 3072     if not replace and os.path.exists(name):
 3073         ret_perms = {}
 3074         # Check and set the permissions if necessary
 3075         if salt.utils.platform.is_windows():
 3076             ret = __salt__["file.check_perms"](
 3077                 path=name,
 3078                 ret=ret,
 3079                 owner=win_owner,
 3080                 grant_perms=win_perms,
 3081                 deny_perms=win_deny_perms,
 3082                 inheritance=win_inheritance,
 3083                 reset=win_perms_reset,
 3084             )
 3085         else:
 3086             ret, ret_perms = __salt__["file.check_perms"](
 3087                 name,
 3088                 ret,
 3089                 user,
 3090                 group,
 3091                 mode,
 3092                 attrs,
 3093                 follow_symlinks,
 3094                 seuser=seuser,
 3095                 serole=serole,
 3096                 setype=setype,
 3097                 serange=serange,
 3098             )
 3099         if __opts__["test"]:
 3100             if (
 3101                 isinstance(ret_perms, dict)
 3102                 and "lmode" in ret_perms
 3103                 and mode != ret_perms["lmode"]
 3104             ):
 3105                 ret["comment"] = (
 3106                     "File {} will be updated with permissions "
 3107                     "{} from its current "
 3108                     "state of {}".format(name, mode, ret_perms["lmode"])
 3109                 )
 3110             else:
 3111                 ret["comment"] = "File {} not updated".format(name)
 3112         elif not ret["changes"] and ret["result"]:
 3113             ret["comment"] = (
 3114                 "File {} exists with proper permissions. "
 3115                 "No changes made.".format(name)
 3116             )
 3117         return ret
 3118 
 3119     accum_data, _ = _load_accumulators()
 3120     if name in accum_data:
 3121         if not context:
 3122             context = {}
 3123         context["accumulator"] = accum_data[name]
 3124 
 3125     try:
 3126         if __opts__["test"]:
 3127             if "file.check_managed_changes" in __salt__:
 3128                 ret["changes"] = __salt__["file.check_managed_changes"](
 3129                     name,
 3130                     source,
 3131                     source_hash,
 3132                     source_hash_name,
 3133                     user,
 3134                     group,
 3135                     mode,
 3136                     attrs,
 3137                     template,
 3138                     context,
 3139                     defaults,
 3140                     __env__,
 3141                     contents,
 3142                     skip_verify,
 3143                     keep_mode,
 3144                     seuser=seuser,
 3145                     serole=serole,
 3146                     setype=setype,
 3147                     serange=serange,
 3148                     verify_ssl=verify_ssl,
 3149                     **kwargs
 3150                 )
 3151 
 3152                 if salt.utils.platform.is_windows():
 3153                     try:
 3154                         ret = __salt__["file.check_perms"](
 3155                             path=name,
 3156                             ret=ret,
 3157                             owner=win_owner,
 3158                             grant_perms=win_perms,
 3159                             deny_perms=win_deny_perms,
 3160                             inheritance=win_inheritance,
 3161                             reset=win_perms_reset,
 3162                         )
 3163                     except CommandExecutionError as exc:
 3164                         if exc.strerror.startswith("Path not found"):
 3165                             ret["changes"]["newfile"] = name
 3166 
 3167             if isinstance(ret["changes"], tuple):
 3168                 ret["result"], ret["comment"] = ret["changes"]
 3169             elif ret["changes"]:
 3170                 ret["result"] = None
 3171                 ret["comment"] = "The file {} is set to be changed".format(name)
 3172                 ret["comment"] += (
 3173                     "\nNote: No changes made, actual changes may\n"
 3174                     "be different due to other states."
 3175                 )
 3176                 if "diff" in ret["changes"] and not show_changes:
 3177                     ret["changes"]["diff"] = "<show_changes=False>"
 3178             else:
 3179                 ret["result"] = True
 3180                 ret["comment"] = "The file {} is in the correct state".format(name)
 3181 
 3182             return ret
 3183 
 3184         # If the source is a list then find which file exists
 3185         source, source_hash = __salt__["file.source_list"](source, source_hash, __env__)
 3186     except CommandExecutionError as exc:
 3187         ret["result"] = False
 3188         ret["comment"] = "Unable to manage file: {}".format(exc)
 3189         return ret
 3190 
 3191     # Gather the source file from the server
 3192     try:
 3193         sfn, source_sum, comment_ = __salt__["file.get_managed"](
 3194             name,
 3195             template,
 3196             source,
 3197             source_hash,
 3198             source_hash_name,
 3199             user,
 3200             group,
 3201             mode,
 3202             attrs,
 3203             __env__,
 3204             context,
 3205             defaults,
 3206             skip_verify,
 3207             verify_ssl=verify_ssl,
 3208             **kwargs
 3209         )
 3210     except Exception as exc:  # pylint: disable=broad-except
 3211         ret["changes"] = {}
 3212         log.debug(traceback.format_exc())
 3213         return _error(ret, "Unable to manage file: {}".format(exc))
 3214 
 3215     tmp_filename = None
 3216 
 3217     if check_cmd:
 3218         tmp_filename = salt.utils.files.mkstemp(suffix=tmp_ext, dir=tmp_dir)
 3219 
 3220         # if exists copy existing file to tmp to compare
 3221         if __salt__["file.file_exists"](name):
 3222             try:
 3223                 __salt__["file.copy"](name, tmp_filename)
 3224             except Exception as exc:  # pylint: disable=broad-except
 3225                 return _error(
 3226                     ret,
 3227                     "Unable to copy file {} to {}: {}".format(name, tmp_filename, exc),
 3228                 )
 3229 
 3230         try:
 3231             ret = __salt__["file.manage_file"](
 3232                 tmp_filename,
 3233                 sfn,
 3234                 ret,
 3235                 source,
 3236                 source_sum,
 3237                 user,
 3238                 group,
 3239                 mode,
 3240                 attrs,
 3241                 __env__,
 3242                 backup,
 3243                 makedirs,
 3244                 template,
 3245                 show_changes,
 3246                 contents,
 3247                 dir_mode,
 3248                 follow_symlinks,
 3249                 skip_verify,
 3250                 keep_mode,
 3251                 win_owner=win_owner,
 3252                 win_perms=win_perms,
 3253                 win_deny_perms=win_deny_perms,
 3254                 win_inheritance=win_inheritance,
 3255                 win_perms_reset=win_perms_reset,
 3256                 encoding=encoding,
 3257                 encoding_errors=encoding_errors,
 3258                 seuser=seuser,
 3259                 serole=serole,
 3260                 setype=setype,
 3261                 serange=serange,
 3262                 **kwargs
 3263             )
 3264         except Exception as exc:  # pylint: disable=broad-except
 3265             ret["changes"] = {}
 3266             log.debug(traceback.format_exc())
 3267             salt.utils.files.remove(tmp_filename)
 3268             if not keep_source:
 3269                 if not sfn and source and _urlparse(source).scheme == "salt":
 3270                     # The file would not have been cached until manage_file was
 3271                     # run, so check again here for a cached copy.
 3272                     sfn = __salt__["cp.is_cached"](source, __env__)
 3273                 if sfn:
 3274                     salt.utils.files.remove(sfn)
 3275             return _error(ret, "Unable to check_cmd file: {}".format(exc))
 3276 
 3277         # file being updated to verify using check_cmd
 3278         if ret["changes"]:
 3279             # Reset ret
 3280             ret = {"changes": {}, "comment": "", "name": name, "result": True}
 3281 
 3282             check_cmd_opts = {}
 3283             if "shell" in __grains__:
 3284                 check_cmd_opts["shell"] = __grains__["shell"]
 3285 
 3286             cret = mod_run_check_cmd(check_cmd, tmp_filename, **check_cmd_opts)
 3287             if isinstance(cret, dict):
 3288                 ret.update(cret)
 3289                 salt.utils.files.remove(tmp_filename)
 3290                 return ret
 3291 
 3292             # Since we generated a new tempfile and we are not returning here
 3293             # lets change the original sfn to the new tempfile or else we will
 3294             # get file not found
 3295 
 3296             sfn = tmp_filename
 3297 
 3298         else:
 3299             ret = {"changes": {}, "comment": "", "name": name, "result": True}
 3300 
 3301     if comment_ and contents is None:
 3302         return _error(ret, comment_)
 3303     else:
 3304         try:
 3305             return __salt__["file.manage_file"](
 3306                 name,
 3307                 sfn,
 3308                 ret,
 3309                 source,
 3310                 source_sum,
 3311                 user,
 3312                 group,
 3313                 mode,
 3314                 attrs,
 3315                 __env__,
 3316                 backup,
 3317                 makedirs,
 3318                 template,
 3319                 show_changes,
 3320                 contents,
 3321                 dir_mode,
 3322                 follow_symlinks,
 3323                 skip_verify,
 3324                 keep_mode,
 3325                 win_owner=win_owner,
 3326                 win_perms=win_perms,
 3327                 win_deny_perms=win_deny_perms,
 3328                 win_inheritance=win_inheritance,
 3329                 win_perms_reset=win_perms_reset,
 3330                 encoding=encoding,
 3331                 encoding_errors=encoding_errors,
 3332                 seuser=seuser,
 3333                 serole=serole,
 3334                 setype=setype,
 3335                 serange=serange,
 3336                 **kwargs
 3337             )
 3338         except Exception as exc:  # pylint: disable=broad-except
 3339             ret["changes"] = {}
 3340             log.debug(traceback.format_exc())
 3341             return _error(ret, "Unable to manage file: {}".format(exc))
 3342         finally:
 3343             if tmp_filename:
 3344                 salt.utils.files.remove(tmp_filename)
 3345             if not keep_source:
 3346                 if not sfn and source and _urlparse(source).scheme == "salt":
 3347                     # The file would not have been cached until manage_file was
 3348                     # run, so check again here for a cached copy.
 3349                     sfn = __salt__["cp.is_cached"](source, __env__)
 3350                 if sfn:
 3351                     salt.utils.files.remove(sfn)
 3352 
 3353 
 3354 _RECURSE_TYPES = ["user", "group", "mode", "ignore_files", "ignore_dirs", "silent"]
 3355 
 3356 
 3357 def _get_recurse_set(recurse):
 3358     """
 3359     Converse *recurse* definition to a set of strings.
 3360 
 3361     Raises TypeError or ValueError when *recurse* has wrong structure.
 3362     """
 3363     if not recurse:
 3364         return set()
 3365     if not isinstance(recurse, list):
 3366         raise TypeError('"recurse" must be formed as a list of strings')
 3367     try:
 3368         recurse_set = set(recurse)
 3369     except TypeError:  # non-hashable elements
 3370         recurse_set = None
 3371     if recurse_set is None or not set(_RECURSE_TYPES) >= recurse_set:
 3372         raise ValueError(
 3373             'Types for "recurse" limited to {}.'.format(
 3374                 ", ".join('"{}"'.format(rtype) for rtype in _RECURSE_TYPES)
 3375             )
 3376         )
 3377     if "ignore_files" in recurse_set and "ignore_dirs" in recurse_set:
 3378         raise ValueError(
 3379             'Must not specify "recurse" options "ignore_files"'
 3380             ' and "ignore_dirs" at the same time.'
 3381         )
 3382     return recurse_set
 3383 
 3384 
 3385 def _depth_limited_walk(top, max_depth=None):
 3386     """
 3387     Walk the directory tree under root up till reaching max_depth.
 3388     With max_depth=None (default), do not limit depth.
 3389     """
 3390     for root, dirs, files in salt.utils.path.os_walk(top):
 3391         if max_depth is not None:
 3392             rel_depth = root.count(os.path.sep) - top.count(os.path.sep)
 3393             if rel_depth >= max_depth:
 3394                 del dirs[:]
 3395         yield (str(root), list(dirs), list(files))
 3396 
 3397 
 3398 def directory(
 3399     name,
 3400     user=None,
 3401     group=None,
 3402     recurse=None,
 3403     max_depth=None,
 3404     dir_mode=None,
 3405     file_mode=None,
 3406     makedirs=False,
 3407     clean=False,
 3408     require=None,
 3409     exclude_pat=None,
 3410     follow_symlinks=False,
 3411     force=False,
 3412     backupname=None,
 3413     allow_symlink=True,
 3414     children_only=False,
 3415     win_owner=None,
 3416     win_perms=None,
 3417     win_deny_perms=None,
 3418     win_inheritance=True,
 3419     win_perms_reset=False,
 3420     **kwargs
 3421 ):
 3422     r"""
 3423     Ensure that a named directory is present and has the right perms
 3424 
 3425     name
 3426         The location to create or manage a directory, as an absolute path
 3427 
 3428     user
 3429         The user to own the directory; this defaults to the user salt is
 3430         running as on the minion
 3431 
 3432     group
 3433         The group ownership set for the directory; this defaults to the group
 3434         salt is running as on the minion. On Windows, this is ignored
 3435 
 3436     recurse
 3437         Enforce user/group ownership and mode of directory recursively. Accepts
 3438         a list of strings representing what you would like to recurse.  If
 3439         ``mode`` is defined, will recurse on both ``file_mode`` and ``dir_mode`` if
 3440         they are defined.  If ``ignore_files`` or ``ignore_dirs`` is included, files or
 3441         directories will be left unchanged respectively.
 3442         directories will be left unchanged respectively. If ``silent`` is defined,
 3443         individual file/directory change notifications will be suppressed.
 3444 
 3445         Example:
 3446 
 3447         .. code-block:: yaml
 3448 
 3449             /var/log/httpd:
 3450               file.directory:
 3451                 - user: root
 3452                 - group: root
 3453                 - dir_mode: 755
 3454                 - file_mode: 644
 3455                 - recurse:
 3456                   - user
 3457                   - group
 3458                   - mode
 3459 
 3460         Leave files or directories unchanged:
 3461 
 3462         .. code-block:: yaml
 3463 
 3464             /var/log/httpd:
 3465               file.directory:
 3466                 - user: root
 3467                 - group: root
 3468                 - dir_mode: 755
 3469                 - file_mode: 644
 3470                 - recurse:
 3471                   - user
 3472                   - group
 3473                   - mode
 3474                   - ignore_dirs
 3475 
 3476         .. versionadded:: 2015.5.0
 3477 
 3478     max_depth
 3479         Limit the recursion depth. The default is no limit=None.
 3480         'max_depth' and 'clean' are mutually exclusive.
 3481 
 3482         .. versionadded:: 2016.11.0
 3483 
 3484     dir_mode / mode
 3485         The permissions mode to set any directories created. Not supported on
 3486         Windows.
 3487 
 3488         The default mode for new files and directories corresponds umask of salt
 3489         process. For existing files and directories it's not enforced.
 3490 
 3491     file_mode
 3492         The permissions mode to set any files created if 'mode' is run in
 3493         'recurse'. This defaults to dir_mode. Not supported on Windows.
 3494 
 3495         The default mode for new files and directories corresponds umask of salt
 3496         process. For existing files and directories it's not enforced.
 3497 
 3498     makedirs
 3499         If the directory is located in a path without a parent directory, then
 3500         the state will fail. If makedirs is set to True, then the parent
 3501         directories will be created to facilitate the creation of the named
 3502         file.
 3503 
 3504     clean
 3505         Make sure that only files that are set up by salt and required by this
 3506         function are kept. If this option is set then everything in this
 3507         directory will be deleted unless it is required.
 3508         'clean' and 'max_depth' are mutually exclusive.
 3509 
 3510     require
 3511         Require other resources such as packages or files
 3512 
 3513     exclude_pat
 3514         When 'clean' is set to True, exclude this pattern from removal list
 3515         and preserve in the destination.
 3516 
 3517     follow_symlinks : False
 3518         If the desired path is a symlink (or ``recurse`` is defined and a
 3519         symlink is encountered while recursing), follow it and check the
 3520         permissions of the directory/file to which the symlink points.
 3521 
 3522         .. versionadded:: 2014.1.4
 3523 
 3524         .. versionchanged:: 3001.1
 3525             If set to False symlinks permissions are ignored on Linux systems
 3526             because it does not support permissions modification. Symlinks
 3527             permissions are always 0o777 on Linux.
 3528 
 3529     force
 3530         If the name of the directory exists and is not a directory and
 3531         force is set to False, the state will fail. If force is set to
 3532         True, the file in the way of the directory will be deleted to
 3533         make room for the directory, unless backupname is set,
 3534         then it will be renamed.
 3535 
 3536         .. versionadded:: 2014.7.0
 3537 
 3538     backupname
 3539         If the name of the directory exists and is not a directory, it will be
 3540         renamed to the backupname. If the backupname already
 3541         exists and force is False, the state will fail. Otherwise, the
 3542         backupname will be removed first.
 3543 
 3544         .. versionadded:: 2014.7.0
 3545 
 3546     allow_symlink : True
 3547         If allow_symlink is True and the specified path is a symlink, it will be
 3548         allowed to remain if it points to a directory. If allow_symlink is False
 3549         then the state will fail, unless force is also set to True, in which case
 3550         it will be removed or renamed, depending on the value of the backupname
 3551         argument.
 3552 
 3553         .. versionadded:: 2014.7.0
 3554 
 3555     children_only : False
 3556         If children_only is True the base of a path is excluded when performing
 3557         a recursive operation. In case of /path/to/base, base will be ignored
 3558         while all of /path/to/base/* are still operated on.
 3559 
 3560     win_owner : None
 3561         The owner of the directory. If this is not passed, user will be used. If
 3562         user is not passed, the account under which Salt is running will be
 3563         used.
 3564 
 3565         .. versionadded:: 2017.7.0
 3566 
 3567     win_perms : None
 3568         A dictionary containing permissions to grant and their propagation. For
 3569         example: ``{'Administrators': {'perms': 'full_control', 'applies_to':
 3570         'this_folder_only'}}`` Can be a single basic perm or a list of advanced
 3571         perms. ``perms`` must be specified. ``applies_to`` is optional and
 3572         defaults to ``this_folder_subfolder_files``.
 3573 
 3574         .. versionadded:: 2017.7.0
 3575 
 3576     win_deny_perms : None
 3577         A dictionary containing permissions to deny and their propagation. For
 3578         example: ``{'Administrators': {'perms': 'full_control', 'applies_to':
 3579         'this_folder_only'}}`` Can be a single basic perm or a list of advanced
 3580         perms.
 3581 
 3582         .. versionadded:: 2017.7.0
 3583 
 3584     win_inheritance : True
 3585         True to inherit permissions from the parent directory, False not to
 3586         inherit permission.
 3587 
 3588         .. versionadded:: 2017.7.0
 3589 
 3590     win_perms_reset : False
 3591         If ``True`` the existing DACL will be cleared and replaced with the
 3592         settings defined in this function. If ``False``, new entries will be
 3593         appended to the existing DACL. Default is ``False``.
 3594 
 3595         .. versionadded:: 2018.3.0
 3596 
 3597     Here's an example using the above ``win_*`` parameters:
 3598 
 3599     .. code-block:: yaml
 3600 
 3601         create_config_dir:
 3602           file.directory:
 3603             - name: 'C:\config\'
 3604             - win_owner: Administrators
 3605             - win_perms:
 3606                 # Basic Permissions
 3607                 dev_ops:
 3608                   perms: full_control
 3609                 # List of advanced permissions
 3610                 appuser:
 3611                   perms:
 3612                     - read_attributes
 3613                     - read_ea
 3614                     - create_folders
 3615                     - read_permissions
 3616                   applies_to: this_folder_only
 3617                 joe_snuffy:
 3618                   perms: read
 3619                   applies_to: this_folder_files
 3620             - win_deny_perms:
 3621                 fred_snuffy:
 3622                   perms: full_control
 3623             - win_inheritance: False
 3624     """
 3625     name = os.path.expanduser(name)
 3626     ret = {"name": name, "changes": {}, "result": True, "comment": ""}
 3627     if not name:
 3628         return _error(ret, "Must provide name to file.directory")
 3629     # Remove trailing slash, if present and we're not working on "/" itself
 3630     if name[-1] == "/" and name != "/":
 3631         name = name[:-1]
 3632 
 3633     if max_depth is not None and clean:
 3634         return _error(ret, "Cannot specify both max_depth and clean")
 3635 
 3636     user = _test_owner(kwargs, user=user)
 3637     if salt.utils.platform.is_windows():
 3638 
 3639         # If win_owner not passed, use user
 3640         if win_owner is None:
 3641             win_owner = user if user else salt.utils.win_functions.get_current_user()
 3642 
 3643         # Group isn't relevant to Windows, use win_perms/win_deny_perms
 3644         if group is not None:
 3645             log.warning(
 3646                 "The group argument for {} has been ignored as this is "
 3647                 "a Windows system. Please use the `win_*` parameters to set "
 3648                 "permissions in Windows.".format(name)
 3649             )
 3650         group = user
 3651 
 3652     if "mode" in kwargs and not dir_mode:
 3653         dir_mode = kwargs.get("mode", [])
 3654 
 3655     if not file_mode:
 3656         file_mode = dir_mode
 3657 
 3658     # Make sure that leading zeros stripped by YAML loader are added back
 3659     dir_mode = salt.utils.files.normalize_mode(dir_mode)
 3660     file_mode = salt.utils.files.normalize_mode(file_mode)
 3661 
 3662     if salt.utils.platform.is_windows():
 3663         # Verify win_owner is valid on the target system
 3664         try:
 3665             salt.utils.win_dacl.get_sid(win_owner)
 3666         except CommandExecutionError as exc:
 3667             return _error(ret, exc)
 3668     else:
 3669         # Verify user and group are valid
 3670         u_check = _check_user(user, group)
 3671         if u_check:
 3672             # The specified user or group do not exist
 3673             return _error(ret, u_check)
 3674 
 3675     # Must be an absolute path
 3676     if not os.path.isabs(name):
 3677         return _error(ret, "Specified file {} is not an absolute path".format(name))
 3678 
 3679     # Check for existing file or symlink
 3680     if (
 3681         os.path.isfile(name)
 3682         or (not allow_symlink and os.path.islink(name))
 3683         or (force and os.path.islink(name))
 3684     ):
 3685         # Was a backupname specified
 3686         if backupname is not None:
 3687             # Make a backup first
 3688             if os.path.lexists(backupname):
 3689                 if not force:
 3690                     return _error(
 3691                         ret,
 3692                         (
 3693                             ("File exists where the backup target {} should go").format(
 3694                                 backupname
 3695                             )
 3696                         ),
 3697                     )
 3698                 else:
 3699                     __salt__["file.remove"](backupname)
 3700             os.rename(name, backupname)
 3701         elif force:
 3702             # Remove whatever is in the way
 3703             if os.path.isfile(name):
 3704                 if __opts__["test"]:
 3705                     ret["changes"]["forced"] = "File would be forcibly replaced"
 3706                 else:
 3707                     os.remove(name)
 3708                     ret["changes"]["forced"] = "File was forcibly replaced"
 3709             elif __salt__["file.is_link"](name):
 3710                 if __opts__["test"]:
 3711                     ret["changes"]["forced"] = "Symlink would be forcibly replaced"
 3712                 else:
 3713                     __salt__["file.remove"](name)
 3714                     ret["changes"]["forced"] = "Symlink was forcibly replaced"
 3715             else:
 3716                 if __opts__["test"]:
 3717                     ret["changes"]["forced"] = "Directory would be forcibly replaced"
 3718                 else:
 3719                     __salt__["file.remove"](name)
 3720                     ret["changes"]["forced"] = "Directory was forcibly replaced"
 3721         else:
 3722             if os.path.isfile(name):
 3723                 return _error(
 3724                     ret, "Specified location {} exists and is a file".format(name)
 3725                 )
 3726             elif os.path.islink(name):
 3727                 return _error(
 3728                     ret, "Specified location {} exists and is a symlink".format(name)
 3729                 )
 3730 
 3731     # Check directory?
 3732     if salt.utils.platform.is_windows():
 3733         tresult, tcomment, tchanges = _check_directory_win(
 3734             name=name,
 3735             win_owner=win_owner,
 3736             win_perms=win_perms,
 3737             win_deny_perms=win_deny_perms,
 3738             win_inheritance=win_inheritance,
 3739             win_perms_reset=win_perms_reset,
 3740         )
 3741     else:
 3742         tresult, tcomment, tchanges = _check_directory(
 3743             name,
 3744             user,
 3745             group,
 3746             recurse or [],
 3747             dir_mode,
 3748             file_mode,
 3749             clean,
 3750             require,
 3751             exclude_pat,
 3752             max_depth,
 3753             follow_symlinks,
 3754         )
 3755 
 3756     if tchanges:
 3757         ret["changes"].update(tchanges)
 3758 
 3759     # Don't run through the reset of the function if there are no changes to be
 3760     # made
 3761     if __opts__["test"] or not ret["changes"]:
 3762         ret["result"] = tresult
 3763         ret["comment"] = tcomment
 3764         return ret
 3765 
 3766     if not os.path.isdir(name):
 3767         # The dir does not exist, make it
 3768         if not os.path.isdir(os.path.dirname(name)):
 3769             # The parent directory does not exist, create them
 3770             if makedirs:
 3771                 # Everything's good, create the parent Dirs
 3772                 try:
 3773                     _makedirs(
 3774                         name=name,
 3775                         user=user,
 3776                         group=group,
 3777                         dir_mode=dir_mode,
 3778                         win_owner=win_owner,
 3779                         win_perms=win_perms,
 3780                         win_deny_perms=win_deny_perms,
 3781                         win_inheritance=win_inheritance,
 3782                     )
 3783                 except CommandExecutionError as exc:
 3784                     return _error(ret, "Drive {} is not mapped".format(exc.message))
 3785             else:
 3786                 return _error(ret, "No directory to create {} in".format(name))
 3787 
 3788         if salt.utils.platform.is_windows():
 3789             __salt__["file.mkdir"](
 3790                 path=name,
 3791                 owner=win_owner,
 3792                 grant_perms=win_perms,
 3793                 deny_perms=win_deny_perms,
 3794                 inheritance=win_inheritance,
 3795                 reset=win_perms_reset,
 3796             )
 3797         else:
 3798             __salt__["file.mkdir"](name, user=user, group=group, mode=dir_mode)
 3799 
 3800         ret["changes"][name] = "New Dir"
 3801 
 3802     if not os.path.isdir(name):
 3803         return _error(ret, "Failed to create directory {}".format(name))
 3804 
 3805     # issue 32707: skip this __salt__['file.check_perms'] call if children_only == True
 3806     # Check permissions
 3807     if not children_only:
 3808         if salt.utils.platform.is_windows():
 3809             ret = __salt__["file.check_perms"](
 3810                 path=name,
 3811                 ret=ret,
 3812                 owner=win_owner,
 3813                 grant_perms=win_perms,
 3814                 deny_perms=win_deny_perms,
 3815                 inheritance=win_inheritance,
 3816                 reset=win_perms_reset,
 3817             )
 3818         else:
 3819             ret, perms = __salt__["file.check_perms"](
 3820                 name, ret, user, group, dir_mode, None, follow_symlinks
 3821             )
 3822 
 3823     errors = []
 3824     if recurse or clean:
 3825         # walk path only once and store the result
 3826         walk_l = list(_depth_limited_walk(name, max_depth))
 3827         # root: (dirs, files) structure, compatible for python2.6
 3828         walk_d = {}
 3829         for i in walk_l:
 3830             walk_d[i[0]] = (i[1], i[2])
 3831 
 3832     recurse_set = None
 3833     if recurse:
 3834         try:
 3835             recurse_set = _get_recurse_set(recurse)
 3836         except (TypeError, ValueError) as exc:
 3837             ret["result"] = False
 3838             ret["comment"] = "{}".format(exc)
 3839             # NOTE: Should this be enough to stop the whole check altogether?
 3840     if recurse_set:
 3841         if "user" in recurse_set:
 3842             if user or isinstance(user, int):
 3843                 uid = __salt__["file.user_to_uid"](user)
 3844                 # file.user_to_uid returns '' if user does not exist. Above
 3845                 # check for user is not fatal, so we need to be sure user
 3846                 # exists.
 3847                 if isinstance(uid, str):
 3848                     ret["result"] = False
 3849                     ret["comment"] = (
 3850                         "Failed to enforce ownership for "
 3851                         "user {} (user does not "
 3852                         "exist)".format(user)
 3853                     )
 3854             else:
 3855                 ret["result"] = False
 3856                 ret["comment"] = (
 3857                     "user not specified, but configured as "
 3858                     "a target for recursive ownership "
 3859                     "management"
 3860                 )
 3861         else:
 3862             user = None
 3863         if "group" in recurse_set:
 3864             if group or isinstance(group, int):
 3865                 gid = __salt__["file.group_to_gid"](group)
 3866                 # As above with user, we need to make sure group exists.
 3867                 if isinstance(gid, str):
 3868                     ret["result"] = False
 3869                     ret["comment"] = (
 3870                         "Failed to enforce group ownership "
 3871                         "for group {}".format(group)
 3872                     )
 3873             else:
 3874                 ret["result"] = False
 3875                 ret["comment"] = (
 3876                     "group not specified, but configured "
 3877                     "as a target for recursive ownership "
 3878                     "management"
 3879                 )
 3880         else:
 3881             group = None
 3882 
 3883         if "mode" not in recurse_set:
 3884             file_mode = None
 3885             dir_mode = None
 3886 
 3887         if "silent" in recurse_set:
 3888             ret["changes"] = {"recursion": "Changes silenced"}
 3889 
 3890         check_files = "ignore_files" not in recurse_set
 3891         check_dirs = "ignore_dirs" not in recurse_set
 3892 
 3893         for root, dirs, files in walk_l:
 3894             if check_files:
 3895                 for fn_ in files:
 3896                     full = os.path.join(root, fn_)
 3897                     try:
 3898                         if salt.utils.platform.is_windows():
 3899                             ret = __salt__["file.check_perms"](
 3900                                 path=full,
 3901                                 ret=ret,
 3902                                 owner=win_owner,
 3903                                 grant_perms=win_perms,
 3904                                 deny_perms=win_deny_perms,
 3905                                 inheritance=win_inheritance,
 3906                                 reset=win_perms_reset,
 3907                             )
 3908                         else:
 3909                             ret, _ = __salt__["file.check_perms"](
 3910                                 full, ret, user, group, file_mode, None, follow_symlinks
 3911                             )
 3912                     except CommandExecutionError as exc:
 3913                         if not exc.strerror.startswith("Path not found"):
 3914                             errors.append(exc.strerror)
 3915 
 3916             if check_dirs:
 3917                 for dir_ in dirs:
 3918                     full = os.path.join(root, dir_)
 3919                     try:
 3920                         if salt.utils.platform.is_windows():
 3921                             ret = __salt__["file.check_perms"](
 3922                                 path=full,
 3923                                 ret=ret,
 3924                                 owner=win_owner,
 3925                                 grant_perms=win_perms,
 3926                                 deny_perms=win_deny_perms,
 3927                                 inheritance=win_inheritance,
 3928                                 reset=win_perms_reset,
 3929                             )
 3930                         else:
 3931                             ret, _ = __salt__["file.check_perms"](
 3932                                 full, ret, user, group, dir_mode, None, follow_symlinks
 3933                             )
 3934                     except CommandExecutionError as exc:
 3935                         if not exc.strerror.startswith("Path not found"):
 3936                             errors.append(exc.strerror)
 3937 
 3938     if clean:
 3939         keep = _gen_keep_files(name, require, walk_d)
 3940         log.debug("List of kept files when use file.directory with clean: %s", keep)
 3941         removed = _clean_dir(name, list(keep), exclude_pat)
 3942         if removed:
 3943             ret["changes"]["removed"] = removed
 3944             ret["comment"] = "Files cleaned from directory {}".format(name)
 3945 
 3946     # issue 32707: reflect children_only selection in comments
 3947     if not ret["comment"]:
 3948         if children_only:
 3949             ret["comment"] = "Directory {}/* updated".format(name)
 3950         else:
 3951             if ret["changes"]:
 3952                 ret["comment"] = "Directory {} updated".format(name)
 3953 
 3954     if __opts__["test"]:
 3955         ret["comment"] = "Directory {} not updated".format(name)
 3956     elif not ret["changes"] and ret["result"]:
 3957         orig_comment = None
 3958         if ret["comment"]:
 3959             orig_comment = ret["comment"]
 3960 
 3961         ret["comment"] = "Directory {} is in the correct state".format(name)
 3962         if orig_comment:
 3963             ret["comment"] = "\n".join([ret["comment"], orig_comment])
 3964 
 3965     if errors:
 3966         ret["result"] = False
 3967         ret["comment"] += "\n\nThe following errors were encountered:\n"
 3968         for error in errors:
 3969             ret["comment"] += "\n- {}".format(error)
 3970 
 3971     return ret
 3972 
 3973 
 3974 def recurse(
 3975     name,
 3976     source,
 3977     keep_source=True,
 3978     clean=False,
 3979     require=None,
 3980     user=None,
 3981     group=None,
 3982     dir_mode=None,
 3983     file_mode=None,
 3984     sym_mode=None,
 3985     template=None,
 3986     context=None,
 3987     replace=True,
 3988     defaults=None,
 3989     include_empty=False,
 3990     backup="",
 3991     include_pat=None,
 3992     exclude_pat=None,
 3993     maxdepth=None,
 3994     keep_symlinks=False,
 3995     force_symlinks=False,
 3996     win_owner=None,
 3997     win_perms=None,
 3998     win_deny_perms=None,
 3999     win_inheritance=True,
 4000     **kwargs
 4001 ):
 4002     """
 4003     Recurse through a subdirectory on the master and copy said subdirectory
 4004     over to the specified path.
 4005 
 4006     name
 4007         The directory to set the recursion in
 4008 
 4009     source
 4010         The source directory, this directory is located on the salt master file
 4011         server and is specified with the salt:// protocol. If the directory is
 4012         located on the master in the directory named spam, and is called eggs,
 4013         the source string is salt://spam/eggs
 4014 
 4015     keep_source : True
 4016         Set to ``False`` to discard the cached copy of the source file once the
 4017         state completes. This can be useful for larger files to keep them from
 4018         taking up space in minion cache. However, keep in mind that discarding
 4019         the source file will result in the state needing to re-download the
 4020         source file if the state is run again.
 4021 
 4022         .. versionadded:: 2017.7.3
 4023 
 4024     clean
 4025         Make sure that only files that are set up by salt and required by this
 4026         function are kept. If this option is set then everything in this
 4027         directory will be deleted unless it is required.
 4028 
 4029     require
 4030         Require other resources such as packages or files
 4031 
 4032     user
 4033         The user to own the directory. This defaults to the user salt is
 4034         running as on the minion
 4035 
 4036     group
 4037         The group ownership set for the directory. This defaults to the group
 4038         salt is running as on the minion. On Windows, this is ignored
 4039 
 4040     dir_mode
 4041         The permissions mode to set on any directories created.
 4042 
 4043         The default mode for new files and directories corresponds umask of salt
 4044         process. For existing files and directories it's not enforced.
 4045 
 4046         .. note::
 4047             This option is **not** supported on Windows.
 4048 
 4049     file_mode
 4050         The permissions mode to set on any files created.
 4051 
 4052         The default mode for new files and directories corresponds umask of salt
 4053         process. For existing files and directories it's not enforced.
 4054 
 4055         .. note::
 4056             This option is **not** supported on Windows.
 4057 
 4058         .. versionchanged:: 2016.11.0
 4059             This option can be set to ``keep``, and Salt will keep the mode
 4060             from the Salt fileserver. This is only supported when the
 4061             ``source`` URL begins with ``salt://``, or for files local to the
 4062             minion. Because the ``source`` option cannot be used with any of
 4063             the ``contents`` options, setting the ``mode`` to ``keep`` is also
 4064             incompatible with the ``contents`` options.
 4065 
 4066     sym_mode
 4067         The permissions mode to set on any symlink created.
 4068 
 4069         The default mode for new files and directories corresponds umask of salt
 4070         process. For existing files and directories it's not enforced.
 4071 
 4072         .. note::
 4073             This option is **not** supported on Windows.
 4074 
 4075     template
 4076         If this setting is applied, the named templating engine will be used to
 4077         render the downloaded file. The following templates are supported:
 4078 
 4079         - :mod:`cheetah<salt.renderers.cheetah>`
 4080         - :mod:`genshi<salt.renderers.genshi>`
 4081         - :mod:`jinja<salt.renderers.jinja>`
 4082         - :mod:`mako<salt.renderers.mako>`
 4083         - :mod:`py<salt.renderers.py>`
 4084         - :mod:`wempy<salt.renderers.wempy>`
 4085 
 4086         .. note::
 4087 
 4088             The template option is required when recursively applying templates.
 4089 
 4090     replace : True
 4091         If set to ``False`` and the file already exists, the file will not be
 4092         modified even if changes would otherwise be made. Permissions and
 4093         ownership will still be enforced, however.
 4094 
 4095     context
 4096         Overrides default context variables passed to the template.
 4097 
 4098     defaults
 4099         Default context passed to the template.
 4100 
 4101     include_empty
 4102         Set this to True if empty directories should also be created
 4103         (default is False)
 4104 
 4105     backup
 4106         Overrides the default backup mode for all replaced files. See
 4107         :ref:`backup_mode documentation <file-state-backups>` for more details.
 4108 
 4109     include_pat
 4110         When copying, include only this pattern, or list of patterns, from the
 4111         source. Default is glob match; if prefixed with 'E@', then regexp match.
 4112         Example:
 4113 
 4114         .. code-block:: text
 4115 
 4116           - include_pat: hello*       :: glob matches 'hello01', 'hello02'
 4117                                          ... but not 'otherhello'
 4118           - include_pat: E@hello      :: regexp matches 'otherhello',
 4119                                          'hello01' ...
 4120 
 4121         .. versionchanged:: 3001
 4122 
 4123             List patterns are now supported
 4124 
 4125         .. code-block:: text
 4126 
 4127             - include_pat:
 4128                 - hello01
 4129                 - hello02
 4130 
 4131     exclude_pat
 4132         Exclude this pattern, or list of patterns, from the source when copying.
 4133         If both `include_pat` and `exclude_pat` are supplied, then it will apply
 4134         conditions cumulatively. i.e. first select based on include_pat, and
 4135         then within that result apply exclude_pat.
 4136 
 4137         Also, when 'clean=True', exclude this pattern from the removal
 4138         list and preserve in the destination.
 4139         Example:
 4140 
 4141         .. code-block:: text
 4142 
 4143           - exclude_pat: APPDATA*               :: glob matches APPDATA.01,
 4144                                                    APPDATA.02,.. for exclusion
 4145           - exclude_pat: E@(APPDATA)|(TEMPDATA) :: regexp matches APPDATA
 4146                                                    or TEMPDATA for exclusion
 4147 
 4148         .. versionchanged:: 3001
 4149 
 4150             List patterns are now supported
 4151 
 4152         .. code-block:: text
 4153 
 4154             - exclude_pat:
 4155                 - APPDATA.01
 4156                 - APPDATA.02
 4157 
 4158     maxdepth
 4159         When copying, only copy paths which are of depth `maxdepth` from the
 4160         source path.
 4161         Example:
 4162 
 4163         .. code-block:: text
 4164 
 4165           - maxdepth: 0      :: Only include files located in the source
 4166                                 directory
 4167           - maxdepth: 1      :: Only include files located in the source
 4168                                 or immediate subdirectories
 4169 
 4170     keep_symlinks
 4171         Keep symlinks when copying from the source. This option will cause
 4172         the copy operation to terminate at the symlink. If desire behavior
 4173         similar to rsync, then set this to True.
 4174 
 4175     force_symlinks
 4176         Force symlink creation. This option will force the symlink creation.
 4177         If a file or directory is obstructing symlink creation it will be
 4178         recursively removed so that symlink creation can proceed. This
 4179         option is usually not needed except in special circumstances.
 4180 
 4181     win_owner : None
 4182         The owner of the symlink and directories if ``makedirs`` is True. If
 4183         this is not passed, ``user`` will be used. If ``user`` is not passed,
 4184         the account under which Salt is running will be used.
 4185 
 4186         .. versionadded:: 2017.7.7
 4187 
 4188     win_perms : None
 4189         A dictionary containing permissions to grant
 4190 
 4191         .. versionadded:: 2017.7.7
 4192 
 4193     win_deny_perms : None
 4194         A dictionary containing permissions to deny
 4195 
 4196         .. versionadded:: 2017.7.7
 4197 
 4198     win_inheritance : None
 4199         True to inherit permissions from parent, otherwise False
 4200 
 4201         .. versionadded:: 2017.7.7
 4202 
 4203     """
 4204     if "env" in kwargs:
 4205         # "env" is not supported; Use "saltenv".
 4206         kwargs.pop("env")
 4207 
 4208     name = os.path.expanduser(salt.utils.data.decode(name))
 4209 
 4210     user = _test_owner(kwargs, user=user)
 4211     if salt.utils.platform.is_windows():
 4212         if group is not None:
 4213             log.warning(
 4214                 "The group argument for {} has been ignored as this "
 4215                 "is a Windows system.".format(name)
 4216             )
 4217         group = user
 4218     ret = {
 4219         "name": name,
 4220         "changes": {},
 4221         "result": True,
 4222         "comment": {},  # { path: [comment, ...] }
 4223     }
 4224 
 4225     if "mode" in kwargs:
 4226         ret["result"] = False
 4227         ret["comment"] = (
 4228             "'mode' is not allowed in 'file.recurse'. Please use "
 4229             "'file_mode' and 'dir_mode'."
 4230         )
 4231         return ret
 4232 
 4233     if (
 4234         any([x is not None for x in (dir_mode, file_mode, sym_mode)])
 4235         and salt.utils.platform.is_windows()
 4236     ):
 4237         return _error(ret, "mode management is not supported on Windows")
 4238 
 4239     # Make sure that leading zeros stripped by YAML loader are added back
 4240     dir_mode = salt.utils.files.normalize_mode(dir_mode)
 4241 
 4242     try:
 4243         keep_mode = file_mode.lower() == "keep"
 4244         if keep_mode:
 4245             # We're not hard-coding the mode, so set it to None
 4246             file_mode = None
 4247     except AttributeError:
 4248         keep_mode = False
 4249 
 4250     file_mode = salt.utils.files.normalize_mode(file_mode)
 4251 
 4252     u_check = _check_user(user, group)
 4253     if u_check:
 4254         # The specified user or group do not exist
 4255         return _error(ret, u_check)
 4256     if not os.path.isabs(name):
 4257         return _error(ret, "Specified file {} is not an absolute path".format(name))
 4258 
 4259     # expand source into source_list
 4260     source_list = _validate_str_list(source)
 4261 
 4262     for idx, val in enumerate(source_list):
 4263         source_list[idx] = val.rstrip("/")
 4264 
 4265     for precheck in source_list:
 4266         if not precheck.startswith("salt://"):
 4267             return _error(
 4268                 ret,
 4269                 ("Invalid source '{}' " "(must be a salt:// URI)".format(precheck)),
 4270             )
 4271 
 4272     # Select the first source in source_list that exists
 4273     try:
 4274         source, source_hash = __salt__["file.source_list"](source_list, "", __env__)
 4275     except CommandExecutionError as exc:
 4276         ret["result"] = False
 4277         ret["comment"] = "Recurse failed: {}".format(exc)
 4278         return ret
 4279 
 4280     # Check source path relative to fileserver root, make sure it is a
 4281     # directory
 4282     srcpath, senv = salt.utils.url.parse(source)
 4283     if senv is None:
 4284         senv = __env__
 4285     master_dirs = __salt__["cp.list_master_dirs"](saltenv=senv)
 4286     if srcpath not in master_dirs and not any(
 4287         x for x in master_dirs if x.startswith(srcpath + "/")
 4288     ):
 4289         ret["result"] = False
 4290         ret["comment"] = (
 4291             "The directory '{}' does not exist on the salt fileserver "
 4292             "in saltenv '{}'".format(srcpath, senv)
 4293         )
 4294         return ret
 4295 
 4296     # Verify the target directory
 4297     if not os.path.isdir(name):
 4298         if os.path.exists(name):
 4299             # it is not a dir, but it exists - fail out
 4300             return _error(ret, "The path {} exists and is not a directory".format(name))
 4301         if not __opts__["test"]:
 4302             if salt.utils.platform.is_windows():
 4303                 win_owner = win_owner if win_owner else user
 4304                 __salt__["file.makedirs_perms"](
 4305                     path=name,
 4306                     owner=win_owner,
 4307                     grant_perms=win_perms,
 4308                     deny_perms=win_deny_perms,
 4309                     inheritance=win_inheritance,
 4310                 )
 4311             else:
 4312                 __salt__["file.makedirs_perms"](
 4313                     name=name, user=user, group=group, mode=dir_mode
 4314                 )
 4315 
 4316     def add_comment(path, comment):
 4317         comments = ret["comment"].setdefault(path, [])
 4318         if isinstance(comment, str):
 4319             comments.append(comment)
 4320         else:
 4321             comments.extend(comment)
 4322 
 4323     def merge_ret(path, _ret):
 4324         # Use the most "negative" result code (out of True, None, False)
 4325         if _ret["result"] is False or ret["result"] is True:
 4326             ret["result"] = _ret["result"]
 4327 
 4328         # Only include comments about files that changed
 4329         if _ret["result"] is not True and _ret["comment"]:
 4330             add_comment(path, _ret["comment"])
 4331 
 4332         if _ret["changes"]:
 4333             ret["changes"][path] = _ret["changes"]
 4334 
 4335     def manage_file(path, source, replace):
 4336         if clean and os.path.exists(path) and os.path.isdir(path) and replace:
 4337             _ret = {"name": name, "changes": {}, "result": True, "comment": ""}
 4338             if __opts__["test"]:
 4339                 _ret["comment"] = "Replacing directory {} with a " "file".format(path)
 4340                 _ret["result"] = None
 4341                 merge_ret(path, _ret)
 4342                 return
 4343             else:
 4344                 __salt__["file.remove"](path)
 4345                 _ret["changes"] = {"diff": "Replaced directory with a " "new file"}
 4346                 merge_ret(path, _ret)
 4347 
 4348         # Conflicts can occur if some kwargs are passed in here
 4349         pass_kwargs = {}
 4350         faults = ["mode", "makedirs"]
 4351         for key in kwargs:
 4352             if key not in faults:
 4353                 pass_kwargs[key] = kwargs[key]
 4354 
 4355         _ret = managed(
 4356             path,
 4357             source=source,
 4358             keep_source=keep_source,
 4359             user=user,
 4360             group=group,
 4361             mode="keep" if keep_mode else file_mode,
 4362             attrs=None,
 4363             template=template,
 4364             makedirs=True,
 4365             replace=replace,
 4366             context=context,
 4367             defaults=defaults,
 4368             backup=backup,
 4369             **pass_kwargs
 4370         )
 4371         merge_ret(path, _ret)
 4372 
 4373     def manage_directory(path):
 4374         if os.path.basename(path) == "..":
 4375             return
 4376         if clean and os.path.exists(path) and not os.path.isdir(path):
 4377             _ret = {"name": name, "changes": {}, "result": True, "comment": ""}
 4378             if __opts__["test"]:
 4379                 _ret["comment"] = "Replacing {} with a directory".format(path)
 4380                 _ret["result"] = None
 4381                 merge_ret(path, _ret)
 4382                 return
 4383             else:
 4384                 __salt__["file.remove"](path)
 4385                 _ret["changes"] = {"diff": "Replaced file with a directory"}
 4386                 merge_ret(path, _ret)
 4387 
 4388         _ret = directory(
 4389             path,
 4390             user=user,
 4391             group=group,
 4392             recurse=[],
 4393             dir_mode=dir_mode,
 4394             file_mode=None,
 4395             makedirs=True,
 4396             clean=False,
 4397             require=None,
 4398         )
 4399         merge_ret(path, _ret)
 4400 
 4401     mng_files, mng_dirs, mng_symlinks, keep = _gen_recurse_managed_files(
 4402         name, source, keep_symlinks, include_pat, exclude_pat, maxdepth, include_empty
 4403     )
 4404 
 4405     for srelpath, ltarget in mng_symlinks:
 4406         _ret = symlink(
 4407             os.path.join(name, srelpath),
 4408             ltarget,
 4409             makedirs=True,
 4410             force=force_symlinks,
 4411             user=user,
 4412             group=group,
 4413             mode=sym_mode,
 4414         )
 4415         if not _ret:
 4416             continue
 4417         merge_ret(os.path.join(name, srelpath), _ret)
 4418     for dirname in mng_dirs:
 4419         manage_directory(dirname)
 4420     for dest, src in mng_files:
 4421         manage_file(dest, src, replace)
 4422 
 4423     if clean:
 4424         # TODO: Use directory(clean=True) instead
 4425         keep.update(_gen_keep_files(name, require))
 4426         removed = _clean_dir(name, list(keep), exclude_pat)
 4427         if removed:
 4428             if __opts__["test"]:
 4429                 if ret["result"]:
 4430                     ret["result"] = None
 4431                 add_comment("removed", removed)
 4432             else:
 4433                 ret["changes"]["removed"] = removed
 4434 
 4435     # Flatten comments until salt command line client learns
 4436     # to display structured comments in a readable fashion
 4437     ret["comment"] = "\n".join(
 4438         "\n#### {} ####\n{}".format(k, v if isinstance(v, str) else "\n".join(v))
 4439         for (k, v) in ret["comment"].items()
 4440     ).strip()
 4441 
 4442     if not ret["comment"]:
 4443         ret["comment"] = "Recursively updated {}".format(name)
 4444 
 4445     if not ret["changes"] and ret["result"]:
 4446         ret["comment"] = "The directory {} is in the correct state".format(name)
 4447 
 4448     return ret
 4449 
 4450 
 4451 def retention_schedule(name, retain, strptime_format=None, timezone=None):
 4452     """
 4453     Apply retention scheduling to backup storage directory.
 4454 
 4455     .. versionadded:: 2016.11.0
 4456 
 4457     :param name:
 4458         The filesystem path to the directory containing backups to be managed.
 4459 
 4460     :param retain:
 4461         Delete the backups, except for the ones we want to keep.
 4462         The N below should be an integer but may also be the special value of ``all``,
 4463         which keeps all files matching the criteria.
 4464         All of the retain options default to None,
 4465         which means to not keep files based on this criteria.
 4466 
 4467         :most_recent N:
 4468             Keep the most recent N files.
 4469 
 4470         :first_of_hour N:
 4471             For the last N hours from now, keep the first file after the hour.
 4472 
 4473         :first_of_day N:
 4474             For the last N days from now, keep the first file after midnight.
 4475             See also ``timezone``.
 4476 
 4477         :first_of_week N:
 4478             For the last N weeks from now, keep the first file after Sunday midnight.
 4479 
 4480         :first_of_month N:
 4481             For the last N months from now, keep the first file after the start of the month.
 4482 
 4483         :first_of_year N:
 4484             For the last N years from now, keep the first file after the start of the year.
 4485 
 4486     :param strptime_format:
 4487         A python strptime format string used to first match the filenames of backups
 4488         and then parse the filename to determine the datetime of the file.
 4489         https://docs.python.org/2/library/datetime.html#datetime.datetime.strptime
 4490         Defaults to None, which considers all files in the directory to be backups eligible for deletion
 4491         and uses ``os.path.getmtime()`` to determine the datetime.
 4492 
 4493     :param timezone:
 4494         The timezone to use when determining midnight.
 4495         This is only used when datetime is pulled from ``os.path.getmtime()``.
 4496         Defaults to ``None`` which uses the timezone from the locale.
 4497 
 4498     Usage example:
 4499 
 4500     .. code-block:: yaml
 4501 
 4502         /var/backups/example_directory:
 4503           file.retention_schedule:
 4504             - retain:
 4505                 most_recent: 5
 4506                 first_of_hour: 4
 4507                 first_of_day: 7
 4508                 first_of_week: 6    # NotImplemented yet.
 4509                 first_of_month: 6
 4510                 first_of_year: all
 4511             - strptime_format: example_name_%Y%m%dT%H%M%S.tar.bz2
 4512             - timezone: None
 4513 
 4514     """
 4515     name = os.path.expanduser(name)
 4516     ret = {
 4517         "name": name,
 4518         "changes": {"retained": [], "deleted": [], "ignored": []},
 4519         "result": True,
 4520         "comment": "",
 4521     }
 4522     if not name:
 4523         return _error(ret, "Must provide name to file.retention_schedule")
 4524     if not os.path.isdir(name):
 4525         return _error(ret, "Name provided to file.retention must be a directory")
 4526 
 4527     # get list of files in directory
 4528     all_files = __salt__["file.readdir"](name)
 4529 
 4530     # if strptime_format is set, filter through the list to find names which parse and get their datetimes.
 4531     beginning_of_unix_time = datetime(1970, 1, 1)
 4532 
 4533     def get_file_time_from_strptime(f):
 4534         try:
 4535             ts = datetime.strptime(f, strptime_format)
 4536             ts_epoch = salt.utils.dateutils.total_seconds(ts - beginning_of_unix_time)
 4537             return (ts, ts_epoch)
 4538         except ValueError:
 4539             # Files which don't match the pattern are not relevant files.
 4540             return (None, None)
 4541 
 4542     def get_file_time_from_mtime(f):
 4543         if f == "." or f == "..":
 4544             return (None, None)
 4545         lstat = __salt__["file.lstat"](os.path.join(name, f))
 4546         if lstat:
 4547             mtime = lstat["st_mtime"]
 4548             return (datetime.fromtimestamp(mtime, timezone), mtime)
 4549         else:  # maybe it was deleted since we did the readdir?
 4550             return (None, None)
 4551 
 4552     get_file_time = (
 4553         get_file_time_from_strptime if strptime_format else get_file_time_from_mtime
 4554     )
 4555 
 4556     # data structures are nested dicts:
 4557     # files_by_ymd = year.month.day.hour.unixtime: filename
 4558     # files_by_y_week_dow = year.week_of_year.day_of_week.unixtime: filename
 4559     # http://the.randomengineer.com/2015/04/28/python-recursive-defaultdict/
 4560     # TODO: move to an ordered dict model and reduce the number of sorts in the rest of the code?
 4561     def dict_maker():
 4562         return defaultdict(dict_maker)
 4563 
 4564     files_by_ymd = dict_maker()
 4565     files_by_y_week_dow = dict_maker()
 4566     relevant_files = set()
 4567     ignored_files = set()
 4568     for f in all_files:
 4569         ts, ts_epoch = get_file_time(f)
 4570         if ts:
 4571             files_by_ymd[ts.year][ts.month][ts.day][ts.hour][ts_epoch] = f
 4572             week_of_year = ts.isocalendar()[1]
 4573             files_by_y_week_dow[ts.year][week_of_year][ts.weekday()][ts_epoch] = f
 4574             relevant_files.add(f)
 4575         else:
 4576             ignored_files.add(f)
 4577 
 4578     # This is tightly coupled with the file_with_times data-structure above.
 4579     RETAIN_TO_DEPTH = {
 4580         "first_of_year": 1,
 4581         "first_of_month": 2,
 4582         "first_of_day": 3,
 4583         "first_of_hour": 4,
 4584         "most_recent": 5,
 4585     }
 4586 
 4587     def get_first(fwt):
 4588         if isinstance(fwt, dict):
 4589             first_sub_key = sorted(fwt.keys())[0]
 4590             return get_first(fwt[first_sub_key])
 4591         else:
 4592             return {fwt}
 4593 
 4594     def get_first_n_at_depth(fwt, depth, n):
 4595         if depth <= 0:
 4596             return get_first(fwt)
 4597         else:
 4598             result_set = set()
 4599             for k in sorted(fwt.keys(), reverse=True):
 4600                 needed = n - len(result_set)
 4601                 if needed < 1:
 4602                     break
 4603                 result_set |= get_first_n_at_depth(fwt[k], depth - 1, needed)
 4604             return result_set
 4605 
 4606     # for each retain criteria, add filenames which match the criteria to the retain set.
 4607     retained_files = set()
 4608     for retention_rule, keep_count in retain.items():
 4609         # This is kind of a hack, since 'all' should really mean all,
 4610         # but I think it's a large enough number that even modern filesystems would
 4611         # choke if they had this many files in a single directory.
 4612         keep_count = sys.maxsize if "all" == keep_count else int(keep_count)
 4613         if "first_of_week" == retention_rule:
 4614             first_of_week_depth = 2  # year + week_of_year = 2
 4615             # I'm adding 1 to keep_count below because it fixed an off-by one
 4616             # issue in the tests. I don't understand why, and that bothers me.
 4617             retained_files |= get_first_n_at_depth(
 4618                 files_by_y_week_dow, first_of_week_depth, keep_count + 1
 4619             )
 4620         else:
 4621             retained_files |= get_first_n_at_depth(
 4622                 files_by_ymd, RETAIN_TO_DEPTH[retention_rule], keep_count
 4623             )
 4624 
 4625     deletable_files = list(relevant_files - retained_files)
 4626     deletable_files.sort(reverse=True)
 4627     changes = {
 4628         "retained": sorted(list(retained_files), reverse=True),
 4629         "deleted": deletable_files,
 4630         "ignored": sorted(list(ignored_files), reverse=True),
 4631     }
 4632     ret["changes"] = changes
 4633 
 4634     # TODO: track and report how much space was / would be reclaimed
 4635     if __opts__["test"]:
 4636         ret["comment"] = "{} backups would have been removed from {}.\n".format(
 4637             len(deletable_files), name
 4638         )
 4639         if deletable_files:
 4640             ret["result"] = None
 4641     else:
 4642         for f in deletable_files:
 4643             __salt__["file.remove"](os.path.join(name, f))
 4644         ret["comment"] = "{} backups were removed from {}.\n".format(
 4645             len(deletable_files), name
 4646         )
 4647         ret["changes"] = changes
 4648 
 4649     return ret
 4650 
 4651 
 4652 def line(
 4653     name,
 4654     content=None,
 4655     match=None,
 4656     mode=None,
 4657     location=None,
 4658     before=None,
 4659     after=None,
 4660     show_changes=True,
 4661     backup=False,
 4662     quiet=False,
 4663     indent=True,
 4664     create=False,
 4665     user=None,
 4666     group=None,
 4667     file_mode=None,
 4668 ):
 4669     """
 4670     Line-focused editing of a file.
 4671 
 4672     .. versionadded:: 2015.8.0
 4673 
 4674     .. note::
 4675 
 4676         ``file.line`` exists for historic reasons, and is not
 4677         generally recommended. It has a lot of quirks.  You may find
 4678         ``file.replace`` to be more suitable.
 4679 
 4680     ``file.line`` is most useful if you have single lines in a file,
 4681     potentially a config file, that you would like to manage. It can
 4682     remove, add, and replace lines.
 4683 
 4684     name
 4685         Filesystem path to the file to be edited.
 4686 
 4687     content
 4688         Content of the line. Allowed to be empty if mode=delete.
 4689 
 4690     match
 4691         Match the target line for an action by
 4692         a fragment of a string or regular expression.
 4693 
 4694         If neither ``before`` nor ``after`` are provided, and ``match``
 4695         is also ``None``, match falls back to the ``content`` value.
 4696 
 4697     mode
 4698         Defines how to edit a line. One of the following options is
 4699         required:
 4700 
 4701         - ensure
 4702             If line does not exist, it will be added. If ``before``
 4703             and ``after`` are specified either zero lines, or lines
 4704             that contain the ``content`` line are allowed to be in between
 4705             ``before`` and ``after``. If there are lines, and none of
 4706             them match then it will produce an error.
 4707         - replace
 4708             If line already exists, it will be replaced.
 4709         - delete
 4710             Delete the line, if found.
 4711         - insert
 4712             Nearly identical to ``ensure``. If a line does not exist,
 4713             it will be added.
 4714 
 4715             The differences are that multiple (and non-matching) lines are
 4716             alloweed between ``before`` and ``after``, if they are
 4717             specified. The line will always be inserted right before
 4718             ``before``. ``insert`` also allows the use of ``location`` to
 4719             specify that the line should be added at the beginning or end of
 4720             the file.
 4721 
 4722         .. note::
 4723 
 4724             If ``mode=insert`` is used, at least one of the following
 4725             options must also be defined: ``location``, ``before``, or
 4726             ``after``. If ``location`` is used, it takes precedence
 4727             over the other two options.
 4728 
 4729     location
 4730         In ``mode=insert`` only, whether to place the ``content`` at the
 4731         beginning or end of a the file. If ``location`` is provided,
 4732         ``before`` and ``after`` are ignored. Valid locations:
 4733 
 4734         - start
 4735             Place the content at the beginning of the file.
 4736         - end
 4737             Place the content at the end of the file.
 4738 
 4739     before
 4740         Regular expression or an exact case-sensitive fragment of the string.
 4741         Will be tried as **both** a regex **and** a part of the line.  Must
 4742         match **exactly** one line in the file.  This value is only used in
 4743         ``ensure`` and ``insert`` modes. The ``content`` will be inserted just
 4744         before this line, matching its ``indent`` unless ``indent=False``.
 4745 
 4746     after
 4747         Regular expression or an exact case-sensitive fragment of the string.
 4748         Will be tried as **both** a regex **and** a part of the line.  Must
 4749         match **exactly** one line in the file.  This value is only used in
 4750         ``ensure`` and ``insert`` modes. The ``content`` will be inserted
 4751         directly after this line, unless ``before`` is also provided. If
 4752         ``before`` is not matched, indentation will match this line, unless
 4753         ``indent=False``.
 4754 
 4755     show_changes
 4756         Output a unified diff of the old file and the new file.
 4757         If ``False`` return a boolean if any changes were made.
 4758         Default is ``True``
 4759 
 4760         .. note::
 4761             Using this option will store two copies of the file in-memory
 4762             (the original version and the edited version) in order to generate the diff.
 4763 
 4764     backup
 4765         Create a backup of the original file with the extension:
 4766         "Year-Month-Day-Hour-Minutes-Seconds".
 4767 
 4768     quiet
 4769         Do not raise any exceptions. E.g. ignore the fact that the file that is
 4770         tried to be edited does not exist and nothing really happened.
 4771 
 4772     indent
 4773         Keep indentation with the previous line. This option is not considered when
 4774         the ``delete`` mode is specified. Default is ``True``.
 4775 
 4776     create
 4777         Create an empty file if doesn't exist.
 4778 
 4779         .. versionadded:: 2016.11.0
 4780 
 4781     user
 4782         The user to own the file, this defaults to the user salt is running as
 4783         on the minion.
 4784 
 4785         .. versionadded:: 2016.11.0
 4786 
 4787     group
 4788         The group ownership set for the file, this defaults to the group salt
 4789         is running as on the minion On Windows, this is ignored.
 4790 
 4791         .. versionadded:: 2016.11.0
 4792 
 4793     file_mode
 4794         The permissions to set on this file, aka 644, 0775, 4664. Not supported
 4795         on Windows.
 4796 
 4797         .. versionadded:: 2016.11.0
 4798 
 4799     If an equal sign (``=``) appears in an argument to a Salt command, it is
 4800     interpreted as a keyword argument in the format of ``key=val``. That
 4801     processing can be bypassed in order to pass an equal sign through to the
 4802     remote shell command by manually specifying the kwarg:
 4803 
 4804     .. code-block:: yaml
 4805 
 4806        update_config:
 4807          file.line:
 4808            - name: /etc/myconfig.conf
 4809            - mode: ensure
 4810            - content: my key = my value
 4811            - before: somekey.*?
 4812 
 4813 
 4814     **Examples:**
 4815 
 4816     Here's a simple config file.
 4817 
 4818     .. code-block:: ini
 4819 
 4820         [some_config]
 4821         # Some config file
 4822         # this line will go away
 4823 
 4824         here=False
 4825         away=True
 4826         goodybe=away
 4827 
 4828     And an sls file:
 4829 
 4830     .. code-block:: yaml
 4831 
 4832         remove_lines:
 4833           file.line:
 4834             - name: /some/file.conf
 4835             - mode: delete
 4836             - match: away
 4837 
 4838     This will produce:
 4839 
 4840     .. code-block:: ini
 4841 
 4842         [some_config]
 4843         # Some config file
 4844 
 4845         here=False
 4846         away=True
 4847         goodbye=away
 4848 
 4849     If that state is executed 2 more times, this will be the result:
 4850 
 4851     .. code-block:: ini
 4852 
 4853         [some_config]
 4854         # Some config file
 4855 
 4856         here=False
 4857 
 4858     Given that original file with this state:
 4859 
 4860     .. code-block:: yaml
 4861 
 4862         replace_things:
 4863           file.line:
 4864             - name: /some/file.conf
 4865             - mode: replace
 4866             - match: away
 4867             - content: here
 4868 
 4869     Three passes will this state will result in this file:
 4870 
 4871     .. code-block:: ini
 4872 
 4873         [some_config]
 4874         # Some config file
 4875         here
 4876 
 4877         here=False
 4878         here
 4879         here
 4880 
 4881     Each pass replacing the first line found.
 4882 
 4883     Given this file:
 4884 
 4885     .. code-block:: text
 4886 
 4887         insert after me
 4888         something
 4889         insert before me
 4890 
 4891     The following state:
 4892 
 4893     .. code-block:: yaml
 4894 
 4895         insert_a_line:
 4896           file.line:
 4897             - name: /some/file.txt
 4898             - mode: insert
 4899             - after: insert after me
 4900             - before: insert before me
 4901             - content: thrice
 4902 
 4903     If this state is executed 3 times, the result will be:
 4904 
 4905     .. code-block:: text
 4906 
 4907         insert after me
 4908         something
 4909         thrice
 4910         thrice
 4911         thrice
 4912         insert before me
 4913 
 4914     If the mode is ensure instead, it will fail each time. To succeed, we need
 4915     to remove the incorrect line between before and after:
 4916 
 4917     .. code-block:: text
 4918 
 4919         insert after me
 4920         insert before me
 4921 
 4922     With an ensure mode, this will insert ``thrice`` the first time and
 4923     make no changes for subsequent calls. For something simple this is
 4924     fine, but if you have instead blocks like this:
 4925 
 4926     .. code-block:: text
 4927 
 4928         Begin SomeBlock
 4929             foo = bar
 4930         End
 4931 
 4932         Begin AnotherBlock
 4933             another = value
 4934         End
 4935 
 4936     And given this state:
 4937 
 4938     .. code-block:: yaml
 4939 
 4940         ensure_someblock:
 4941           file.line:
 4942             - name: /some/file.conf
 4943             - mode: ensure
 4944             - after: Begin SomeBlock
 4945             - content: this = should be my content
 4946             - before: End
 4947 
 4948     This will fail because there are multiple ``End`` lines. Without that
 4949     problem, it still would fail because there is a non-matching line,
 4950     ``foo = bar``. Ensure **only** allows either zero, or the matching
 4951     line present to be present in between ``before`` and ``after``.
 4952     """
 4953     name = os.path.expanduser(name)
 4954     ret = {"name": name, "changes": {}, "result": True, "comment": ""}
 4955     if not name:
 4956         return _error(ret, "Must provide name to file.line")
 4957 
 4958     managed(name, create=create, user=user, group=group, mode=file_mode, replace=False)
 4959 
 4960     check_res, check_msg = _check_file(name)
 4961     if not check_res:
 4962         return _error(ret, check_msg)
 4963 
 4964     # We've set the content to be empty in the function params but we want to make sure
 4965     # it gets passed when needed. Feature #37092
 4966     mode = mode and mode.lower() or mode
 4967     if mode is None:
 4968         return _error(ret, "Mode was not defined. How to process the file?")
 4969 
 4970     modeswithemptycontent = ["delete"]
 4971     if mode not in modeswithemptycontent and content is None:
 4972         return _error(
 4973             ret,
 4974             "Content can only be empty if mode is {}".format(modeswithemptycontent),
 4975         )
 4976     del modeswithemptycontent
 4977 
 4978     changes = __salt__["file.line"](
 4979         name,
 4980         content,
 4981         match=match,
 4982         mode=mode,
 4983         location=location,
 4984         before=before,
 4985         after=after,
 4986         show_changes=show_changes,
 4987         backup=backup,
 4988         quiet=quiet,
 4989         indent=indent,
 4990     )
 4991     if changes:
 4992         ret["changes"]["diff"] = changes
 4993         if __opts__["test"]:
 4994             ret["result"] = None
 4995             ret["comment"] = "Changes would be made"
 4996         else:
 4997             ret["result"] = True
 4998             ret["comment"] = "Changes were made"
 4999     else:
 5000         ret["result"] = True
 5001         ret["comment"] = "No changes needed to be made"
 5002 
 5003     return ret
 5004 
 5005 
 5006 def replace(
 5007     name,
 5008     pattern,
 5009     repl,
 5010     count=0,
 5011     flags=8,
 5012     bufsize=1,
 5013     append_if_not_found=False,
 5014     prepend_if_not_found=False,
 5015     not_found_content=None,
 5016     backup=".bak",
 5017     show_changes=True,
 5018     ignore_if_missing=False,
 5019     backslash_literal=False,
 5020 ):
 5021     r"""
 5022     Maintain an edit in a file.
 5023 
 5024     .. versionadded:: 0.17.0
 5025 
 5026     name
 5027         Filesystem path to the file to be edited. If a symlink is specified, it
 5028         will be resolved to its target.
 5029 
 5030     pattern
 5031         A regular expression, to be matched using Python's
 5032         :py:func:`re.search`.
 5033 
 5034         .. note::
 5035 
 5036             If you need to match a literal string that contains regex special
 5037             characters, you may want to use salt's custom Jinja filter,
 5038             ``regex_escape``.
 5039 
 5040             .. code-block:: jinja
 5041 
 5042                 {{ 'http://example.com?foo=bar%20baz' | regex_escape }}
 5043 
 5044     repl
 5045         The replacement text
 5046 
 5047     count
 5048         Maximum number of pattern occurrences to be replaced.  Defaults to 0.
 5049         If count is a positive integer n, no more than n occurrences will be
 5050         replaced, otherwise all occurrences will be replaced.
 5051 
 5052     flags
 5053         A list of flags defined in the ``re`` module documentation from the
 5054         Python standard library. Each list item should be a string that will
 5055         correlate to the human-friendly flag name. E.g., ``['IGNORECASE',
 5056         'MULTILINE']``.  Optionally, ``flags`` may be an int, with a value
 5057         corresponding to the XOR (``|``) of all the desired flags. Defaults to
 5058         ``8`` (which equates to ``['MULTILINE']``).
 5059 
 5060         .. note::
 5061 
 5062             ``file.replace`` reads the entire file as a string to support
 5063             multiline regex patterns. Therefore, when using anchors such as
 5064             ``^`` or ``$`` in the pattern, those anchors may be relative to
 5065             the line OR relative to the file. The default for ``file.replace``
 5066             is to treat anchors as relative to the line, which is implemented
 5067             by setting the default value of ``flags`` to ``['MULTILINE']``.
 5068             When overriding the default value for ``flags``, if
 5069             ``'MULTILINE'`` is not present then anchors will be relative to
 5070             the file. If the desired behavior is for anchors to be relative to
 5071             the line, then simply add ``'MULTILINE'`` to the list of flags.
 5072 
 5073     bufsize
 5074         How much of the file to buffer into memory at once. The default value
 5075         ``1`` processes one line at a time. The special value ``file`` may be
 5076         specified which will read the entire file into memory before
 5077         processing.
 5078 
 5079     append_if_not_found : False
 5080         If set to ``True``, and pattern is not found, then the content will be
 5081         appended to the file.
 5082 
 5083         .. versionadded:: 2014.7.0
 5084 
 5085     prepend_if_not_found : False
 5086         If set to ``True`` and pattern is not found, then the content will be
 5087         prepended to the file.
 5088 
 5089         .. versionadded:: 2014.7.0
 5090 
 5091     not_found_content
 5092         Content to use for append/prepend if not found. If ``None`` (default),
 5093         uses ``repl``. Useful when ``repl`` uses references to group in
 5094         pattern.
 5095 
 5096         .. versionadded:: 2014.7.0
 5097 
 5098     backup
 5099         The file extension to use for a backup of the file before editing. Set
 5100         to ``False`` to skip making a backup.
 5101 
 5102     show_changes : True
 5103         Output a unified diff of the old file and the new file. If ``False``
 5104         return a boolean if any changes were made. Returns a boolean or a
 5105         string.
 5106 
 5107         .. note:
 5108             Using this option will store two copies of the file in memory (the
 5109             original version and the edited version) in order to generate the
 5110             diff. This may not normally be a concern, but could impact
 5111             performance if used with large files.
 5112 
 5113     ignore_if_missing : False
 5114         .. versionadded:: 2016.3.4
 5115 
 5116         Controls what to do if the file is missing. If set to ``False``, the
 5117         state will display an error raised by the execution module. If set to
 5118         ``True``, the state will simply report no changes.
 5119 
 5120     backslash_literal : False
 5121         .. versionadded:: 2016.11.7
 5122 
 5123         Interpret backslashes as literal backslashes for the repl and not
 5124         escape characters.  This will help when using append/prepend so that
 5125         the backslashes are not interpreted for the repl on the second run of
 5126         the state.
 5127 
 5128     For complex regex patterns, it can be useful to avoid the need for complex
 5129     quoting and escape sequences by making use of YAML's multiline string
 5130     syntax.
 5131 
 5132     .. code-block:: yaml
 5133 
 5134         complex_search_and_replace:
 5135           file.replace:
 5136             # <...snip...>
 5137             - pattern: |
 5138                 CentOS \(2.6.32[^\\n]+\\n\s+root[^\\n]+\\n\)+
 5139 
 5140     .. note::
 5141 
 5142        When using YAML multiline string syntax in ``pattern:``, make sure to
 5143        also use that syntax in the ``repl:`` part, or you might loose line
 5144        feeds.
 5145 
 5146     When regex capture groups are used in ``pattern:``, their captured value is
 5147     available for reuse in the ``repl:`` part as a backreference (ex. ``\1``).
 5148 
 5149     .. code-block:: yaml
 5150 
 5151         add_login_group_to_winbind_ssh_access_list:
 5152           file.replace:
 5153             - name: '/etc/security/pam_winbind.conf'
 5154             - pattern: '^(require_membership_of = )(.*)$'
 5155             - repl: '\1\2,append-new-group-to-line'
 5156 
 5157     .. note::
 5158 
 5159        The ``file.replace`` state uses Python's ``re`` module.
 5160        For more advanced options, see https://docs.python.org/2/library/re.html
 5161     """
 5162     name = os.path.expanduser(name)
 5163 
 5164     ret = {"name": name, "changes": {}, "result": True, "comment": ""}
 5165     if not name:
 5166         return _error(ret, "Must provide name to file.replace")
 5167 
 5168     check_res, check_msg = _check_file(name)
 5169     if not check_res:
 5170         if ignore_if_missing and "file not found" in check_msg:
 5171             ret["comment"] = "No changes needed to be made"
 5172             return ret
 5173         else:
 5174             return _error(ret, check_msg)
 5175 
 5176     changes = __salt__["file.replace"](
 5177         name,
 5178         pattern,
 5179         repl,
 5180         count=count,
 5181         flags=flags,
 5182         bufsize=bufsize,
 5183         append_if_not_found=append_if_not_found,
 5184         prepend_if_not_found=prepend_if_not_found,
 5185         not_found_content=not_found_content,
 5186         backup=backup,
 5187         dry_run=__opts__["test"],
 5188         show_changes=show_changes,
 5189         ignore_if_missing=ignore_if_missing,
 5190         backslash_literal=backslash_literal,
 5191     )
 5192 
 5193     if changes:
 5194         ret["changes"]["diff"] = changes
 5195         if __opts__["test"]:
 5196             ret["result"] = None
 5197             ret["comment"] = "Changes would have been made"
 5198         else:
 5199             ret["result"] = True
 5200             ret["comment"] = "Changes were made"
 5201     else:
 5202         ret["result"] = True
 5203         ret["comment"] = "No changes needed to be made"
 5204 
 5205     return ret
 5206 
 5207 
 5208 def keyvalue(
 5209     name,
 5210     key=None,
 5211     value=None,
 5212     key_values=None,
 5213     separator="=",
 5214     append_if_not_found=False,
 5215     prepend_if_not_found=False,
 5216     search_only=False,
 5217     show_changes=True,
 5218     ignore_if_missing=False,
 5219     count=1,
 5220     uncomment=None,
 5221     key_ignore_case=False,
 5222     value_ignore_case=False,
 5223 ):
 5224     """
 5225     Key/Value based editing of a file.
 5226 
 5227     .. versionadded:: 3001
 5228 
 5229     This function differs from ``file.replace`` in that it is able to search for
 5230     keys, followed by a customizable separator, and replace the value with the
 5231     given value. Should the value be the same as the one already in the file, no
 5232     changes will be made.
 5233 
 5234     Either supply both ``key`` and ``value`` parameters, or supply a dictionary
 5235     with key / value pairs. It is an error to supply both.
 5236 
 5237     name
 5238         Name of the file to search/replace in.
 5239 
 5240     key
 5241         Key to search for when ensuring a value. Use in combination with a
 5242         ``value`` parameter.
 5243 
 5244     value
 5245         Value to set for a given key. Use in combination with a ``key``
 5246         parameter.
 5247 
 5248     key_values
 5249         Dictionary of key / value pairs to search for and ensure values for.
 5250         Used to specify multiple key / values at once.
 5251 
 5252     separator : "="
 5253         Separator which separates key from value.
 5254 
 5255     append_if_not_found : False
 5256         Append the key/value to the end of the file if not found. Note that this
 5257         takes precedence over ``prepend_if_not_found``.
 5258 
 5259     prepend_if_not_found : False
 5260         Prepend the key/value to the beginning of the file if not found. Note
 5261         that ``append_if_not_found`` takes precedence.
 5262 
 5263     show_changes : True
 5264         Show a diff of the resulting removals and inserts.
 5265 
 5266     ignore_if_missing : False
 5267         Return with success even if the file is not found (or not readable).
 5268 
 5269     count : 1
 5270         Number of occurrences to allow (and correct), default is 1. Set to -1 to
 5271         replace all, or set to 0 to remove all lines with this key regardsless
 5272         of its value.
 5273 
 5274     .. note::
 5275         Any additional occurrences after ``count`` are removed.
 5276         A count of -1 will only replace all occurrences that are currently
 5277         uncommented already. Lines commented out will be left alone.
 5278 
 5279     uncomment : None
 5280         Disregard and remove supplied leading characters when finding keys. When
 5281         set to None, lines that are commented out are left for what they are.
 5282 
 5283     .. note::
 5284         The argument to ``uncomment`` is not a prefix string. Rather; it is a
 5285         set of characters, each of which are stripped.
 5286 
 5287     key_ignore_case : False
 5288         Keys are matched case insensitively. When a value is changed the matched
 5289         key is kept as-is.
 5290 
 5291     value_ignore_case : False
 5292         Values are checked case insensitively, trying to set e.g. 'Yes' while
 5293         the current value is 'yes', will not result in changes when
 5294         ``value_ignore_case`` is set to True.
 5295 
 5296     An example of using ``file.keyvalue`` to ensure sshd does not allow
 5297     for root to login with a password and at the same time setting the
 5298     login-gracetime to 1 minute and disabling all forwarding:
 5299 
 5300     .. code-block:: yaml
 5301 
 5302         sshd_config_harden:
 5303             file.keyvalue:
 5304               - name: /etc/ssh/sshd_config
 5305               - key_values:
 5306                   permitrootlogin: 'without-password'
 5307                   LoginGraceTime: '1m'
 5308                   DisableForwarding: 'yes'
 5309               - separator: ' '
 5310               - uncomment: '# '
 5311               - key_ignore_case: True
 5312               - append_if_not_found: True
 5313 
 5314     The same example, except for only ensuring PermitRootLogin is set correctly.
 5315     Thus being able to use the shorthand ``key`` and ``value`` parameters
 5316     instead of ``key_values``.
 5317 
 5318     .. code-block:: yaml
 5319 
 5320         sshd_config_harden:
 5321             file.keyvalue:
 5322               - name: /etc/ssh/sshd_config
 5323               - key: PermitRootLogin
 5324               - value: without-password
 5325               - separator: ' '
 5326               - uncomment: '# '
 5327               - key_ignore_case: True
 5328               - append_if_not_found: True
 5329 
 5330     .. note::
 5331         Notice how the key is not matched case-sensitively, this way it will
 5332         correctly identify both 'PermitRootLogin' as well as 'permitrootlogin'.
 5333 
 5334     """
 5335     name = os.path.expanduser(name)
 5336 
 5337     # default return values
 5338     ret = {
 5339         "name": name,
 5340         "changes": {},
 5341         "result": None,
 5342         "comment": "",
 5343     }
 5344 
 5345     if not name:
 5346         return _error(ret, "Must provide name to file.keyvalue")
 5347     if key is not None and value is not None:
 5348         if type(key_values) is dict:
 5349             return _error(
 5350                 ret, "file.keyvalue can not combine key_values with key and value"
 5351             )
 5352         key_values = {str(key): value}
 5353 
 5354     elif not isinstance(key_values, dict) or not key_values:
 5355         msg = "is not a dictionary"
 5356         if not key_values:
 5357             msg = "is empty"
 5358         return _error(
 5359             ret, "file.keyvalue key and value not supplied and key_values " + msg,
 5360         )
 5361 
 5362     # try to open the file and only return a comment if ignore_if_missing is
 5363     # enabled, also mark as an error if not
 5364     file_contents = []
 5365     try:
 5366         with salt.utils.files.fopen(name, "r") as fd:
 5367             file_contents = fd.readlines()
 5368     except OSError:
 5369         ret["comment"] = "unable to open {n}".format(n=name)
 5370         ret["result"] = True if ignore_if_missing else False
 5371         return ret
 5372 
 5373     # used to store diff combinations and check if anything has changed
 5374     diff = []
 5375     # store the final content of the file in case it needs to be rewritten
 5376     content = []
 5377     # target format is templated like this
 5378     tmpl = "{key}{sep}{value}" + os.linesep
 5379     # number of lines changed
 5380     changes = 0
 5381     # keep track of number of times a key was updated
 5382     diff_count = {k: count for k in key_values.keys()}
 5383 
 5384     # read all the lines from the file
 5385     for line in file_contents:
 5386         test_line = line.lstrip(uncomment)
 5387         did_uncomment = True if len(line) > len(test_line) else False
 5388 
 5389         if key_ignore_case:
 5390             test_line = test_line.lower()
 5391 
 5392         for key, value in key_values.items():
 5393             test_key = key.lower() if key_ignore_case else key
 5394             # if the line starts with the key
 5395             if test_line.startswith(test_key):
 5396                 # if the testline got uncommented then the real line needs to
 5397                 # be uncommented too, otherwhise there might be separation on
 5398                 # a character which is part of the comment set
 5399                 working_line = line.lstrip(uncomment) if did_uncomment else line
 5400 
 5401                 # try to separate the line into its' components
 5402                 line_key, line_sep, line_value = working_line.partition(separator)
 5403 
 5404                 # if separation was unsuccessful then line_sep is empty so
 5405                 # no need to keep trying. continue instead
 5406                 if line_sep != separator:
 5407                     continue
 5408 
 5409                 # start on the premises the key does not match the actual line
 5410                 keys_match = False
 5411                 if key_ignore_case:
 5412                     if line_key.lower() == test_key:
 5413                         keys_match = True
 5414                 else:
 5415                     if line_key == test_key:
 5416                         keys_match = True
 5417 
 5418                 # if the key was found in the line and separation was successful
 5419                 if keys_match:
 5420                     # trial and error have shown it's safest to strip whitespace
 5421                     # from values for the sake of matching
 5422                     line_value = line_value.strip()
 5423                     # make sure the value is an actual string at this point
 5424                     test_value = str(value).strip()
 5425                     # convert test_value and line_value to lowercase if need be
 5426                     if value_ignore_case:
 5427                         line_value = line_value.lower()
 5428                         test_value = test_value.lower()
 5429 
 5430                     # values match if they are equal at this point
 5431                     values_match = True if line_value == test_value else False
 5432 
 5433                     # in case a line had its comment removed there are some edge
 5434                     # cases that need considderation where changes are needed
 5435                     # regardless of values already matching.
 5436                     needs_changing = False
 5437                     if did_uncomment:
 5438                         # irrespective of a value, if it was commented out and
 5439                         # changes are still to be made, then it needs to be
 5440                         # commented in
 5441                         if diff_count[key] > 0:
 5442                             needs_changing = True
 5443                         # but if values did not match but there are really no
 5444                         # changes expected anymore either then leave this line
 5445                         elif not values_match:
 5446                             values_match = True
 5447                     else:
 5448                         # a line needs to be removed if it has been seen enough
 5449                         # times and was not commented out, regardless of value
 5450                         if diff_count[key] == 0:
 5451                             needs_changing = True
 5452 
 5453                     # then start checking to see if the value needs replacing
 5454                     if not values_match or needs_changing:
 5455                         # the old line always needs to go, so that will be
 5456                         # reflected in the diff (this is the original line from
 5457                         # the file being read)
 5458                         diff.append("- {}".format(line))
 5459                         line = line[:0]
 5460 
 5461                         # any non-zero value means something needs to go back in
 5462                         # its place. negative values are replacing all lines not
 5463                         # commented out, positive values are having their count
 5464                         # reduced by one every replacement
 5465                         if diff_count[key] != 0:
 5466                             # rebuild the line using the key and separator found
 5467                             # and insert the correct value.
 5468                             line = str(
 5469                                 tmpl.format(key=line_key, sep=line_sep, value=value)
 5470                             )
 5471 
 5472                             # display a comment in case a value got converted
 5473                             # into a string
 5474                             if not isinstance(value, str):
 5475                                 diff.append(
 5476                                     "+ {} (from {} type){}".format(
 5477                                         line.rstrip(), type(value).__name__, os.linesep
 5478                                     )
 5479                                 )
 5480                             else:
 5481                                 diff.append("+ {}".format(line))
 5482                         changes += 1
 5483                     # subtract one from the count if it was larger than 0, so
 5484                     # next lines are removed. if it is less than 0 then count is
 5485                     # ignored and all lines will be updated.
 5486                     if diff_count[key] > 0:
 5487                         diff_count[key] -= 1
 5488                     # at this point a continue saves going through the rest of
 5489                     # the keys to see if they match since this line already
 5490                     # matched the current key
 5491                     continue
 5492         # with the line having been checked for all keys (or matched before all
 5493         # keys needed searching), the line can be added to the content to be
 5494         # written once the last checks have been performed
 5495         content.append(line)
 5496     # finally, close the file
 5497     fd.close()
 5498 
 5499     # if append_if_not_found was requested, then append any key/value pairs
 5500     # still having a count left on them
 5501     if append_if_not_found:
 5502         tmpdiff = []
 5503         for key, value in key_values.items():
 5504             if diff_count[key] > 0:
 5505                 line = tmpl.format(key=key, sep=separator, value=value)
 5506                 tmpdiff.append("+ {}".format(line))
 5507                 content.append(line)
 5508                 changes += 1
 5509         if tmpdiff:
 5510             tmpdiff.insert(0, "- <EOF>" + os.linesep)
 5511             tmpdiff.append("+ <EOF>" + os.linesep)
 5512             diff.extend(tmpdiff)
 5513     # only if append_if_not_found was not set should prepend_if_not_found be
 5514     # considered, benefit of this is that the number of counts left does not
 5515     # mean there might be both a prepend and append happening
 5516     elif prepend_if_not_found:
 5517         did_diff = False
 5518         for key, value in key_values.items():
 5519             if diff_count[key] > 0:
 5520                 line = tmpl.format(key=key, sep=separator, value=value)
 5521                 if not did_diff:
 5522                     diff.insert(0, "  <SOF>" + os.linesep)
 5523                     did_diff = True
 5524                 diff.insert(1, "+ {}".format(line))
 5525                 content.insert(0, line)
 5526                 changes += 1
 5527 
 5528     # if a diff was made
 5529     if changes > 0:
 5530         # return comment of changes if test
 5531         if __opts__["test"]:
 5532             ret["comment"] = "File {n} is set to be changed ({c} lines)".format(
 5533                 n=name, c=changes
 5534             )
 5535             if show_changes:
 5536                 # For some reason, giving an actual diff even in test=True mode
 5537                 # will be seen as both a 'changed' and 'unchanged'. this seems to
 5538                 # match the other modules behaviour though
 5539                 ret["changes"]["diff"] = "".join(diff)
 5540 
 5541                 # add changes to comments for now as well because of how
 5542                 # stateoutputter seems to handle changes etc.
 5543                 # See: https://github.com/saltstack/salt/issues/40208
 5544                 ret["comment"] += "\nPredicted diff:\n\r\t\t"
 5545                 ret["comment"] += "\r\t\t".join(diff)
 5546                 ret["result"] = None
 5547 
 5548         # otherwise return the actual diff lines
 5549         else:
 5550             ret["comment"] = "Changed {c} lines".format(c=changes)
 5551             if show_changes:
 5552                 ret["changes"]["diff"] = "".join(diff)
 5553     else:
 5554         ret["result"] = True
 5555         return ret
 5556 
 5557     # if not test=true, try and write the file
 5558     if not __opts__["test"]:
 5559         try:
 5560             with salt.utils.files.fopen(name, "w") as fd:
 5561                 # write all lines to the file which was just truncated
 5562                 fd.writelines(content)
 5563                 fd.close()
 5564         except OSError:
 5565             # return an error if the file was not writable
 5566             ret["comment"] = "{n} not writable".format(n=name)
 5567             ret["result"] = False
 5568             return ret
 5569         # if all went well, then set result to true
 5570         ret["result"] = True
 5571 
 5572     return ret
 5573 
 5574 
 5575 def blockreplace(
 5576     name,
 5577     marker_start="#-- start managed zone --",
 5578     marker_end="#-- end managed zone --",
 5579     source=None,
 5580     source_hash=None,
 5581     template="jinja",
 5582     sources=None,
 5583     source_hashes=None,
 5584     defaults=None,
 5585     context=None,
 5586     content="",
 5587     append_if_not_found=False,
 5588     prepend_if_not_found=False,
 5589     backup=".bak",
 5590     show_changes=True,
 5591     append_newline=None,
 5592     insert_before_match=None,
 5593     insert_after_match=None,
 5594 ):
 5595     """
 5596     Maintain an edit in a file in a zone delimited by two line markers
 5597 
 5598     .. versionadded:: 2014.1.0
 5599     .. versionchanged:: 2017.7.5,2018.3.1
 5600         ``append_newline`` argument added. Additionally, to improve
 5601         idempotence, if the string represented by ``marker_end`` is found in
 5602         the middle of the line, the content preceding the marker will be
 5603         removed when the block is replaced. This allows one to remove
 5604         ``append_newline: False`` from the SLS and have the block properly
 5605         replaced if the end of the content block is immediately followed by the
 5606         ``marker_end`` (i.e. no newline before the marker).
 5607 
 5608     A block of content delimited by comments can help you manage several lines
 5609     entries without worrying about old entries removal. This can help you
 5610     maintaining an un-managed file containing manual edits.
 5611 
 5612     .. note::
 5613         This function will store two copies of the file in-memory (the original
 5614         version and the edited version) in order to detect changes and only
 5615         edit the targeted file if necessary.
 5616 
 5617         Additionally, you can use :py:func:`file.accumulated
 5618         <salt.states.file.accumulated>` and target this state. All accumulated
 5619         data dictionaries' content will be added in the content block.
 5620 
 5621     name
 5622         Filesystem path to the file to be edited
 5623 
 5624     marker_start
 5625         The line content identifying a line as the start of the content block.
 5626         Note that the whole line containing this marker will be considered, so
 5627         whitespace or extra content before or after the marker is included in
 5628         final output
 5629 
 5630     marker_end
 5631         The line content identifying the end of the content block. As of
 5632         versions 2017.7.5 and 2018.3.1, everything up to the text matching the
 5633         marker will be replaced, so it's important to ensure that your marker
 5634         includes the beginning of the text you wish to replace.
 5635 
 5636     content
 5637         The content to be used between the two lines identified by
 5638         ``marker_start`` and ``marker_end``
 5639 
 5640     source
 5641         The source file to download to the minion, this source file can be
 5642         hosted on either the salt master server, or on an HTTP or FTP server.
 5643         Both HTTPS and HTTP are supported as well as downloading directly
 5644         from Amazon S3 compatible URLs with both pre-configured and automatic
 5645         IAM credentials. (see s3.get state documentation)
 5646         File retrieval from Openstack Swift object storage is supported via
 5647         swift://container/object_path URLs, see swift.get documentation.
 5648         For files hosted on the salt file server, if the file is located on
 5649         the master in the directory named spam, and is called eggs, the source
 5650         string is salt://spam/eggs. If source is left blank or None
 5651         (use ~ in YAML), the file will be created as an empty file and
 5652         the content will not be managed. This is also the case when a file
 5653         already exists and the source is undefined; the contents of the file
 5654         will not be changed or managed.
 5655 
 5656         If the file is hosted on a HTTP or FTP server then the source_hash
 5657         argument is also required.
 5658 
 5659         A list of sources can also be passed in to provide a default source and
 5660         a set of fallbacks. The first source in the list that is found to exist
 5661         will be used and subsequent entries in the list will be ignored.
 5662 
 5663         .. code-block:: yaml
 5664 
 5665             file_override_example:
 5666               file.blockreplace:
 5667                 - name: /etc/example.conf
 5668                 - source:
 5669                   - salt://file_that_does_not_exist
 5670                   - salt://file_that_exists
 5671 
 5672     source_hash
 5673         This can be one of the following:
 5674             1. a source hash string
 5675             2. the URI of a file that contains source hash strings
 5676 
 5677         The function accepts the first encountered long unbroken alphanumeric
 5678         string of correct length as a valid hash, in order from most secure to
 5679         least secure:
 5680 
 5681         .. code-block:: text
 5682 
 5683             Type    Length
 5684             ======  ======
 5685             sha512     128
 5686             sha384      96
 5687             sha256      64
 5688             sha224      56
 5689             sha1        40
 5690             md5         32
 5691 
 5692         See the ``source_hash`` parameter description for :mod:`file.managed
 5693         <salt.states.file.managed>` function for more details and examples.
 5694 
 5695     template : jinja
 5696         Templating engine to be used to render the downloaded file. The
 5697         following engines are supported:
 5698 
 5699         - :mod:`cheetah <salt.renderers.cheetah>`
 5700         - :mod:`genshi <salt.renderers.genshi>`
 5701         - :mod:`jinja <salt.renderers.jinja>`
 5702         - :mod:`mako <salt.renderers.mako>`
 5703         - :mod:`py <salt.renderers.py>`
 5704         - :mod:`wempy <salt.renderers.wempy>`
 5705 
 5706     context
 5707         Overrides default context variables passed to the template
 5708 
 5709     defaults
 5710         Default context passed to the template
 5711 
 5712     append_if_not_found : False
 5713         If markers are not found and this option is set to ``True``, the
 5714         content block will be appended to the file.
 5715 
 5716     prepend_if_not_found : False
 5717         If markers are not found and this option is set to ``True``, the
 5718         content block will be prepended to the file.
 5719 
 5720     insert_before_match
 5721         If markers are not found, this parameter can be set to a regex which will
 5722         insert the block before the first found occurrence in the file.
 5723 
 5724         .. versionadded:: Sodium
 5725 
 5726     insert_after_match
 5727         If markers are not found, this parameter can be set to a regex which will
 5728         insert the block after the first found occurrence in the file.
 5729 
 5730         .. versionadded:: Sodium
 5731 
 5732     backup
 5733         The file extension to use for a backup of the file if any edit is made.
 5734         Set this to ``False`` to skip making a backup.
 5735 
 5736     dry_run : False
 5737         If ``True``, do not make any edits to the file and simply return the
 5738         changes that *would* be made.
 5739 
 5740     show_changes : True
 5741         Controls how changes are presented. If ``True``, the ``Changes``
 5742         section of the state return will contain a unified diff of the changes
 5743         made. If False, then it will contain a boolean (``True`` if any changes
 5744         were made, otherwise ``False``).
 5745 
 5746     append_newline
 5747         Controls whether or not a newline is appended to the content block. If
 5748         the value of this argument is ``True`` then a newline will be added to
 5749         the content block. If it is ``False``, then a newline will *not* be
 5750         added to the content block. If it is unspecified, then a newline will
 5751         only be added to the content block if it does not already end in a
 5752         newline.
 5753 
 5754         .. versionadded:: 2017.7.5,2018.3.1
 5755 
 5756     Example of usage with an accumulator and with a variable:
 5757 
 5758     .. code-block:: jinja
 5759 
 5760         {% set myvar = 42 %}
 5761         hosts-config-block-{{ myvar }}:
 5762           file.blockreplace:
 5763             - name: /etc/hosts
 5764             - marker_start: "# START managed zone {{ myvar }} -DO-NOT-EDIT-"
 5765             - marker_end: "# END managed zone {{ myvar }} --"
 5766             - content: 'First line of content'
 5767             - append_if_not_found: True
 5768             - backup: '.bak'
 5769             - show_changes: True
 5770 
 5771         hosts-config-block-{{ myvar }}-accumulated1:
 5772           file.accumulated:
 5773             - filename: /etc/hosts
 5774             - name: my-accumulator-{{ myvar }}
 5775             - text: "text 2"
 5776             - require_in:
 5777               - file: hosts-config-block-{{ myvar }}
 5778 
 5779         hosts-config-block-{{ myvar }}-accumulated2:
 5780           file.accumulated:
 5781             - filename: /etc/hosts
 5782             - name: my-accumulator-{{ myvar }}
 5783             - text: |
 5784                  text 3
 5785                  text 4
 5786             - require_in:
 5787               - file: hosts-config-block-{{ myvar }}
 5788 
 5789     will generate and maintain a block of content in ``/etc/hosts``:
 5790 
 5791     .. code-block:: text
 5792 
 5793         # START managed zone 42 -DO-NOT-EDIT-
 5794         First line of content
 5795         text 2
 5796         text 3
 5797         text 4
 5798         # END managed zone 42 --
 5799     """
 5800     name = os.path.expanduser(name)
 5801 
 5802     ret = {"name": name, "changes": {}, "result": False, "comment": ""}
 5803     if not name:
 5804         return _error(ret, "Must provide name to file.blockreplace")
 5805 
 5806     if sources is None:
 5807         sources = []
 5808     if source_hashes is None:
 5809         source_hashes = []
 5810 
 5811     (ok_, err, sl_) = _unify_sources_and_hashes(
 5812         source=source,
 5813         source_hash=source_hash,
 5814         sources=sources,
 5815         source_hashes=source_hashes,
 5816     )
 5817     if not ok_:
 5818         return _error(ret, err)
 5819 
 5820     check_res, check_msg = _check_file(name)
 5821     if not check_res:
 5822         return _error(ret, check_msg)
 5823 
 5824     accum_data, accum_deps = _load_accumulators()
 5825     if name in accum_data:
 5826         accumulator = accum_data[name]
 5827         # if we have multiple accumulators for a file, only apply the one
 5828         # required at a time
 5829         deps = accum_deps.get(name, [])
 5830         filtered = [
 5831             a for a in deps if __low__["__id__"] in deps[a] and a in accumulator
 5832         ]
 5833         if not filtered:
 5834             filtered = [a for a in accumulator]
 5835         for acc in filtered:
 5836             acc_content = accumulator[acc]
 5837             for line in acc_content:
 5838                 if content == "":
 5839                     content = line
 5840                 else:
 5841                     content += "\n" + line
 5842 
 5843     if sl_:
 5844         tmpret = _get_template_texts(
 5845             source_list=sl_, template=template, defaults=defaults, context=context
 5846         )
 5847         if not tmpret["result"]:
 5848             return tmpret
 5849         text = tmpret["data"]
 5850 
 5851         for index, item in enumerate(text):
 5852             content += str(item)
 5853 
 5854     try:
 5855         changes = __salt__["file.blockreplace"](
 5856             name,
 5857             marker_start,
 5858             marker_end,
 5859             content=content,
 5860             append_if_not_found=append_if_not_found,
 5861             prepend_if_not_found=prepend_if_not_found,
 5862             insert_before_match=insert_before_match,
 5863             insert_after_match=insert_after_match,
 5864             backup=backup,
 5865             dry_run=__opts__["test"],
 5866             show_changes=show_changes,
 5867             append_newline=append_newline,
 5868         )
 5869     except Exception as exc:  # pylint: disable=broad-except
 5870         log.exception("Encountered error managing block")
 5871         ret["comment"] = (
 5872             "Encountered error managing block: {}. "
 5873             "See the log for details.".format(exc)
 5874         )
 5875         return ret
 5876 
 5877     if changes:
 5878         ret["changes"]["diff"] = changes
 5879         if __opts__["test"]:
 5880             ret["result"] = None
 5881             ret["comment"] = "Changes would be made"
 5882         else:
 5883             ret["result"] = True
 5884             ret["comment"] = "Changes were made"
 5885     else:
 5886         ret["result"] = True
 5887         ret["comment"] = "No changes needed to be made"
 5888 
 5889     return ret
 5890 
 5891 
 5892 def comment(name, regex, char="#", backup=".bak"):
 5893     """
 5894     Comment out specified lines in a file.
 5895 
 5896     name
 5897         The full path to the file to be edited
 5898     regex
 5899         A regular expression used to find the lines that are to be commented;
 5900         this pattern will be wrapped in parenthesis and will move any
 5901         preceding/trailing ``^`` or ``$`` characters outside the parenthesis
 5902         (e.g., the pattern ``^foo$`` will be rewritten as ``^(foo)$``)
 5903         Note that you _need_ the leading ^, otherwise each time you run
 5904         highstate, another comment char will be inserted.
 5905     char : ``#``
 5906         The character to be inserted at the beginning of a line in order to
 5907         comment it out
 5908     backup : ``.bak``
 5909         The file will be backed up before edit with this file extension
 5910 
 5911         .. warning::
 5912 
 5913             This backup will be overwritten each time ``sed`` / ``comment`` /
 5914             ``uncomment`` is called. Meaning the backup will only be useful
 5915             after the first invocation.
 5916 
 5917         Set to False/None to not keep a backup.
 5918 
 5919     Usage:
 5920 
 5921     .. code-block:: yaml
 5922 
 5923         /etc/fstab:
 5924           file.comment:
 5925             - regex: ^bind 127.0.0.1
 5926 
 5927     .. versionadded:: 0.9.5
 5928     """
 5929     name = os.path.expanduser(name)
 5930 
 5931     ret = {"name": name, "changes": {}, "result": False, "comment": ""}
 5932     if not name:
 5933         return _error(ret, "Must provide name to file.comment")
 5934 
 5935     check_res, check_msg = _check_file(name)
 5936     if not check_res:
 5937         return _error(ret, check_msg)
 5938 
 5939     # remove (?i)-like flags, ^ and $
 5940     unanchor_regex = re.sub(r"^(\(\?[iLmsux]\))?\^?(.*?)\$?$", r"\2", regex)
 5941 
 5942     comment_regex = char + unanchor_regex
 5943 
 5944     # Make sure the pattern appears in the file before continuing
 5945     if not __salt__["file.search"](name, regex, multiline=True):
 5946         if __salt__["file.search"](name, comment_regex, multiline=True):
 5947             ret["comment"] = "Pattern already commented"
 5948             ret["result"] = True
 5949             return ret
 5950         else:
 5951             return _error(ret, "{}: Pattern not found".format(unanchor_regex))
 5952 
 5953     if __opts__["test"]:
 5954         ret["changes"][name] = "updated"
 5955         ret["comment"] = "File {} is set to be updated".format(name)
 5956         ret["result"] = None
 5957         return ret
 5958     with salt.utils.files.fopen(name, "rb") as fp_:
 5959         slines = fp_.read()
 5960         slines = slines.decode(__salt_system_encoding__)
 5961         slines = slines.splitlines(True)
 5962 
 5963     # Perform the edit
 5964     __salt__["file.comment_line"](name, regex, char, True, backup)
 5965 
 5966     with salt.utils.files.fopen(name, "rb") as fp_:
 5967         nlines = fp_.read()
 5968         nlines = nlines.decode(__salt_system_encoding__)
 5969         nlines = nlines.splitlines(True)
 5970 
 5971     # Check the result
 5972     ret["result"] = __salt__["file.search"](name, unanchor_regex, multiline=True)
 5973 
 5974     if slines != nlines:
 5975         if not __utils__["files.is_text"](name):
 5976             ret["changes"]["diff"