"Fossies" - the Fresh Open Source Software Archive 
Member "salt-3002.2/salt/states/file.py" (18 Nov 2020, 309936 Bytes) of package /linux/misc/salt-3002.2.tar.gz:
As a special service "Fossies" has tried to format the requested source page into HTML format using (guessed) Python source code syntax highlighting (style:
standard) with prefixed line numbers.
Alternatively you can here
view or
download the uninterpreted source code file.
For more information about "file.py" see the
Fossies "Dox" file reference documentation and the latest
Fossies "Diffs" side-by-side code changes report:
3002.1_vs_3002.2.
1 """
2 Operations on regular files, special files, directories, and symlinks
3 =====================================================================
4
5 Salt States can aggressively manipulate files on a system. There are a number
6 of ways in which files can be managed.
7
8 Regular files can be enforced with the :mod:`file.managed
9 <salt.states.file.managed>` state. This state downloads files from the salt
10 master and places them on the target system. Managed files can be rendered as a
11 jinja, mako, or wempy template, adding a dynamic component to file management.
12 An example of :mod:`file.managed <salt.states.file.managed>` which makes use of
13 the jinja templating system would look like this:
14
15 .. code-block:: jinja
16
17 /etc/http/conf/http.conf:
18 file.managed:
19 - source: salt://apache/http.conf
20 - user: root
21 - group: root
22 - mode: 644
23 - attrs: ai
24 - template: jinja
25 - defaults:
26 custom_var: "default value"
27 other_var: 123
28 {% if grains['os'] == 'Ubuntu' %}
29 - context:
30 custom_var: "override"
31 {% endif %}
32
33 It is also possible to use the :mod:`py renderer <salt.renderers.py>` as a
34 templating option. The template would be a Python script which would need to
35 contain a function called ``run()``, which returns a string. All arguments
36 to the state will be made available to the Python script as globals. The
37 returned string will be the contents of the managed file. For example:
38
39 .. code-block:: python
40
41 def run():
42 lines = ['foo', 'bar', 'baz']
43 lines.extend([source, name, user, context]) # Arguments as globals
44 return '\\n\\n'.join(lines)
45
46 .. note::
47
48 The ``defaults`` and ``context`` arguments require extra indentation (four
49 spaces instead of the normal two) in order to create a nested dictionary.
50 :ref:`More information <nested-dict-indentation>`.
51
52 If using a template, any user-defined template variables in the file defined in
53 ``source`` must be passed in using the ``defaults`` and/or ``context``
54 arguments. The general best practice is to place default values in
55 ``defaults``, with conditional overrides going into ``context``, as seen above.
56
57 The template will receive a variable ``custom_var``, which would be accessed in
58 the template using ``{{ custom_var }}``. If the operating system is Ubuntu, the
59 value of the variable ``custom_var`` would be *override*, otherwise it is the
60 default *default value*
61
62 The ``source`` parameter can be specified as a list. If this is done, then the
63 first file to be matched will be the one that is used. This allows you to have
64 a default file on which to fall back if the desired file does not exist on the
65 salt fileserver. Here's an example:
66
67 .. code-block:: jinja
68
69 /etc/foo.conf:
70 file.managed:
71 - source:
72 - salt://foo.conf.{{ grains['fqdn'] }}
73 - salt://foo.conf.fallback
74 - user: foo
75 - group: users
76 - mode: 644
77 - attrs: i
78 - backup: minion
79
80 .. note::
81
82 Salt supports backing up managed files via the backup option. For more
83 details on this functionality please review the
84 :ref:`backup_mode documentation <file-state-backups>`.
85
86 The ``source`` parameter can also specify a file in another Salt environment.
87 In this example ``foo.conf`` in the ``dev`` environment will be used instead.
88
89 .. code-block:: yaml
90
91 /etc/foo.conf:
92 file.managed:
93 - source:
94 - 'salt://foo.conf?saltenv=dev'
95 - user: foo
96 - group: users
97 - mode: '0644'
98 - attrs: i
99
100 .. warning::
101
102 When using a mode that includes a leading zero you must wrap the
103 value in single quotes. If the value is not wrapped in quotes it
104 will be read by YAML as an integer and evaluated as an octal.
105
106 The ``names`` parameter, which is part of the state compiler, can be used to
107 expand the contents of a single state declaration into multiple, single state
108 declarations. Each item in the ``names`` list receives its own individual state
109 ``name`` and is converted into its own low-data structure. This is a convenient
110 way to manage several files with similar attributes.
111
112 .. code-block:: yaml
113
114 salt_master_conf:
115 file.managed:
116 - user: root
117 - group: root
118 - mode: '0644'
119 - names:
120 - /etc/salt/master.d/master.conf:
121 - source: salt://saltmaster/master.conf
122 - /etc/salt/minion.d/minion-99.conf:
123 - source: salt://saltmaster/minion.conf
124
125 .. note::
126
127 There is more documentation about this feature in the :ref:`Names declaration
128 <names-declaration>` section of the :ref:`Highstate docs <states-highstate>`.
129
130 Special files can be managed via the ``mknod`` function. This function will
131 create and enforce the permissions on a special file. The function supports the
132 creation of character devices, block devices, and FIFO pipes. The function will
133 create the directory structure up to the special file if it is needed on the
134 minion. The function will not overwrite or operate on (change major/minor
135 numbers) existing special files with the exception of user, group, and
136 permissions. In most cases the creation of some special files require root
137 permissions on the minion. This would require that the minion to be run as the
138 root user. Here is an example of a character device:
139
140 .. code-block:: yaml
141
142 /var/named/chroot/dev/random:
143 file.mknod:
144 - ntype: c
145 - major: 1
146 - minor: 8
147 - user: named
148 - group: named
149 - mode: 660
150
151 Here is an example of a block device:
152
153 .. code-block:: yaml
154
155 /var/named/chroot/dev/loop0:
156 file.mknod:
157 - ntype: b
158 - major: 7
159 - minor: 0
160 - user: named
161 - group: named
162 - mode: 660
163
164 Here is an example of a fifo pipe:
165
166 .. code-block:: yaml
167
168 /var/named/chroot/var/log/logfifo:
169 file.mknod:
170 - ntype: p
171 - user: named
172 - group: named
173 - mode: 660
174
175 Directories can be managed via the ``directory`` function. This function can
176 create and enforce the permissions on a directory. A directory statement will
177 look like this:
178
179 .. code-block:: yaml
180
181 /srv/stuff/substuf:
182 file.directory:
183 - user: fred
184 - group: users
185 - mode: 755
186 - makedirs: True
187
188 If you need to enforce user and/or group ownership or permissions recursively
189 on the directory's contents, you can do so by adding a ``recurse`` directive:
190
191 .. code-block:: yaml
192
193 /srv/stuff/substuf:
194 file.directory:
195 - user: fred
196 - group: users
197 - mode: 755
198 - makedirs: True
199 - recurse:
200 - user
201 - group
202 - mode
203
204 As a default, ``mode`` will resolve to ``dir_mode`` and ``file_mode``, to
205 specify both directory and file permissions, use this form:
206
207 .. code-block:: yaml
208
209 /srv/stuff/substuf:
210 file.directory:
211 - user: fred
212 - group: users
213 - file_mode: 744
214 - dir_mode: 755
215 - makedirs: True
216 - recurse:
217 - user
218 - group
219 - mode
220
221 Symlinks can be easily created; the symlink function is very simple and only
222 takes a few arguments:
223
224 .. code-block:: yaml
225
226 /etc/grub.conf:
227 file.symlink:
228 - target: /boot/grub/grub.conf
229
230 Recursive directory management can also be set via the ``recurse``
231 function. Recursive directory management allows for a directory on the salt
232 master to be recursively copied down to the minion. This is a great tool for
233 deploying large code and configuration systems. A state using ``recurse``
234 would look something like this:
235
236 .. code-block:: yaml
237
238 /opt/code/flask:
239 file.recurse:
240 - source: salt://code/flask
241 - include_empty: True
242
243 A more complex ``recurse`` example:
244
245 .. code-block:: jinja
246
247 {% set site_user = 'testuser' %}
248 {% set site_name = 'test_site' %}
249 {% set project_name = 'test_proj' %}
250 {% set sites_dir = 'test_dir' %}
251
252 django-project:
253 file.recurse:
254 - name: {{ sites_dir }}/{{ site_name }}/{{ project_name }}
255 - user: {{ site_user }}
256 - dir_mode: 2775
257 - file_mode: '0644'
258 - template: jinja
259 - source: salt://project/templates_dir
260 - include_empty: True
261
262 Retention scheduling can be applied to manage contents of backup directories.
263 For example:
264
265 .. code-block:: yaml
266
267 /var/backups/example_directory:
268 file.retention_schedule:
269 - strptime_format: example_name_%Y%m%dT%H%M%S.tar.bz2
270 - retain:
271 most_recent: 5
272 first_of_hour: 4
273 first_of_day: 14
274 first_of_week: 6
275 first_of_month: 6
276 first_of_year: all
277
278 """
279
280
281 import copy
282 import difflib
283 import itertools
284 import logging
285 import os
286 import posixpath
287 import re
288 import shutil
289 import sys
290 import time
291 import traceback
292 from collections import defaultdict
293 from collections.abc import Iterable, Mapping
294 from datetime import date, datetime # python3 problem in the making?
295
296 import salt.loader
297 import salt.payload
298 import salt.utils.data
299 import salt.utils.dateutils
300 import salt.utils.dictupdate
301 import salt.utils.files
302 import salt.utils.hashutils
303 import salt.utils.path
304 import salt.utils.platform
305 import salt.utils.stringutils
306 import salt.utils.templates
307 import salt.utils.url
308 import salt.utils.versions
309 from salt.exceptions import CommandExecutionError
310 from salt.ext.six.moves import zip_longest
311 from salt.ext.six.moves.urllib.parse import urlparse as _urlparse
312 from salt.serializers import DeserializationError
313 from salt.state import get_accumulator_dir as _get_accumulator_dir
314
315 if salt.utils.platform.is_windows():
316 import salt.utils.win_dacl
317 import salt.utils.win_functions
318 import salt.utils.winapi
319
320 if salt.utils.platform.is_windows():
321 import pywintypes
322 import win32com.client
323
324 log = logging.getLogger(__name__)
325
326 COMMENT_REGEX = r"^([[:space:]]*){0}[[:space:]]?"
327 __NOT_FOUND = object()
328
329 __func_alias__ = {
330 "copy_": "copy",
331 }
332
333
334 def _get_accumulator_filepath():
335 """
336 Return accumulator data path.
337 """
338 return os.path.join(_get_accumulator_dir(__opts__["cachedir"]), __instance_id__)
339
340
341 def _load_accumulators():
342 def _deserialize(path):
343 serial = salt.payload.Serial(__opts__)
344 ret = {"accumulators": {}, "accumulators_deps": {}}
345 try:
346 with salt.utils.files.fopen(path, "rb") as f:
347 loaded = serial.load(f)
348 return loaded if loaded else ret
349 except (OSError, NameError):
350 # NameError is a msgpack error from salt-ssh
351 return ret
352
353 loaded = _deserialize(_get_accumulator_filepath())
354
355 return loaded["accumulators"], loaded["accumulators_deps"]
356
357
358 def _persist_accummulators(accumulators, accumulators_deps):
359 accumm_data = {"accumulators": accumulators, "accumulators_deps": accumulators_deps}
360
361 serial = salt.payload.Serial(__opts__)
362 try:
363 with salt.utils.files.fopen(_get_accumulator_filepath(), "w+b") as f:
364 serial.dump(accumm_data, f)
365 except NameError:
366 # msgpack error from salt-ssh
367 pass
368
369
370 def _check_user(user, group):
371 """
372 Checks if the named user and group are present on the minion
373 """
374 err = ""
375 if user:
376 uid = __salt__["file.user_to_uid"](user)
377 if uid == "":
378 err += "User {} is not available ".format(user)
379 if group:
380 gid = __salt__["file.group_to_gid"](group)
381 if gid == "":
382 err += "Group {} is not available".format(group)
383 return err
384
385
386 def _is_valid_relpath(relpath, maxdepth=None):
387 """
388 Performs basic sanity checks on a relative path.
389
390 Requires POSIX-compatible paths (i.e. the kind obtained through
391 cp.list_master or other such calls).
392
393 Ensures that the path does not contain directory transversal, and
394 that it does not exceed a stated maximum depth (if specified).
395 """
396 # Check relpath surrounded by slashes, so that `..` can be caught as
397 # a path component at the start, end, and in the middle of the path.
398 sep, pardir = posixpath.sep, posixpath.pardir
399 if sep + pardir + sep in sep + relpath + sep:
400 return False
401
402 # Check that the relative path's depth does not exceed maxdepth
403 if maxdepth is not None:
404 path_depth = relpath.strip(sep).count(sep)
405 if path_depth > maxdepth:
406 return False
407
408 return True
409
410
411 def _salt_to_os_path(path):
412 """
413 Converts a path from the form received via salt master to the OS's native
414 path format.
415 """
416 return os.path.normpath(path.replace(posixpath.sep, os.path.sep))
417
418
419 def _gen_recurse_managed_files(
420 name,
421 source,
422 keep_symlinks=False,
423 include_pat=None,
424 exclude_pat=None,
425 maxdepth=None,
426 include_empty=False,
427 **kwargs
428 ):
429 """
430 Generate the list of files managed by a recurse state
431 """
432
433 # Convert a relative path generated from salt master paths to an OS path
434 # using "name" as the base directory
435 def full_path(master_relpath):
436 return os.path.join(name, _salt_to_os_path(master_relpath))
437
438 # Process symlinks and return the updated filenames list
439 def process_symlinks(filenames, symlinks):
440 for lname, ltarget in symlinks.items():
441 srelpath = posixpath.relpath(lname, srcpath)
442 if not _is_valid_relpath(srelpath, maxdepth=maxdepth):
443 continue
444 if not salt.utils.stringutils.check_include_exclude(
445 srelpath, include_pat, exclude_pat
446 ):
447 continue
448 # Check for all paths that begin with the symlink
449 # and axe it leaving only the dirs/files below it.
450 # This needs to use list() otherwise they reference
451 # the same list.
452 _filenames = list(filenames)
453 for filename in _filenames:
454 if filename.startswith(lname):
455 log.debug(
456 "** skipping file ** {}, it intersects a "
457 "symlink".format(filename)
458 )
459 filenames.remove(filename)
460 # Create the symlink along with the necessary dirs.
461 # The dir perms/ownership will be adjusted later
462 # if needed
463 managed_symlinks.add((srelpath, ltarget))
464
465 # Add the path to the keep set in case clean is set to True
466 keep.add(full_path(srelpath))
467 vdir.update(keep)
468 return filenames
469
470 managed_files = set()
471 managed_directories = set()
472 managed_symlinks = set()
473 keep = set()
474 vdir = set()
475
476 srcpath, senv = salt.utils.url.parse(source)
477 if senv is None:
478 senv = __env__
479 if not srcpath.endswith(posixpath.sep):
480 # we're searching for things that start with this *directory*.
481 srcpath = srcpath + posixpath.sep
482 fns_ = __salt__["cp.list_master"](senv, srcpath)
483
484 # If we are instructed to keep symlinks, then process them.
485 if keep_symlinks:
486 # Make this global so that emptydirs can use it if needed.
487 symlinks = __salt__["cp.list_master_symlinks"](senv, srcpath)
488 fns_ = process_symlinks(fns_, symlinks)
489
490 for fn_ in fns_:
491 if not fn_.strip():
492 continue
493
494 # fn_ here is the absolute (from file_roots) source path of
495 # the file to copy from; it is either a normal file or an
496 # empty dir(if include_empty==true).
497
498 relname = salt.utils.data.decode(posixpath.relpath(fn_, srcpath))
499 if not _is_valid_relpath(relname, maxdepth=maxdepth):
500 continue
501
502 # Check if it is to be excluded. Match only part of the path
503 # relative to the target directory
504 if not salt.utils.stringutils.check_include_exclude(
505 relname, include_pat, exclude_pat
506 ):
507 continue
508 dest = full_path(relname)
509 dirname = os.path.dirname(dest)
510 keep.add(dest)
511
512 if dirname not in vdir:
513 # verify the directory perms if they are set
514 managed_directories.add(dirname)
515 vdir.add(dirname)
516
517 src = salt.utils.url.create(fn_, saltenv=senv)
518 managed_files.add((dest, src))
519
520 if include_empty:
521 mdirs = __salt__["cp.list_master_dirs"](senv, srcpath)
522 for mdir in mdirs:
523 relname = posixpath.relpath(mdir, srcpath)
524 if not _is_valid_relpath(relname, maxdepth=maxdepth):
525 continue
526 if not salt.utils.stringutils.check_include_exclude(
527 relname, include_pat, exclude_pat
528 ):
529 continue
530 mdest = full_path(relname)
531 # Check for symlinks that happen to point to an empty dir.
532 if keep_symlinks:
533 islink = False
534 for link in symlinks:
535 if mdir.startswith(link, 0):
536 log.debug(
537 "** skipping empty dir ** {}, it intersects"
538 " a symlink".format(mdir)
539 )
540 islink = True
541 break
542 if islink:
543 continue
544
545 managed_directories.add(mdest)
546 keep.add(mdest)
547
548 return managed_files, managed_directories, managed_symlinks, keep
549
550
551 def _gen_keep_files(name, require, walk_d=None):
552 """
553 Generate the list of files that need to be kept when a dir based function
554 like directory or recurse has a clean.
555 """
556
557 def _is_child(path, directory):
558 """
559 Check whether ``path`` is child of ``directory``
560 """
561 path = os.path.abspath(path)
562 directory = os.path.abspath(directory)
563
564 relative = os.path.relpath(path, directory)
565
566 return not relative.startswith(os.pardir)
567
568 def _add_current_path(path):
569 _ret = set()
570 if os.path.isdir(path):
571 dirs, files = walk_d.get(path, ((), ()))
572 _ret.add(path)
573 for _name in files:
574 _ret.add(os.path.join(path, _name))
575 for _name in dirs:
576 _ret.add(os.path.join(path, _name))
577 return _ret
578
579 def _process_by_walk_d(name, ret):
580 if os.path.isdir(name):
581 walk_ret.update(_add_current_path(name))
582 dirs, _ = walk_d.get(name, ((), ()))
583 for _d in dirs:
584 p = os.path.join(name, _d)
585 walk_ret.update(_add_current_path(p))
586 _process_by_walk_d(p, ret)
587
588 def _process(name):
589 ret = set()
590 if os.path.isdir(name):
591 for root, dirs, files in salt.utils.path.os_walk(name):
592 ret.add(name)
593 for name in files:
594 ret.add(os.path.join(root, name))
595 for name in dirs:
596 ret.add(os.path.join(root, name))
597 return ret
598
599 keep = set()
600 if isinstance(require, list):
601 required_files = [comp for comp in require if "file" in comp]
602 for comp in required_files:
603 for low in __lowstate__:
604 # A requirement should match either the ID and the name of
605 # another state.
606 if low["name"] == comp["file"] or low["__id__"] == comp["file"]:
607 fn = low["name"]
608 fun = low["fun"]
609 if os.path.isdir(fn):
610 if _is_child(fn, name):
611 if fun == "recurse":
612 fkeep = _gen_recurse_managed_files(**low)[3]
613 log.debug("Keep from {}: {}".format(fn, fkeep))
614 keep.update(fkeep)
615 elif walk_d:
616 walk_ret = set()
617 _process_by_walk_d(fn, walk_ret)
618 keep.update(walk_ret)
619 else:
620 keep.update(_process(fn))
621 else:
622 keep.add(fn)
623 log.debug("Files to keep from required states: {}".format(list(keep)))
624 return list(keep)
625
626
627 def _check_file(name):
628 ret = True
629 msg = ""
630
631 if not os.path.isabs(name):
632 ret = False
633 msg = "Specified file {} is not an absolute path".format(name)
634 elif not os.path.exists(name):
635 ret = False
636 msg = "{}: file not found".format(name)
637
638 return ret, msg
639
640
641 def _find_keep_files(root, keep):
642 """
643 Compile a list of valid keep files (and directories).
644 Used by _clean_dir()
645 """
646 real_keep = set()
647 real_keep.add(root)
648 if isinstance(keep, list):
649 for fn_ in keep:
650 if not os.path.isabs(fn_):
651 continue
652 fn_ = os.path.normcase(os.path.abspath(fn_))
653 real_keep.add(fn_)
654 while True:
655 fn_ = os.path.abspath(os.path.dirname(fn_))
656 real_keep.add(fn_)
657 drive, path = os.path.splitdrive(fn_)
658 if not path.lstrip(os.sep):
659 break
660 return real_keep
661
662
663 def _clean_dir(root, keep, exclude_pat):
664 """
665 Clean out all of the files and directories in a directory (root) while
666 preserving the files in a list (keep) and part of exclude_pat
667 """
668 root = os.path.normcase(root)
669 real_keep = _find_keep_files(root, keep)
670 removed = set()
671
672 def _delete_not_kept(nfn):
673 if nfn not in real_keep:
674 # -- check if this is a part of exclude_pat(only). No need to
675 # check include_pat
676 if not salt.utils.stringutils.check_include_exclude(
677 os.path.relpath(nfn, root), None, exclude_pat
678 ):
679 return
680 removed.add(nfn)
681 if not __opts__["test"]:
682 try:
683 os.remove(nfn)
684 except OSError:
685 __salt__["file.remove"](nfn)
686
687 for roots, dirs, files in salt.utils.path.os_walk(root):
688 for name in itertools.chain(dirs, files):
689 _delete_not_kept(os.path.join(roots, name))
690 return list(removed)
691
692
693 def _error(ret, err_msg):
694 ret["result"] = False
695 ret["comment"] = err_msg
696 return ret
697
698
699 def _check_directory(
700 name,
701 user=None,
702 group=None,
703 recurse=False,
704 dir_mode=None,
705 file_mode=None,
706 clean=False,
707 require=False,
708 exclude_pat=None,
709 max_depth=None,
710 follow_symlinks=False,
711 ):
712 """
713 Check what changes need to be made on a directory
714 """
715 changes = {}
716 if recurse or clean:
717 assert max_depth is None or not clean
718 # walk path only once and store the result
719 walk_l = list(_depth_limited_walk(name, max_depth))
720 # root: (dirs, files) structure, compatible for python2.6
721 walk_d = {}
722 for i in walk_l:
723 walk_d[i[0]] = (i[1], i[2])
724
725 if recurse:
726 try:
727 recurse_set = _get_recurse_set(recurse)
728 except (TypeError, ValueError) as exc:
729 return False, "{}".format(exc), changes
730 if "user" not in recurse_set:
731 user = None
732 if "group" not in recurse_set:
733 group = None
734 if "mode" not in recurse_set:
735 dir_mode = None
736 file_mode = None
737
738 check_files = "ignore_files" not in recurse_set
739 check_dirs = "ignore_dirs" not in recurse_set
740 for root, dirs, files in walk_l:
741 if check_files:
742 for fname in files:
743 fchange = {}
744 path = os.path.join(root, fname)
745 stats = __salt__["file.stats"](path, None, follow_symlinks)
746 if user is not None and user != stats.get("user"):
747 fchange["user"] = user
748 if group is not None and group != stats.get("group"):
749 fchange["group"] = group
750 smode = salt.utils.files.normalize_mode(stats.get("mode"))
751 file_mode = salt.utils.files.normalize_mode(file_mode)
752 if (
753 file_mode is not None
754 and file_mode != smode
755 and (
756 # Ignore mode for symlinks on linux based systems where we can not
757 # change symlink file permissions
758 follow_symlinks
759 or stats.get("type") != "link"
760 or not salt.utils.platform.is_linux()
761 )
762 ):
763 fchange["mode"] = file_mode
764 if fchange:
765 changes[path] = fchange
766 if check_dirs:
767 for name_ in dirs:
768 path = os.path.join(root, name_)
769 fchange = _check_dir_meta(
770 path, user, group, dir_mode, follow_symlinks
771 )
772 if fchange:
773 changes[path] = fchange
774 # Recurse skips root (we always do dirs, not root), so always check root:
775 fchange = _check_dir_meta(name, user, group, dir_mode, follow_symlinks)
776 if fchange:
777 changes[name] = fchange
778 if clean:
779 keep = _gen_keep_files(name, require, walk_d)
780
781 def _check_changes(fname):
782 path = os.path.join(root, fname)
783 if path in keep:
784 return {}
785 else:
786 if not salt.utils.stringutils.check_include_exclude(
787 os.path.relpath(path, name), None, exclude_pat
788 ):
789 return {}
790 else:
791 return {path: {"removed": "Removed due to clean"}}
792
793 for root, dirs, files in walk_l:
794 for fname in files:
795 changes.update(_check_changes(fname))
796 for name_ in dirs:
797 changes.update(_check_changes(name_))
798
799 if not os.path.isdir(name):
800 changes[name] = {"directory": "new"}
801 if changes:
802 comments = ["The following files will be changed:\n"]
803 for fn_ in changes:
804 for key, val in changes[fn_].items():
805 comments.append("{}: {} - {}\n".format(fn_, key, val))
806 return None, "".join(comments), changes
807 return True, "The directory {} is in the correct state".format(name), changes
808
809
810 def _check_directory_win(
811 name,
812 win_owner=None,
813 win_perms=None,
814 win_deny_perms=None,
815 win_inheritance=None,
816 win_perms_reset=None,
817 ):
818 """
819 Check what changes need to be made on a directory
820 """
821 changes = {}
822
823 if not os.path.isdir(name):
824 changes = {name: {"directory": "new"}}
825 else:
826 # Check owner by SID
827 if win_owner is not None:
828 current_owner = salt.utils.win_dacl.get_owner(name)
829 current_owner_sid = salt.utils.win_functions.get_sid_from_name(
830 current_owner
831 )
832 expected_owner_sid = salt.utils.win_functions.get_sid_from_name(win_owner)
833 if not current_owner_sid == expected_owner_sid:
834 changes["owner"] = win_owner
835
836 # Check perms
837 perms = salt.utils.win_dacl.get_permissions(name)
838
839 # Verify Permissions
840 if win_perms is not None:
841 for user in win_perms:
842 # Check that user exists:
843 try:
844 salt.utils.win_dacl.get_name(user)
845 except CommandExecutionError:
846 continue
847
848 grant_perms = []
849 # Check for permissions
850 if isinstance(win_perms[user]["perms"], str):
851 if not salt.utils.win_dacl.has_permission(
852 name, user, win_perms[user]["perms"]
853 ):
854 grant_perms = win_perms[user]["perms"]
855 else:
856 for perm in win_perms[user]["perms"]:
857 if not salt.utils.win_dacl.has_permission(
858 name, user, perm, exact=False
859 ):
860 grant_perms.append(win_perms[user]["perms"])
861 if grant_perms:
862 if "grant_perms" not in changes:
863 changes["grant_perms"] = {}
864 if user not in changes["grant_perms"]:
865 changes["grant_perms"][user] = {}
866 changes["grant_perms"][user]["perms"] = grant_perms
867
868 # Check Applies to
869 if "applies_to" not in win_perms[user]:
870 applies_to = "this_folder_subfolders_files"
871 else:
872 applies_to = win_perms[user]["applies_to"]
873
874 if user in perms:
875 user = salt.utils.win_dacl.get_name(user)
876
877 # Get the proper applies_to text
878 at_flag = salt.utils.win_dacl.flags().ace_prop["file"][applies_to]
879 applies_to_text = salt.utils.win_dacl.flags().ace_prop["file"][
880 at_flag
881 ]
882
883 if "grant" in perms[user]:
884 if not perms[user]["grant"]["applies to"] == applies_to_text:
885 if "grant_perms" not in changes:
886 changes["grant_perms"] = {}
887 if user not in changes["grant_perms"]:
888 changes["grant_perms"][user] = {}
889 changes["grant_perms"][user]["applies_to"] = applies_to
890
891 # Verify Deny Permissions
892 if win_deny_perms is not None:
893 for user in win_deny_perms:
894 # Check that user exists:
895 try:
896 salt.utils.win_dacl.get_name(user)
897 except CommandExecutionError:
898 continue
899
900 deny_perms = []
901 # Check for permissions
902 if isinstance(win_deny_perms[user]["perms"], str):
903 if not salt.utils.win_dacl.has_permission(
904 name, user, win_deny_perms[user]["perms"], "deny"
905 ):
906 deny_perms = win_deny_perms[user]["perms"]
907 else:
908 for perm in win_deny_perms[user]["perms"]:
909 if not salt.utils.win_dacl.has_permission(
910 name, user, perm, "deny", exact=False
911 ):
912 deny_perms.append(win_deny_perms[user]["perms"])
913 if deny_perms:
914 if "deny_perms" not in changes:
915 changes["deny_perms"] = {}
916 if user not in changes["deny_perms"]:
917 changes["deny_perms"][user] = {}
918 changes["deny_perms"][user]["perms"] = deny_perms
919
920 # Check Applies to
921 if "applies_to" not in win_deny_perms[user]:
922 applies_to = "this_folder_subfolders_files"
923 else:
924 applies_to = win_deny_perms[user]["applies_to"]
925
926 if user in perms:
927 user = salt.utils.win_dacl.get_name(user)
928
929 # Get the proper applies_to text
930 at_flag = salt.utils.win_dacl.flags().ace_prop["file"][applies_to]
931 applies_to_text = salt.utils.win_dacl.flags().ace_prop["file"][
932 at_flag
933 ]
934
935 if "deny" in perms[user]:
936 if not perms[user]["deny"]["applies to"] == applies_to_text:
937 if "deny_perms" not in changes:
938 changes["deny_perms"] = {}
939 if user not in changes["deny_perms"]:
940 changes["deny_perms"][user] = {}
941 changes["deny_perms"][user]["applies_to"] = applies_to
942
943 # Check inheritance
944 if win_inheritance is not None:
945 if not win_inheritance == salt.utils.win_dacl.get_inheritance(name):
946 changes["inheritance"] = win_inheritance
947
948 # Check reset
949 if win_perms_reset:
950 for user_name in perms:
951 if user_name not in win_perms:
952 if (
953 "grant" in perms[user_name]
954 and not perms[user_name]["grant"]["inherited"]
955 ):
956 if "remove_perms" not in changes:
957 changes["remove_perms"] = {}
958 changes["remove_perms"].update({user_name: perms[user_name]})
959 if user_name not in win_deny_perms:
960 if (
961 "deny" in perms[user_name]
962 and not perms[user_name]["deny"]["inherited"]
963 ):
964 if "remove_perms" not in changes:
965 changes["remove_perms"] = {}
966 changes["remove_perms"].update({user_name: perms[user_name]})
967
968 if changes:
969 return None, 'The directory "{}" will be changed'.format(name), changes
970
971 return True, "The directory {} is in the correct state".format(name), changes
972
973
974 def _check_dir_meta(name, user, group, mode, follow_symlinks=False):
975 """
976 Check the changes in directory metadata
977 """
978 try:
979 stats = __salt__["file.stats"](name, None, follow_symlinks)
980 except CommandExecutionError:
981 stats = {}
982
983 changes = {}
984 if not stats:
985 changes["directory"] = "new"
986 return changes
987 if user is not None and user != stats["user"] and user != stats.get("uid"):
988 changes["user"] = user
989 if group is not None and group != stats["group"] and group != stats.get("gid"):
990 changes["group"] = group
991 # Normalize the dir mode
992 smode = salt.utils.files.normalize_mode(stats["mode"])
993 mode = salt.utils.files.normalize_mode(mode)
994 if (
995 mode is not None
996 and mode != smode
997 and (
998 # Ignore mode for symlinks on linux based systems where we can not
999 # change symlink file permissions
1000 follow_symlinks
1001 or stats.get("type") != "link"
1002 or not salt.utils.platform.is_linux()
1003 )
1004 ):
1005 changes["mode"] = mode
1006 return changes
1007
1008
1009 def _check_touch(name, atime, mtime):
1010 """
1011 Check to see if a file needs to be updated or created
1012 """
1013 ret = {
1014 "result": None,
1015 "comment": "",
1016 "changes": {"new": name},
1017 }
1018 if not os.path.exists(name):
1019 ret["comment"] = "File {} is set to be created".format(name)
1020 else:
1021 stats = __salt__["file.stats"](name, follow_symlinks=False)
1022 if (atime is not None and str(atime) != str(stats["atime"])) or (
1023 mtime is not None and str(mtime) != str(stats["mtime"])
1024 ):
1025 ret["comment"] = "Times set to be updated on file {}".format(name)
1026 ret["changes"] = {"touched": name}
1027 else:
1028 ret["result"] = True
1029 ret["comment"] = "File {} exists and has the correct times".format(name)
1030 return ret
1031
1032
1033 def _get_symlink_ownership(path):
1034 if salt.utils.platform.is_windows():
1035 owner = salt.utils.win_dacl.get_owner(path)
1036 return owner, owner
1037 else:
1038 return (
1039 __salt__["file.get_user"](path, follow_symlinks=False),
1040 __salt__["file.get_group"](path, follow_symlinks=False),
1041 )
1042
1043
1044 def _check_symlink_ownership(path, user, group, win_owner):
1045 """
1046 Check if the symlink ownership matches the specified user and group
1047 """
1048 cur_user, cur_group = _get_symlink_ownership(path)
1049 if salt.utils.platform.is_windows():
1050 return win_owner == cur_user
1051 else:
1052 return (cur_user == user) and (cur_group == group)
1053
1054
1055 def _set_symlink_ownership(path, user, group, win_owner):
1056 """
1057 Set the ownership of a symlink and return a boolean indicating
1058 success/failure
1059 """
1060 if salt.utils.platform.is_windows():
1061 try:
1062 salt.utils.win_dacl.set_owner(path, win_owner)
1063 except CommandExecutionError:
1064 pass
1065 else:
1066 try:
1067 __salt__["file.lchown"](path, user, group)
1068 except OSError:
1069 pass
1070 return _check_symlink_ownership(path, user, group, win_owner)
1071
1072
1073 def _symlink_check(name, target, force, user, group, win_owner):
1074 """
1075 Check the symlink function
1076 """
1077 changes = {}
1078 if not os.path.exists(name) and not __salt__["file.is_link"](name):
1079 changes["new"] = name
1080 return (
1081 None,
1082 "Symlink {} to {} is set for creation".format(name, target),
1083 changes,
1084 )
1085 if __salt__["file.is_link"](name):
1086 if __salt__["file.readlink"](name) != target:
1087 changes["change"] = name
1088 return (
1089 None,
1090 "Link {} target is set to be changed to {}".format(name, target),
1091 changes,
1092 )
1093 else:
1094 result = True
1095 msg = "The symlink {} is present".format(name)
1096 if not _check_symlink_ownership(name, user, group, win_owner):
1097 result = None
1098 changes["ownership"] = "{}:{}".format(*_get_symlink_ownership(name))
1099 msg += (
1100 ", but the ownership of the symlink would be changed "
1101 "from {2}:{3} to {0}:{1}"
1102 ).format(user, group, *_get_symlink_ownership(name))
1103 return result, msg, changes
1104 else:
1105 if force:
1106 return (
1107 None,
1108 (
1109 "The file or directory {} is set for removal to "
1110 "make way for a new symlink targeting {}".format(name, target)
1111 ),
1112 changes,
1113 )
1114 return (
1115 False,
1116 (
1117 "File or directory exists where the symlink {} "
1118 "should be. Did you mean to use force?".format(name)
1119 ),
1120 changes,
1121 )
1122
1123
1124 def _hardlink_same(name, target):
1125 """
1126 Check to see if the inodes match for the name and the target
1127 """
1128 res = __salt__["file.stats"](name, None, follow_symlinks=False)
1129 if "inode" not in res:
1130 return False
1131 name_i = res["inode"]
1132
1133 res = __salt__["file.stats"](target, None, follow_symlinks=False)
1134 if "inode" not in res:
1135 return False
1136 target_i = res["inode"]
1137
1138 return name_i == target_i
1139
1140
1141 def _hardlink_check(name, target, force):
1142 """
1143 Check the hardlink function
1144 """
1145 changes = {}
1146 if not os.path.exists(target):
1147 msg = "Target {} for hard link does not exist".format(target)
1148 return False, msg, changes
1149
1150 elif os.path.isdir(target):
1151 msg = "Unable to hard link from directory {}".format(target)
1152 return False, msg, changes
1153
1154 if os.path.isdir(name):
1155 msg = "Unable to hard link to directory {}".format(name)
1156 return False, msg, changes
1157
1158 elif not os.path.exists(name):
1159 msg = "Hard link {} to {} is set for creation".format(name, target)
1160 changes["new"] = name
1161 return None, msg, changes
1162
1163 elif __salt__["file.is_hardlink"](name):
1164 if _hardlink_same(name, target):
1165 msg = "The hard link {} is presently targetting {}".format(name, target)
1166 return True, msg, changes
1167
1168 msg = "Link {} target is set to be changed to {}".format(name, target)
1169 changes["change"] = name
1170 return None, msg, changes
1171
1172 if force:
1173 msg = (
1174 "The file or directory {} is set for removal to "
1175 "make way for a new hard link targeting {}".format(name, target)
1176 )
1177 return None, msg, changes
1178
1179 msg = (
1180 "File or directory exists where the hard link {} "
1181 "should be. Did you mean to use force?".format(name)
1182 )
1183 return False, msg, changes
1184
1185
1186 def _test_owner(kwargs, user=None):
1187 """
1188 Convert owner to user, since other config management tools use owner,
1189 no need to punish people coming from other systems.
1190 PLEASE DO NOT DOCUMENT THIS! WE USE USER, NOT OWNER!!!!
1191 """
1192 if user:
1193 return user
1194 if "owner" in kwargs:
1195 log.warning(
1196 'Use of argument owner found, "owner" is invalid, please ' 'use "user"'
1197 )
1198 return kwargs["owner"]
1199
1200 return user
1201
1202
1203 def _unify_sources_and_hashes(
1204 source=None, source_hash=None, sources=None, source_hashes=None
1205 ):
1206 """
1207 Silly little function to give us a standard tuple list for sources and
1208 source_hashes
1209 """
1210 if sources is None:
1211 sources = []
1212
1213 if source_hashes is None:
1214 source_hashes = []
1215
1216 if source and sources:
1217 return (False, "source and sources are mutually exclusive", [])
1218
1219 if source_hash and source_hashes:
1220 return (False, "source_hash and source_hashes are mutually exclusive", [])
1221
1222 if source:
1223 return (True, "", [(source, source_hash)])
1224
1225 # Make a nice neat list of tuples exactly len(sources) long..
1226 return True, "", list(zip_longest(sources, source_hashes[: len(sources)]))
1227
1228
1229 def _get_template_texts(
1230 source_list=None, template="jinja", defaults=None, context=None, **kwargs
1231 ):
1232 """
1233 Iterate a list of sources and process them as templates.
1234 Returns a list of 'chunks' containing the rendered templates.
1235 """
1236
1237 ret = {
1238 "name": "_get_template_texts",
1239 "changes": {},
1240 "result": True,
1241 "comment": "",
1242 "data": [],
1243 }
1244
1245 if source_list is None:
1246 return _error(ret, "_get_template_texts called with empty source_list")
1247
1248 txtl = []
1249
1250 for (source, source_hash) in source_list:
1251
1252 tmpctx = defaults if defaults else {}
1253 if context:
1254 tmpctx.update(context)
1255 rndrd_templ_fn = __salt__["cp.get_template"](
1256 source, "", template=template, saltenv=__env__, context=tmpctx, **kwargs
1257 )
1258 msg = "cp.get_template returned {0} (Called with: {1})"
1259 log.debug(msg.format(rndrd_templ_fn, source))
1260 if rndrd_templ_fn:
1261 tmplines = None
1262 with salt.utils.files.fopen(rndrd_templ_fn, "rb") as fp_:
1263 tmplines = fp_.read()
1264 tmplines = salt.utils.stringutils.to_unicode(tmplines)
1265 tmplines = tmplines.splitlines(True)
1266 if not tmplines:
1267 msg = "Failed to read rendered template file {0} ({1})"
1268 log.debug(msg.format(rndrd_templ_fn, source))
1269 ret["name"] = source
1270 return _error(ret, msg.format(rndrd_templ_fn, source))
1271 txtl.append("".join(tmplines))
1272 else:
1273 msg = "Failed to load template file {}".format(source)
1274 log.debug(msg)
1275 ret["name"] = source
1276 return _error(ret, msg)
1277
1278 ret["data"] = txtl
1279 return ret
1280
1281
1282 def _validate_str_list(arg, encoding=None):
1283 """
1284 ensure ``arg`` is a list of strings
1285 """
1286 if isinstance(arg, bytes):
1287 ret = [salt.utils.stringutils.to_unicode(arg, encoding=encoding)]
1288 elif isinstance(arg, str):
1289 ret = [arg]
1290 elif isinstance(arg, Iterable) and not isinstance(arg, Mapping):
1291 ret = []
1292 for item in arg:
1293 if isinstance(item, str):
1294 ret.append(item)
1295 else:
1296 ret.append(str(item))
1297 else:
1298 ret = [str(arg)]
1299 return ret
1300
1301
1302 def _get_shortcut_ownership(path):
1303 return __salt__["file.get_user"](path, follow_symlinks=False)
1304
1305
1306 def _check_shortcut_ownership(path, user):
1307 """
1308 Check if the shortcut ownership matches the specified user
1309 """
1310 cur_user = _get_shortcut_ownership(path)
1311 return cur_user == user
1312
1313
1314 def _set_shortcut_ownership(path, user):
1315 """
1316 Set the ownership of a shortcut and return a boolean indicating
1317 success/failure
1318 """
1319 try:
1320 __salt__["file.lchown"](path, user)
1321 except OSError:
1322 pass
1323 return _check_shortcut_ownership(path, user)
1324
1325
1326 def _shortcut_check(
1327 name, target, arguments, working_dir, description, icon_location, force, user
1328 ):
1329 """
1330 Check the shortcut function
1331 """
1332 changes = {}
1333 if not os.path.exists(name):
1334 changes["new"] = name
1335 return (
1336 None,
1337 'Shortcut "{}" to "{}" is set for creation'.format(name, target),
1338 changes,
1339 )
1340
1341 if os.path.isfile(name):
1342 with salt.utils.winapi.Com():
1343 shell = win32com.client.Dispatch("WScript.Shell")
1344 scut = shell.CreateShortcut(name)
1345 state_checks = [scut.TargetPath.lower() == target.lower()]
1346 if arguments is not None:
1347 state_checks.append(scut.Arguments == arguments)
1348 if working_dir is not None:
1349 state_checks.append(
1350 scut.WorkingDirectory.lower() == working_dir.lower()
1351 )
1352 if description is not None:
1353 state_checks.append(scut.Description == description)
1354 if icon_location is not None:
1355 state_checks.append(scut.IconLocation.lower() == icon_location.lower())
1356
1357 if not all(state_checks):
1358 changes["change"] = name
1359 return (
1360 None,
1361 'Shortcut "{}" target is set to be changed to "{}"'.format(
1362 name, target
1363 ),
1364 changes,
1365 )
1366 else:
1367 result = True
1368 msg = 'The shortcut "{}" is present'.format(name)
1369 if not _check_shortcut_ownership(name, user):
1370 result = None
1371 changes["ownership"] = "{}".format(_get_shortcut_ownership(name))
1372 msg += (
1373 ", but the ownership of the shortcut would be changed "
1374 "from {1} to {0}"
1375 ).format(user, _get_shortcut_ownership(name))
1376 return result, msg, changes
1377 else:
1378 if force:
1379 return (
1380 None,
1381 (
1382 'The link or directory "{}" is set for removal to '
1383 'make way for a new shortcut targeting "{}"'.format(name, target)
1384 ),
1385 changes,
1386 )
1387 return (
1388 False,
1389 (
1390 'Link or directory exists where the shortcut "{}" '
1391 "should be. Did you mean to use force?".format(name)
1392 ),
1393 changes,
1394 )
1395
1396
1397 def _makedirs(
1398 name,
1399 user=None,
1400 group=None,
1401 dir_mode=None,
1402 win_owner=None,
1403 win_perms=None,
1404 win_deny_perms=None,
1405 win_inheritance=None,
1406 ):
1407 """
1408 Helper function for creating directories when the ``makedirs`` option is set
1409 to ``True``. Handles Unix and Windows based systems
1410
1411 .. versionadded:: 2017.7.8
1412
1413 Args:
1414 name (str): The directory path to create
1415 user (str): The linux user to own the directory
1416 group (str): The linux group to own the directory
1417 dir_mode (str): The linux mode to apply to the directory
1418 win_owner (str): The Windows user to own the directory
1419 win_perms (dict): A dictionary of grant permissions for Windows
1420 win_deny_perms (dict): A dictionary of deny permissions for Windows
1421 win_inheritance (bool): True to inherit permissions on Windows
1422
1423 Returns:
1424 bool: True if successful, otherwise False on Windows
1425 str: Error messages on failure on Linux
1426 None: On successful creation on Linux
1427
1428 Raises:
1429 CommandExecutionError: If the drive is not mounted on Windows
1430 """
1431 if salt.utils.platform.is_windows():
1432 # Make sure the drive is mapped before trying to create the
1433 # path in windows
1434 drive, path = os.path.splitdrive(name)
1435 if not os.path.isdir(drive):
1436 raise CommandExecutionError(drive)
1437 win_owner = win_owner if win_owner else user
1438 return __salt__["file.makedirs"](
1439 path=name,
1440 owner=win_owner,
1441 grant_perms=win_perms,
1442 deny_perms=win_deny_perms,
1443 inheritance=win_inheritance,
1444 )
1445 else:
1446 return __salt__["file.makedirs"](
1447 path=name, user=user, group=group, mode=dir_mode
1448 )
1449
1450
1451 def hardlink(
1452 name,
1453 target,
1454 force=False,
1455 makedirs=False,
1456 user=None,
1457 group=None,
1458 dir_mode=None,
1459 **kwargs
1460 ):
1461 """
1462 Create a hard link
1463 If the file already exists and is a hard link pointing to any location other
1464 than the specified target, the hard link will be replaced. If the hard link
1465 is a regular file or directory then the state will return False. If the
1466 regular file is desired to be replaced with a hard link pass force: True
1467
1468 name
1469 The location of the hard link to create
1470 target
1471 The location that the hard link points to
1472 force
1473 If the name of the hard link exists and force is set to False, the
1474 state will fail. If force is set to True, the file or directory in the
1475 way of the hard link file will be deleted to make room for the hard
1476 link, unless backupname is set, when it will be renamed
1477 makedirs
1478 If the location of the hard link does not already have a parent directory
1479 then the state will fail, setting makedirs to True will allow Salt to
1480 create the parent directory
1481 user
1482 The user to own any directories made if makedirs is set to true. This
1483 defaults to the user salt is running as on the minion
1484 group
1485 The group ownership set on any directories made if makedirs is set to
1486 true. This defaults to the group salt is running as on the minion. On
1487 Windows, this is ignored
1488 dir_mode
1489 If directories are to be created, passing this option specifies the
1490 permissions for those directories.
1491 """
1492 name = os.path.expanduser(name)
1493
1494 # Make sure that leading zeros stripped by YAML loader are added back
1495 dir_mode = salt.utils.files.normalize_mode(dir_mode)
1496
1497 user = _test_owner(kwargs, user=user)
1498 ret = {"name": name, "changes": {}, "result": True, "comment": ""}
1499 if not name:
1500 return _error(ret, "Must provide name to file.hardlink")
1501
1502 if user is None:
1503 user = __opts__["user"]
1504
1505 if salt.utils.platform.is_windows():
1506 if group is not None:
1507 log.warning(
1508 "The group argument for {} has been ignored as this "
1509 "is a Windows system.".format(name)
1510 )
1511 group = user
1512
1513 if group is None:
1514 group = __salt__["file.gid_to_group"](__salt__["user.info"](user).get("gid", 0))
1515
1516 preflight_errors = []
1517 uid = __salt__["file.user_to_uid"](user)
1518 gid = __salt__["file.group_to_gid"](group)
1519
1520 if uid == "":
1521 preflight_errors.append("User {} does not exist".format(user))
1522
1523 if gid == "":
1524 preflight_errors.append("Group {} does not exist".format(group))
1525
1526 if not os.path.isabs(name):
1527 preflight_errors.append(
1528 "Specified file {} is not an absolute path".format(name)
1529 )
1530
1531 if not os.path.isabs(target):
1532 preflight_errors.append(
1533 "Specified target {} is not an absolute path".format(target)
1534 )
1535
1536 if preflight_errors:
1537 msg = ". ".join(preflight_errors)
1538 if len(preflight_errors) > 1:
1539 msg += "."
1540 return _error(ret, msg)
1541
1542 if __opts__["test"]:
1543 tresult, tcomment, tchanges = _hardlink_check(name, target, force)
1544 ret["result"] = tresult
1545 ret["comment"] = tcomment
1546 ret["changes"] = tchanges
1547 return ret
1548
1549 # We use zip_longest here because there's a number of issues in pylint's
1550 # tracker that complains about not linking the zip builtin.
1551 for direction, item in zip_longest(["to", "from"], [name, target]):
1552 if os.path.isdir(item):
1553 msg = "Unable to hard link {} directory {}".format(direction, item)
1554 return _error(ret, msg)
1555
1556 if not os.path.exists(target):
1557 msg = "Target {} for hard link does not exist".format(target)
1558 return _error(ret, msg)
1559
1560 # Check that the directory to write the hard link to exists
1561 if not os.path.isdir(os.path.dirname(name)):
1562 if makedirs:
1563 __salt__["file.makedirs"](name, user=user, group=group, mode=dir_mode)
1564
1565 else:
1566 return _error(
1567 ret,
1568 "Directory {} for hard link is not present".format(
1569 os.path.dirname(name)
1570 ),
1571 )
1572
1573 # If file is not a hard link and we're actually overwriting it, then verify
1574 # that this was forced.
1575 if os.path.isfile(name) and not __salt__["file.is_hardlink"](name):
1576
1577 # Remove whatever is in the way. This should then hit the else case
1578 # of the file.is_hardlink check below
1579 if force:
1580 os.remove(name)
1581 ret["changes"]["forced"] = "File for hard link was forcibly replaced"
1582
1583 # Otherwise throw an error
1584 else:
1585 return _error(
1586 ret, ("File exists where the hard link {} should be".format(name))
1587 )
1588
1589 # If the file is a hard link, then we can simply rewrite its target since
1590 # nothing is really being lost here.
1591 if __salt__["file.is_hardlink"](name):
1592
1593 # If the inodes point to the same thing, then there's nothing to do
1594 # except for let the user know that this has already happened.
1595 if _hardlink_same(name, target):
1596 ret["result"] = True
1597 ret["comment"] = (
1598 "Target of hard link {} is already pointing "
1599 "to {}".format(name, target)
1600 )
1601 return ret
1602
1603 # First remove the old hard link since a reference to it already exists
1604 os.remove(name)
1605
1606 # Now we can remake it
1607 try:
1608 __salt__["file.link"](target, name)
1609
1610 # Or not...
1611 except CommandExecutionError as E:
1612 ret["result"] = False
1613 ret["comment"] = "Unable to set target of hard link {} -> " "{}: {}".format(
1614 name, target, E
1615 )
1616 return ret
1617
1618 # Good to go
1619 ret["result"] = True
1620 ret["comment"] = "Set target of hard link {} -> {}".format(name, target)
1621 ret["changes"]["new"] = name
1622
1623 # The link is not present, so simply make it
1624 elif not os.path.exists(name):
1625 try:
1626 __salt__["file.link"](target, name)
1627
1628 # Or not...
1629 except CommandExecutionError as E:
1630 ret["result"] = False
1631 ret["comment"] = "Unable to create new hard link {} -> " "{}: {}".format(
1632 name, target, E
1633 )
1634 return ret
1635
1636 # Made a new hard link, things are ok
1637 ret["result"] = True
1638 ret["comment"] = "Created new hard link {} -> {}".format(name, target)
1639 ret["changes"]["new"] = name
1640
1641 return ret
1642
1643
1644 def symlink(
1645 name,
1646 target,
1647 force=False,
1648 backupname=None,
1649 makedirs=False,
1650 user=None,
1651 group=None,
1652 mode=None,
1653 win_owner=None,
1654 win_perms=None,
1655 win_deny_perms=None,
1656 win_inheritance=None,
1657 **kwargs
1658 ):
1659 """
1660 Create a symbolic link (symlink, soft link)
1661
1662 If the file already exists and is a symlink pointing to any location other
1663 than the specified target, the symlink will be replaced. If an entry with
1664 the same name exists then the state will return False. If the existing
1665 entry is desired to be replaced with a symlink pass force: True, if it is
1666 to be renamed, pass a backupname.
1667
1668 name
1669 The location of the symlink to create
1670
1671 target
1672 The location that the symlink points to
1673
1674 force
1675 If the name of the symlink exists and is not a symlink and
1676 force is set to False, the state will fail. If force is set to
1677 True, the existing entry in the way of the symlink file
1678 will be deleted to make room for the symlink, unless
1679 backupname is set, when it will be renamed
1680
1681 .. versionchanged:: Neon
1682 Force will now remove all types of existing file system entries,
1683 not just files, directories and symlinks.
1684
1685 backupname
1686 If the name of the symlink exists and is not a symlink, it will be
1687 renamed to the backupname. If the backupname already
1688 exists and force is False, the state will fail. Otherwise, the
1689 backupname will be removed first.
1690 An absolute path OR a basename file/directory name must be provided.
1691 The latter will be placed relative to the symlink destination's parent
1692 directory.
1693
1694 makedirs
1695 If the location of the symlink does not already have a parent directory
1696 then the state will fail, setting makedirs to True will allow Salt to
1697 create the parent directory
1698
1699 user
1700 The user to own the file, this defaults to the user salt is running as
1701 on the minion
1702
1703 group
1704 The group ownership set for the file, this defaults to the group salt
1705 is running as on the minion. On Windows, this is ignored
1706
1707 mode
1708 The permissions to set on this file, aka 644, 0775, 4664. Not supported
1709 on Windows.
1710
1711 The default mode for new files and directories corresponds umask of salt
1712 process. For existing files and directories it's not enforced.
1713
1714 win_owner : None
1715 The owner of the symlink and directories if ``makedirs`` is True. If
1716 this is not passed, ``user`` will be used. If ``user`` is not passed,
1717 the account under which Salt is running will be used.
1718
1719 .. versionadded:: 2017.7.7
1720
1721 win_perms : None
1722 A dictionary containing permissions to grant
1723
1724 .. versionadded:: 2017.7.7
1725
1726 win_deny_perms : None
1727 A dictionary containing permissions to deny
1728
1729 .. versionadded:: 2017.7.7
1730
1731 win_inheritance : None
1732 True to inherit permissions from parent, otherwise False
1733
1734 .. versionadded:: 2017.7.7
1735 """
1736 name = os.path.expanduser(name)
1737
1738 # Make sure that leading zeros stripped by YAML loader are added back
1739 mode = salt.utils.files.normalize_mode(mode)
1740
1741 user = _test_owner(kwargs, user=user)
1742 ret = {"name": name, "changes": {}, "result": True, "comment": ""}
1743 if not name:
1744 return _error(ret, "Must provide name to file.symlink")
1745
1746 if user is None:
1747 user = __opts__["user"]
1748
1749 if salt.utils.platform.is_windows():
1750
1751 # Make sure the user exists in Windows
1752 # Salt default is 'root'
1753 if not __salt__["user.info"](user):
1754 # User not found, use the account salt is running under
1755 # If username not found, use System
1756 user = __salt__["user.current"]()
1757 if not user:
1758 user = "SYSTEM"
1759
1760 # If win_owner is not passed, use user
1761 if win_owner is None:
1762 win_owner = user if user else None
1763
1764 # Group isn't relevant to Windows, use win_perms/win_deny_perms
1765 if group is not None:
1766 log.warning(
1767 "The group argument for {} has been ignored as this "
1768 "is a Windows system. Please use the `win_*` parameters to set "
1769 "permissions in Windows.".format(name)
1770 )
1771 group = user
1772
1773 if group is None:
1774 group = __salt__["file.gid_to_group"](__salt__["user.info"](user).get("gid", 0))
1775
1776 preflight_errors = []
1777 if salt.utils.platform.is_windows():
1778 # Make sure the passed owner exists
1779 try:
1780 salt.utils.win_functions.get_sid_from_name(win_owner)
1781 except CommandExecutionError as exc:
1782 preflight_errors.append("User {} does not exist".format(win_owner))
1783
1784 # Make sure users passed in win_perms exist
1785 if win_perms:
1786 for name_check in win_perms:
1787 try:
1788 salt.utils.win_functions.get_sid_from_name(name_check)
1789 except CommandExecutionError as exc:
1790 preflight_errors.append("User {} does not exist".format(name_check))
1791
1792 # Make sure users passed in win_deny_perms exist
1793 if win_deny_perms:
1794 for name_check in win_deny_perms:
1795 try:
1796 salt.utils.win_functions.get_sid_from_name(name_check)
1797 except CommandExecutionError as exc:
1798 preflight_errors.append("User {} does not exist".format(name_check))
1799 else:
1800 uid = __salt__["file.user_to_uid"](user)
1801 gid = __salt__["file.group_to_gid"](group)
1802
1803 if uid == "":
1804 preflight_errors.append("User {} does not exist".format(user))
1805
1806 if gid == "":
1807 preflight_errors.append("Group {} does not exist".format(group))
1808
1809 if not os.path.isabs(name):
1810 preflight_errors.append(
1811 "Specified file {} is not an absolute path".format(name)
1812 )
1813
1814 if preflight_errors:
1815 msg = ". ".join(preflight_errors)
1816 if len(preflight_errors) > 1:
1817 msg += "."
1818 return _error(ret, msg)
1819
1820 tresult, tcomment, tchanges = _symlink_check(
1821 name, target, force, user, group, win_owner
1822 )
1823
1824 if not os.path.isdir(os.path.dirname(name)):
1825 if makedirs:
1826 if __opts__["test"]:
1827 tcomment += "\n{} will be created".format(os.path.dirname(name))
1828 else:
1829 try:
1830 _makedirs(
1831 name=name,
1832 user=user,
1833 group=group,
1834 dir_mode=mode,
1835 win_owner=win_owner,
1836 win_perms=win_perms,
1837 win_deny_perms=win_deny_perms,
1838 win_inheritance=win_inheritance,
1839 )
1840 except CommandExecutionError as exc:
1841 return _error(ret, "Drive {} is not mapped".format(exc.message))
1842 else:
1843 if __opts__["test"]:
1844 tcomment += "\nDirectory {} for symlink is not present" "".format(
1845 os.path.dirname(name)
1846 )
1847 else:
1848 return _error(
1849 ret,
1850 "Directory {} for symlink is not present".format(
1851 os.path.dirname(name)
1852 ),
1853 )
1854
1855 if __opts__["test"]:
1856 ret["result"] = tresult
1857 ret["comment"] = tcomment
1858 ret["changes"] = tchanges
1859 return ret
1860
1861 if __salt__["file.is_link"](name):
1862 # The link exists, verify that it matches the target
1863 if os.path.normpath(__salt__["file.readlink"](name)) != os.path.normpath(
1864 target
1865 ):
1866 # The target is wrong, delete the link
1867 os.remove(name)
1868 else:
1869 if _check_symlink_ownership(name, user, group, win_owner):
1870 # The link looks good!
1871 if salt.utils.platform.is_windows():
1872 ret["comment"] = "Symlink {} is present and owned by {}" "".format(
1873 name, win_owner
1874 )
1875 else:
1876 ret["comment"] = (
1877 "Symlink {} is present and owned by "
1878 "{}:{}".format(name, user, group)
1879 )
1880 else:
1881 if _set_symlink_ownership(name, user, group, win_owner):
1882 if salt.utils.platform.is_windows():
1883 ret["comment"] = "Set ownership of symlink {} to " "{}".format(
1884 name, win_owner
1885 )
1886 ret["changes"]["ownership"] = win_owner
1887 else:
1888 ret["comment"] = (
1889 "Set ownership of symlink {} to "
1890 "{}:{}".format(name, user, group)
1891 )
1892 ret["changes"]["ownership"] = "{}:{}".format(user, group)
1893 else:
1894 ret["result"] = False
1895 if salt.utils.platform.is_windows():
1896 ret["comment"] += (
1897 "Failed to set ownership of symlink "
1898 "{} to {}".format(name, win_owner)
1899 )
1900 else:
1901 ret["comment"] += (
1902 "Failed to set ownership of symlink {} to "
1903 "{}:{}".format(name, user, group)
1904 )
1905 return ret
1906
1907 elif os.path.exists(name):
1908 # It is not a link, but a file, dir, socket, FIFO etc.
1909 if backupname is not None:
1910 if not os.path.isabs(backupname):
1911 if backupname == os.path.basename(backupname):
1912 backupname = os.path.join(
1913 os.path.dirname(os.path.normpath(name)), backupname
1914 )
1915 else:
1916 return _error(
1917 ret,
1918 (
1919 (
1920 "Backupname must be an absolute path "
1921 "or a file name: {}"
1922 ).format(backupname)
1923 ),
1924 )
1925 # Make a backup first
1926 if os.path.lexists(backupname):
1927 if not force:
1928 return _error(
1929 ret,
1930 (
1931 (
1932 "Symlink & backup dest exists and Force not set."
1933 " {} -> {} - backup: {}"
1934 ).format(name, target, backupname)
1935 ),
1936 )
1937 else:
1938 __salt__["file.remove"](backupname)
1939 try:
1940 __salt__["file.move"](name, backupname)
1941 except Exception as exc: # pylint: disable=broad-except
1942 ret["changes"] = {}
1943 log.debug(
1944 "Encountered error renaming %s to %s",
1945 name,
1946 backupname,
1947 exc_info=True,
1948 )
1949 return _error(
1950 ret,
1951 (
1952 "Unable to rename {} to backup {} -> "
1953 ": {}".format(name, backupname, exc)
1954 ),
1955 )
1956 elif force:
1957 # Remove whatever is in the way
1958 if __salt__["file.is_link"](name):
1959 __salt__["file.remove"](name)
1960 ret["changes"]["forced"] = "Symlink was forcibly replaced"
1961 else:
1962 __salt__["file.remove"](name)
1963 else:
1964 # Otherwise throw an error
1965 fs_entry_type = (
1966 "File"
1967 if os.path.isfile(name)
1968 else "Directory"
1969 if os.path.isdir(name)
1970 else "File system entry"
1971 )
1972 return _error(
1973 ret,
1974 (
1975 "{} exists where the symlink {} should be".format(
1976 fs_entry_type, name
1977 )
1978 ),
1979 )
1980
1981 if not os.path.exists(name):
1982 # The link is not present, make it
1983 try:
1984 __salt__["file.symlink"](target, name)
1985 except OSError as exc:
1986 ret["result"] = False
1987 ret["comment"] = "Unable to create new symlink {} -> " "{}: {}".format(
1988 name, target, exc
1989 )
1990 return ret
1991 else:
1992 ret["comment"] = "Created new symlink {} -> " "{}".format(name, target)
1993 ret["changes"]["new"] = name
1994
1995 if not _check_symlink_ownership(name, user, group, win_owner):
1996 if not _set_symlink_ownership(name, user, group, win_owner):
1997 ret["result"] = False
1998 ret[
1999 "comment"
2000 ] += ", but was unable to set ownership to " "{}:{}".format(user, group)
2001 return ret
2002
2003
2004 def absent(name, **kwargs):
2005 """
2006 Make sure that the named file or directory is absent. If it exists, it will
2007 be deleted. This will work to reverse any of the functions in the file
2008 state module. If a directory is supplied, it will be recursively deleted.
2009
2010 name
2011 The path which should be deleted
2012 """
2013 name = os.path.expanduser(name)
2014
2015 ret = {"name": name, "changes": {}, "result": True, "comment": ""}
2016 if not name:
2017 return _error(ret, "Must provide name to file.absent")
2018 if not os.path.isabs(name):
2019 return _error(ret, "Specified file {} is not an absolute path".format(name))
2020 if name == "/":
2021 return _error(ret, 'Refusing to make "/" absent')
2022 if os.path.isfile(name) or os.path.islink(name):
2023 if __opts__["test"]:
2024 ret["result"] = None
2025 ret["changes"]["removed"] = name
2026 ret["comment"] = "File {} is set for removal".format(name)
2027 return ret
2028 try:
2029 if salt.utils.platform.is_windows():
2030 __salt__["file.remove"](name, force=True)
2031 else:
2032 __salt__["file.remove"](name)
2033 ret["comment"] = "Removed file {}".format(name)
2034 ret["changes"]["removed"] = name
2035 return ret
2036 except CommandExecutionError as exc:
2037 return _error(ret, "{}".format(exc))
2038
2039 elif os.path.isdir(name):
2040 if __opts__["test"]:
2041 ret["result"] = None
2042 ret["changes"]["removed"] = name
2043 ret["comment"] = "Directory {} is set for removal".format(name)
2044 return ret
2045 try:
2046 if salt.utils.platform.is_windows():
2047 __salt__["file.remove"](name, force=True)
2048 else:
2049 __salt__["file.remove"](name)
2050 ret["comment"] = "Removed directory {}".format(name)
2051 ret["changes"]["removed"] = name
2052 return ret
2053 except OSError:
2054 return _error(ret, "Failed to remove directory {}".format(name))
2055
2056 ret["comment"] = "File {} is not present".format(name)
2057 return ret
2058
2059
2060 def tidied(name, age=0, matches=None, rmdirs=False, size=0, **kwargs):
2061 """
2062 Remove unwanted files based on specific criteria. Multiple criteria
2063 are OR’d together, so a file that is too large but is not old enough
2064 will still get tidied.
2065
2066 If neither age nor size is given all files which match a pattern in
2067 matches will be removed.
2068
2069 name
2070 The directory tree that should be tidied
2071
2072 age
2073 Maximum age in days after which files are considered for removal
2074
2075 matches
2076 List of regular expressions to restrict what gets removed. Default: ['.*']
2077
2078 rmdirs
2079 Whether or not it's allowed to remove directories
2080
2081 size
2082 Maximum allowed file size. Files greater or equal to this size are
2083 removed. Doesn't apply to directories or symbolic links
2084
2085 .. code-block:: yaml
2086
2087 cleanup:
2088 file.tidied:
2089 - name: /tmp/salt_test
2090 - rmdirs: True
2091 - matches:
2092 - foo
2093 - b.*r
2094 """
2095 name = os.path.expanduser(name)
2096
2097 ret = {"name": name, "changes": {}, "result": True, "comment": ""}
2098
2099 # Check preconditions
2100 if not os.path.isabs(name):
2101 return _error(ret, "Specified file {} is not an absolute path".format(name))
2102 if not os.path.isdir(name):
2103 return _error(ret, "{} does not exist or is not a directory.".format(name))
2104
2105 # Define some variables
2106 todelete = []
2107 today = date.today()
2108
2109 # Compile regular expressions
2110 if matches is None:
2111 matches = [".*"]
2112 progs = []
2113 for regex in matches:
2114 progs.append(re.compile(regex))
2115
2116 # Helper to match a given name against one or more pre-compiled regular
2117 # expressions
2118 def _matches(name):
2119 for prog in progs:
2120 if prog.match(name):
2121 return True
2122 return False
2123
2124 # Iterate over given directory tree, depth-first
2125 for root, dirs, files in os.walk(top=name, topdown=False):
2126 # Check criteria for the found files and directories
2127 for elem in files + dirs:
2128 myage = 0
2129 mysize = 0
2130 deleteme = True
2131 path = os.path.join(root, elem)
2132 if os.path.islink(path):
2133 # Get age of symlink (not symlinked file)
2134 myage = abs(today - date.fromtimestamp(os.lstat(path).st_atime))
2135 elif elem in dirs:
2136 # Get age of directory, check if directories should be deleted at all
2137 myage = abs(today - date.fromtimestamp(os.path.getatime(path)))
2138 deleteme = rmdirs
2139 else:
2140 # Get age and size of regular file
2141 myage = abs(today - date.fromtimestamp(os.path.getatime(path)))
2142 mysize = os.path.getsize(path)
2143 # Verify against given criteria, collect all elements that should be removed
2144 if (
2145 (mysize >= size or myage.days >= age)
2146 and _matches(name=elem)
2147 and deleteme
2148 ):
2149 todelete.append(path)
2150
2151 # Now delete the stuff
2152 if todelete:
2153 if __opts__["test"]:
2154 ret["result"] = None
2155 ret["comment"] = "{} is set for tidy".format(name)
2156 ret["changes"] = {"removed": todelete}
2157 return ret
2158 ret["changes"]["removed"] = []
2159 # Iterate over collected items
2160 try:
2161 for path in todelete:
2162 if salt.utils.platform.is_windows():
2163 __salt__["file.remove"](path, force=True)
2164 else:
2165 __salt__["file.remove"](path)
2166 # Remember what we've removed, will appear in the summary
2167 ret["changes"]["removed"].append(path)
2168 except CommandExecutionError as exc:
2169 return _error(ret, "{}".format(exc))
2170 # Set comment for the summary
2171 ret["comment"] = "Removed {} files or directories from directory {}".format(
2172 len(todelete), name
2173 )
2174 else:
2175 # Set comment in case there was nothing to remove
2176 ret["comment"] = "Nothing to remove from directory {}".format(name)
2177 return ret
2178
2179
2180 def exists(name, **kwargs):
2181 """
2182 Verify that the named file or directory is present or exists.
2183 Ensures pre-requisites outside of Salt's purview
2184 (e.g., keytabs, private keys, etc.) have been previously satisfied before
2185 deployment.
2186
2187 This function does not create the file if it doesn't exist, it will return
2188 an error.
2189
2190 name
2191 Absolute path which must exist
2192 """
2193 name = os.path.expanduser(name)
2194
2195 ret = {"name": name, "changes": {}, "result": True, "comment": ""}
2196 if not name:
2197 return _error(ret, "Must provide name to file.exists")
2198 if not os.path.exists(name):
2199 return _error(ret, "Specified path {} does not exist".format(name))
2200
2201 ret["comment"] = "Path {} exists".format(name)
2202 return ret
2203
2204
2205 def missing(name, **kwargs):
2206 """
2207 Verify that the named file or directory is missing, this returns True only
2208 if the named file is missing but does not remove the file if it is present.
2209
2210 name
2211 Absolute path which must NOT exist
2212 """
2213 name = os.path.expanduser(name)
2214
2215 ret = {"name": name, "changes": {}, "result": True, "comment": ""}
2216 if not name:
2217 return _error(ret, "Must provide name to file.missing")
2218 if os.path.exists(name):
2219 return _error(ret, "Specified path {} exists".format(name))
2220
2221 ret["comment"] = "Path {} is missing".format(name)
2222 return ret
2223
2224
2225 def managed(
2226 name,
2227 source=None,
2228 source_hash="",
2229 source_hash_name=None,
2230 keep_source=True,
2231 user=None,
2232 group=None,
2233 mode=None,
2234 attrs=None,
2235 template=None,
2236 makedirs=False,
2237 dir_mode=None,
2238 context=None,
2239 replace=True,
2240 defaults=None,
2241 backup="",
2242 show_changes=True,
2243 create=True,
2244 contents=None,
2245 tmp_dir="",
2246 tmp_ext="",
2247 contents_pillar=None,
2248 contents_grains=None,
2249 contents_newline=True,
2250 contents_delimiter=":",
2251 encoding=None,
2252 encoding_errors="strict",
2253 allow_empty=True,
2254 follow_symlinks=True,
2255 check_cmd=None,
2256 skip_verify=False,
2257 selinux=None,
2258 win_owner=None,
2259 win_perms=None,
2260 win_deny_perms=None,
2261 win_inheritance=True,
2262 win_perms_reset=False,
2263 verify_ssl=True,
2264 **kwargs
2265 ):
2266 r"""
2267 Manage a given file, this function allows for a file to be downloaded from
2268 the salt master and potentially run through a templating system.
2269
2270 name
2271 The location of the file to manage, as an absolute path.
2272
2273 source
2274 The source file to download to the minion, this source file can be
2275 hosted on either the salt master server (``salt://``), the salt minion
2276 local file system (``/``), or on an HTTP or FTP server (``http(s)://``,
2277 ``ftp://``).
2278
2279 Both HTTPS and HTTP are supported as well as downloading directly
2280 from Amazon S3 compatible URLs with both pre-configured and automatic
2281 IAM credentials. (see s3.get state documentation)
2282 File retrieval from Openstack Swift object storage is supported via
2283 swift://container/object_path URLs, see swift.get documentation.
2284 For files hosted on the salt file server, if the file is located on
2285 the master in the directory named spam, and is called eggs, the source
2286 string is salt://spam/eggs. If source is left blank or None
2287 (use ~ in YAML), the file will be created as an empty file and
2288 the content will not be managed. This is also the case when a file
2289 already exists and the source is undefined; the contents of the file
2290 will not be changed or managed. If source is left blank or None, please
2291 also set replaced to False to make your intention explicit.
2292
2293
2294 If the file is hosted on a HTTP or FTP server then the source_hash
2295 argument is also required.
2296
2297 A list of sources can also be passed in to provide a default source and
2298 a set of fallbacks. The first source in the list that is found to exist
2299 will be used and subsequent entries in the list will be ignored. Source
2300 list functionality only supports local files and remote files hosted on
2301 the salt master server or retrievable via HTTP, HTTPS, or FTP.
2302
2303 .. code-block:: yaml
2304
2305 file_override_example:
2306 file.managed:
2307 - source:
2308 - salt://file_that_does_not_exist
2309 - salt://file_that_exists
2310
2311 source_hash
2312 This can be one of the following:
2313 1. a source hash string
2314 2. the URI of a file that contains source hash strings
2315
2316 The function accepts the first encountered long unbroken alphanumeric
2317 string of correct length as a valid hash, in order from most secure to
2318 least secure:
2319
2320 .. code-block:: text
2321
2322 Type Length
2323 ====== ======
2324 sha512 128
2325 sha384 96
2326 sha256 64
2327 sha224 56
2328 sha1 40
2329 md5 32
2330
2331 **Using a Source Hash File**
2332 The file can contain several checksums for several files. Each line
2333 must contain both the file name and the hash. If no file name is
2334 matched, the first hash encountered will be used, otherwise the most
2335 secure hash with the correct source file name will be used.
2336
2337 When using a source hash file the source_hash argument needs to be a
2338 url, the standard download urls are supported, ftp, http, salt etc:
2339
2340 Example:
2341
2342 .. code-block:: yaml
2343
2344 tomdroid-src-0.7.3.tar.gz:
2345 file.managed:
2346 - name: /tmp/tomdroid-src-0.7.3.tar.gz
2347 - source: https://launchpad.net/tomdroid/beta/0.7.3/+download/tomdroid-src-0.7.3.tar.gz
2348 - source_hash: https://launchpad.net/tomdroid/beta/0.7.3/+download/tomdroid-src-0.7.3.hash
2349
2350 The following lines are all supported formats:
2351
2352 .. code-block:: text
2353
2354 /etc/rc.conf ef6e82e4006dee563d98ada2a2a80a27
2355 sha254c8525aee419eb649f0233be91c151178b30f0dff8ebbdcc8de71b1d5c8bcc06a /etc/resolv.conf
2356 ead48423703509d37c4a90e6a0d53e143b6fc268
2357
2358 Debian file type ``*.dsc`` files are also supported.
2359
2360 **Inserting the Source Hash in the SLS Data**
2361
2362 The source_hash can be specified as a simple checksum, like so:
2363
2364 .. code-block:: yaml
2365
2366 tomdroid-src-0.7.3.tar.gz:
2367 file.managed:
2368 - name: /tmp/tomdroid-src-0.7.3.tar.gz
2369 - source: https://launchpad.net/tomdroid/beta/0.7.3/+download/tomdroid-src-0.7.3.tar.gz
2370 - source_hash: 79eef25f9b0b2c642c62b7f737d4f53f
2371
2372 .. note::
2373 Releases prior to 2016.11.0 must also include the hash type, like
2374 in the below example:
2375
2376 .. code-block:: yaml
2377
2378 tomdroid-src-0.7.3.tar.gz:
2379 file.managed:
2380 - name: /tmp/tomdroid-src-0.7.3.tar.gz
2381 - source: https://launchpad.net/tomdroid/beta/0.7.3/+download/tomdroid-src-0.7.3.tar.gz
2382 - source_hash: md5=79eef25f9b0b2c642c62b7f737d4f53f
2383
2384 Known issues:
2385 If the remote server URL has the hash file as an apparent
2386 sub-directory of the source file, the module will discover that it
2387 has already cached a directory where a file should be cached. For
2388 example:
2389
2390 .. code-block:: yaml
2391
2392 tomdroid-src-0.7.3.tar.gz:
2393 file.managed:
2394 - name: /tmp/tomdroid-src-0.7.3.tar.gz
2395 - source: https://launchpad.net/tomdroid/beta/0.7.3/+download/tomdroid-src-0.7.3.tar.gz
2396 - source_hash: https://launchpad.net/tomdroid/beta/0.7.3/+download/tomdroid-src-0.7.3.tar.gz/+md5
2397
2398 source_hash_name
2399 When ``source_hash`` refers to a hash file, Salt will try to find the
2400 correct hash by matching the filename/URI associated with that hash. By
2401 default, Salt will look for the filename being managed. When managing a
2402 file at path ``/tmp/foo.txt``, then the following line in a hash file
2403 would match:
2404
2405 .. code-block:: text
2406
2407 acbd18db4cc2f85cedef654fccc4a4d8 foo.txt
2408
2409 However, sometimes a hash file will include multiple similar paths:
2410
2411 .. code-block:: text
2412
2413 37b51d194a7513e45b56f6524f2d51f2 ./dir1/foo.txt
2414 acbd18db4cc2f85cedef654fccc4a4d8 ./dir2/foo.txt
2415 73feffa4b7f6bb68e44cf984c85f6e88 ./dir3/foo.txt
2416
2417 In cases like this, Salt may match the incorrect hash. This argument
2418 can be used to tell Salt which filename to match, to ensure that the
2419 correct hash is identified. For example:
2420
2421 .. code-block:: yaml
2422
2423 /tmp/foo.txt:
2424 file.managed:
2425 - source: https://mydomain.tld/dir2/foo.txt
2426 - source_hash: https://mydomain.tld/hashes
2427 - source_hash_name: ./dir2/foo.txt
2428
2429 .. note::
2430 This argument must contain the full filename entry from the
2431 checksum file, as this argument is meant to disambiguate matches
2432 for multiple files that have the same basename. So, in the
2433 example above, simply using ``foo.txt`` would not match.
2434
2435 .. versionadded:: 2016.3.5
2436
2437 keep_source : True
2438 Set to ``False`` to discard the cached copy of the source file once the
2439 state completes. This can be useful for larger files to keep them from
2440 taking up space in minion cache. However, keep in mind that discarding
2441 the source file will result in the state needing to re-download the
2442 source file if the state is run again.
2443
2444 .. versionadded:: 2017.7.3
2445
2446 user
2447 The user to own the file, this defaults to the user salt is running as
2448 on the minion
2449
2450 group
2451 The group ownership set for the file, this defaults to the group salt
2452 is running as on the minion. On Windows, this is ignored
2453
2454 mode
2455 The permissions to set on this file, e.g. ``644``, ``0775``, or
2456 ``4664``.
2457
2458 The default mode for new files and directories corresponds to the
2459 umask of the salt process. The mode of existing files and directories
2460 will only be changed if ``mode`` is specified.
2461
2462 .. note::
2463 This option is **not** supported on Windows.
2464
2465 .. versionchanged:: 2016.11.0
2466 This option can be set to ``keep``, and Salt will keep the mode
2467 from the Salt fileserver. This is only supported when the
2468 ``source`` URL begins with ``salt://``, or for files local to the
2469 minion. Because the ``source`` option cannot be used with any of
2470 the ``contents`` options, setting the ``mode`` to ``keep`` is also
2471 incompatible with the ``contents`` options.
2472
2473 .. note:: keep does not work with salt-ssh.
2474
2475 As a consequence of how the files are transferred to the minion, and
2476 the inability to connect back to the master with salt-ssh, salt is
2477 unable to stat the file as it exists on the fileserver and thus
2478 cannot mirror the mode on the salt-ssh minion
2479
2480 attrs
2481 The attributes to have on this file, e.g. ``a``, ``i``. The attributes
2482 can be any or a combination of the following characters:
2483 ``aAcCdDeijPsStTu``.
2484
2485 .. note::
2486 This option is **not** supported on Windows.
2487
2488 .. versionadded:: 2018.3.0
2489
2490 template
2491 If this setting is applied, the named templating engine will be used to
2492 render the downloaded file. The following templates are supported:
2493
2494 - :mod:`cheetah<salt.renderers.cheetah>`
2495 - :mod:`genshi<salt.renderers.genshi>`
2496 - :mod:`jinja<salt.renderers.jinja>`
2497 - :mod:`mako<salt.renderers.mako>`
2498 - :mod:`py<salt.renderers.py>`
2499 - :mod:`wempy<salt.renderers.wempy>`
2500
2501 makedirs : False
2502 If set to ``True``, then the parent directories will be created to
2503 facilitate the creation of the named file. If ``False``, and the parent
2504 directory of the destination file doesn't exist, the state will fail.
2505
2506 dir_mode
2507 If directories are to be created, passing this option specifies the
2508 permissions for those directories. If this is not set, directories
2509 will be assigned permissions by adding the execute bit to the mode of
2510 the files.
2511
2512 The default mode for new files and directories corresponds umask of salt
2513 process. For existing files and directories it's not enforced.
2514
2515 replace : True
2516 If set to ``False`` and the file already exists, the file will not be
2517 modified even if changes would otherwise be made. Permissions and
2518 ownership will still be enforced, however.
2519
2520 context
2521 Overrides default context variables passed to the template.
2522
2523 defaults
2524 Default context passed to the template.
2525
2526 backup
2527 Overrides the default backup mode for this specific file. See
2528 :ref:`backup_mode documentation <file-state-backups>` for more details.
2529
2530 show_changes
2531 Output a unified diff of the old file and the new file. If ``False``
2532 return a boolean if any changes were made.
2533
2534 create : True
2535 If set to ``False``, then the file will only be managed if the file
2536 already exists on the system.
2537
2538 contents
2539 Specify the contents of the file. Cannot be used in combination with
2540 ``source``. Ignores hashes and does not use a templating engine.
2541
2542 This value can be either a single string, a multiline YAML string or a
2543 list of strings. If a list of strings, then the strings will be joined
2544 together with newlines in the resulting file. For example, the below
2545 two example states would result in identical file contents:
2546
2547 .. code-block:: yaml
2548
2549 /path/to/file1:
2550 file.managed:
2551 - contents:
2552 - This is line 1
2553 - This is line 2
2554
2555 /path/to/file2:
2556 file.managed:
2557 - contents: |
2558 This is line 1
2559 This is line 2
2560
2561
2562 contents_pillar
2563 .. versionadded:: 0.17.0
2564 .. versionchanged:: 2016.11.0
2565 contents_pillar can also be a list, and the pillars will be
2566 concatenated together to form one file.
2567
2568
2569 Operates like ``contents``, but draws from a value stored in pillar,
2570 using the pillar path syntax used in :mod:`pillar.get
2571 <salt.modules.pillar.get>`. This is useful when the pillar value
2572 contains newlines, as referencing a pillar variable using a jinja/mako
2573 template can result in YAML formatting issues due to the newlines
2574 causing indentation mismatches.
2575
2576 For example, the following could be used to deploy an SSH private key:
2577
2578 .. code-block:: yaml
2579
2580 /home/deployer/.ssh/id_rsa:
2581 file.managed:
2582 - user: deployer
2583 - group: deployer
2584 - mode: 600
2585 - attrs: a
2586 - contents_pillar: userdata:deployer:id_rsa
2587
2588 This would populate ``/home/deployer/.ssh/id_rsa`` with the contents of
2589 ``pillar['userdata']['deployer']['id_rsa']``. An example of this pillar
2590 setup would be like so:
2591
2592 .. code-block:: yaml
2593
2594 userdata:
2595 deployer:
2596 id_rsa: |
2597 -----BEGIN RSA PRIVATE KEY-----
2598 MIIEowIBAAKCAQEAoQiwO3JhBquPAalQF9qP1lLZNXVjYMIswrMe2HcWUVBgh+vY
2599 U7sCwx/dH6+VvNwmCoqmNnP+8gTPKGl1vgAObJAnMT623dMXjVKwnEagZPRJIxDy
2600 B/HaAre9euNiY3LvIzBTWRSeMfT+rWvIKVBpvwlgGrfgz70m0pqxu+UyFbAGLin+
2601 GpxzZAMaFpZw4sSbIlRuissXZj/sHpQb8p9M5IeO4Z3rjkCP1cxI
2602 -----END RSA PRIVATE KEY-----
2603
2604 .. note::
2605 The private key above is shortened to keep the example brief, but
2606 shows how to do multiline string in YAML. The key is followed by a
2607 pipe character, and the multiline string is indented two more
2608 spaces.
2609
2610 To avoid the hassle of creating an indented multiline YAML string,
2611 the :mod:`file_tree external pillar <salt.pillar.file_tree>` can
2612 be used instead. However, this will not work for binary files in
2613 Salt releases before 2015.8.4.
2614
2615 contents_grains
2616 .. versionadded:: 2014.7.0
2617
2618 Operates like ``contents``, but draws from a value stored in grains,
2619 using the grains path syntax used in :mod:`grains.get
2620 <salt.modules.grains.get>`. This functionality works similarly to
2621 ``contents_pillar``, but with grains.
2622
2623 For example, the following could be used to deploy a "message of the day"
2624 file:
2625
2626 .. code-block:: yaml
2627
2628 write_motd:
2629 file.managed:
2630 - name: /etc/motd
2631 - contents_grains: motd
2632
2633 This would populate ``/etc/motd`` file with the contents of the ``motd``
2634 grain. The ``motd`` grain is not a default grain, and would need to be
2635 set prior to running the state:
2636
2637 .. code-block:: bash
2638
2639 salt '*' grains.set motd 'Welcome! This system is managed by Salt.'
2640
2641 contents_newline : True
2642 .. versionadded:: 2014.7.0
2643 .. versionchanged:: 2015.8.4
2644 This option is now ignored if the contents being deployed contain
2645 binary data.
2646
2647 If ``True``, files managed using ``contents``, ``contents_pillar``, or
2648 ``contents_grains`` will have a newline added to the end of the file if
2649 one is not present. Setting this option to ``False`` will ensure the
2650 final line, or entry, does not contain a new line. If the last line, or
2651 entry in the file does contain a new line already, this option will not
2652 remove it.
2653
2654 contents_delimiter
2655 .. versionadded:: 2015.8.4
2656
2657 Can be used to specify an alternate delimiter for ``contents_pillar``
2658 or ``contents_grains``. This delimiter will be passed through to
2659 :py:func:`pillar.get <salt.modules.pillar.get>` or :py:func:`grains.get
2660 <salt.modules.grains.get>` when retrieving the contents.
2661
2662 encoding
2663 If specified, then the specified encoding will be used. Otherwise, the
2664 file will be encoded using the system locale (usually UTF-8). See
2665 https://docs.python.org/3/library/codecs.html#standard-encodings for
2666 the list of available encodings.
2667
2668 .. versionadded:: 2017.7.0
2669
2670 encoding_errors : 'strict'
2671 Error encoding scheme. Default is ```'strict'```.
2672 See https://docs.python.org/2/library/codecs.html#codec-base-classes
2673 for the list of available schemes.
2674
2675 .. versionadded:: 2017.7.0
2676
2677 allow_empty : True
2678 .. versionadded:: 2015.8.4
2679
2680 If set to ``False``, then the state will fail if the contents specified
2681 by ``contents_pillar`` or ``contents_grains`` are empty.
2682
2683 follow_symlinks : True
2684 .. versionadded:: 2014.7.0
2685
2686 If the desired path is a symlink follow it and make changes to the
2687 file to which the symlink points.
2688
2689 check_cmd
2690 .. versionadded:: 2014.7.0
2691
2692 The specified command will be run with an appended argument of a
2693 *temporary* file containing the new managed contents. If the command
2694 exits with a zero status the new managed contents will be written to
2695 the managed destination. If the command exits with a nonzero exit
2696 code, the state will fail and no changes will be made to the file.
2697
2698 For example, the following could be used to verify sudoers before making
2699 changes:
2700
2701 .. code-block:: yaml
2702
2703 /etc/sudoers:
2704 file.managed:
2705 - user: root
2706 - group: root
2707 - mode: 0440
2708 - attrs: i
2709 - source: salt://sudoers/files/sudoers.jinja
2710 - template: jinja
2711 - check_cmd: /usr/sbin/visudo -c -f
2712
2713 **NOTE**: This ``check_cmd`` functions differently than the requisite
2714 ``check_cmd``.
2715
2716 tmp_dir
2717 Directory for temp file created by ``check_cmd``. Useful for checkers
2718 dependent on config file location (e.g. daemons restricted to their
2719 own config directories by an apparmor profile).
2720
2721 .. code-block:: yaml
2722
2723 /etc/dhcp/dhcpd.conf:
2724 file.managed:
2725 - user: root
2726 - group: root
2727 - mode: 0755
2728 - tmp_dir: '/etc/dhcp'
2729 - contents: "# Managed by Salt"
2730 - check_cmd: dhcpd -t -cf
2731
2732 tmp_ext
2733 Suffix for temp file created by ``check_cmd``. Useful for checkers
2734 dependent on config file extension (e.g. the init-checkconf upstart
2735 config checker).
2736
2737 .. code-block:: yaml
2738
2739 /etc/init/test.conf:
2740 file.managed:
2741 - user: root
2742 - group: root
2743 - mode: 0440
2744 - tmp_ext: '.conf'
2745 - contents:
2746 - 'description "Salt Minion"'
2747 - 'start on started mountall'
2748 - 'stop on shutdown'
2749 - 'respawn'
2750 - 'exec salt-minion'
2751 - check_cmd: init-checkconf -f
2752
2753 skip_verify : False
2754 If ``True``, hash verification of remote file sources (``http://``,
2755 ``https://``, ``ftp://``) will be skipped, and the ``source_hash``
2756 argument will be ignored.
2757
2758 .. versionadded:: 2016.3.0
2759
2760 selinux : None
2761 Allows setting the selinux user, role, type, and range of a managed file
2762
2763 .. code-block:: yaml
2764
2765 /tmp/selinux.test
2766 file.managed:
2767 - user: root
2768 - selinux:
2769 seuser: system_u
2770 serole: object_r
2771 setype: system_conf_t
2772 seranage: s0
2773
2774 .. versionadded:: Neon
2775
2776 win_owner : None
2777 The owner of the directory. If this is not passed, user will be used. If
2778 user is not passed, the account under which Salt is running will be
2779 used.
2780
2781 .. versionadded:: 2017.7.0
2782
2783 win_perms : None
2784 A dictionary containing permissions to grant and their propagation. For
2785 example: ``{'Administrators': {'perms': 'full_control'}}`` Can be a
2786 single basic perm or a list of advanced perms. ``perms`` must be
2787 specified. ``applies_to`` does not apply to file objects.
2788
2789 .. versionadded:: 2017.7.0
2790
2791 win_deny_perms : None
2792 A dictionary containing permissions to deny and their propagation. For
2793 example: ``{'Administrators': {'perms': 'full_control'}}`` Can be a
2794 single basic perm or a list of advanced perms. ``perms`` must be
2795 specified. ``applies_to`` does not apply to file objects.
2796
2797 .. versionadded:: 2017.7.0
2798
2799 win_inheritance : True
2800 True to inherit permissions from the parent directory, False not to
2801 inherit permission.
2802
2803 .. versionadded:: 2017.7.0
2804
2805 win_perms_reset : False
2806 If ``True`` the existing DACL will be cleared and replaced with the
2807 settings defined in this function. If ``False``, new entries will be
2808 appended to the existing DACL. Default is ``False``.
2809
2810 .. versionadded:: 2018.3.0
2811
2812 Here's an example using the above ``win_*`` parameters:
2813
2814 .. code-block:: yaml
2815
2816 create_config_file:
2817 file.managed:
2818 - name: C:\config\settings.cfg
2819 - source: salt://settings.cfg
2820 - win_owner: Administrators
2821 - win_perms:
2822 # Basic Permissions
2823 dev_ops:
2824 perms: full_control
2825 # List of advanced permissions
2826 appuser:
2827 perms:
2828 - read_attributes
2829 - read_ea
2830 - create_folders
2831 - read_permissions
2832 joe_snuffy:
2833 perms: read
2834 - win_deny_perms:
2835 fred_snuffy:
2836 perms: full_control
2837 - win_inheritance: False
2838
2839 verify_ssl
2840 If ``False``, remote https file sources (``https://``) and source_hash
2841 will not attempt to validate the servers certificate. Default is True.
2842
2843 .. versionadded:: 3002
2844 """
2845 if "env" in kwargs:
2846 # "env" is not supported; Use "saltenv".
2847 kwargs.pop("env")
2848
2849 name = os.path.expanduser(name)
2850
2851 ret = {"changes": {}, "comment": "", "name": name, "result": True}
2852
2853 if not name:
2854 return _error(ret, "Destination file name is required")
2855
2856 if mode is not None and salt.utils.platform.is_windows():
2857 return _error(ret, "The 'mode' option is not supported on Windows")
2858
2859 if attrs is not None and salt.utils.platform.is_windows():
2860 return _error(ret, "The 'attrs' option is not supported on Windows")
2861
2862 if selinux is not None and not salt.utils.platform.is_linux():
2863 return _error(ret, "The 'selinux' option is only supported on Linux")
2864
2865 if selinux:
2866 seuser = selinux.get("seuser", None)
2867 serole = selinux.get("serole", None)
2868 setype = selinux.get("setype", None)
2869 serange = selinux.get("serange", None)
2870 else:
2871 seuser = serole = setype = serange = None
2872
2873 try:
2874 keep_mode = mode.lower() == "keep"
2875 if keep_mode:
2876 # We're not hard-coding the mode, so set it to None
2877 mode = None
2878 except AttributeError:
2879 keep_mode = False
2880
2881 # Make sure that any leading zeros stripped by YAML loader are added back
2882 mode = salt.utils.files.normalize_mode(mode)
2883
2884 contents_count = len(
2885 [x for x in (contents, contents_pillar, contents_grains) if x is not None]
2886 )
2887
2888 if source and contents_count > 0:
2889 return _error(
2890 ret,
2891 "'source' cannot be used in combination with 'contents', "
2892 "'contents_pillar', or 'contents_grains'",
2893 )
2894 elif keep_mode and contents_count > 0:
2895 return _error(
2896 ret,
2897 "Mode preservation cannot be used in combination with 'contents', "
2898 "'contents_pillar', or 'contents_grains'",
2899 )
2900 elif contents_count > 1:
2901 return _error(
2902 ret,
2903 "Only one of 'contents', 'contents_pillar', and "
2904 "'contents_grains' is permitted",
2905 )
2906
2907 # If no source is specified, set replace to False, as there is nothing
2908 # with which to replace the file.
2909 if not source and contents_count == 0 and replace:
2910 replace = False
2911 log.warning(
2912 "State for file: {} - Neither 'source' nor 'contents' nor "
2913 "'contents_pillar' nor 'contents_grains' was defined, yet "
2914 "'replace' was set to 'True'. As there is no source to "
2915 "replace the file with, 'replace' has been set to 'False' to "
2916 "avoid reading the file unnecessarily.".format(name)
2917 )
2918
2919 if "file_mode" in kwargs:
2920 ret.setdefault("warnings", []).append(
2921 "The 'file_mode' argument will be ignored. "
2922 "Please use 'mode' instead to set file permissions."
2923 )
2924
2925 # Use this below to avoid multiple '\0' checks and save some CPU cycles
2926 if contents_pillar is not None:
2927 if isinstance(contents_pillar, list):
2928 list_contents = []
2929 for nextp in contents_pillar:
2930 nextc = __salt__["pillar.get"](
2931 nextp, __NOT_FOUND, delimiter=contents_delimiter
2932 )
2933 if nextc is __NOT_FOUND:
2934 return _error(ret, "Pillar {} does not exist".format(nextp))
2935 list_contents.append(nextc)
2936 use_contents = os.linesep.join(list_contents)
2937 else:
2938 use_contents = __salt__["pillar.get"](
2939 contents_pillar, __NOT_FOUND, delimiter=contents_delimiter
2940 )
2941 if use_contents is __NOT_FOUND:
2942 return _error(ret, "Pillar {} does not exist".format(contents_pillar))
2943
2944 elif contents_grains is not None:
2945 if isinstance(contents_grains, list):
2946 list_contents = []
2947 for nextg in contents_grains:
2948 nextc = __salt__["grains.get"](
2949 nextg, __NOT_FOUND, delimiter=contents_delimiter
2950 )
2951 if nextc is __NOT_FOUND:
2952 return _error(ret, "Grain {} does not exist".format(nextc))
2953 list_contents.append(nextc)
2954 use_contents = os.linesep.join(list_contents)
2955 else:
2956 use_contents = __salt__["grains.get"](
2957 contents_grains, __NOT_FOUND, delimiter=contents_delimiter
2958 )
2959 if use_contents is __NOT_FOUND:
2960 return _error(ret, "Grain {} does not exist".format(contents_grains))
2961
2962 elif contents is not None:
2963 use_contents = contents
2964
2965 else:
2966 use_contents = None
2967
2968 if use_contents is not None:
2969 if not allow_empty and not use_contents:
2970 if contents_pillar:
2971 contents_id = "contents_pillar {}".format(contents_pillar)
2972 elif contents_grains:
2973 contents_id = "contents_grains {}".format(contents_grains)
2974 else:
2975 contents_id = "'contents'"
2976 return _error(
2977 ret,
2978 "{} value would result in empty contents. Set allow_empty "
2979 "to True to allow the managed file to be empty.".format(contents_id),
2980 )
2981
2982 try:
2983 validated_contents = _validate_str_list(use_contents, encoding=encoding)
2984 if not validated_contents:
2985 return _error(
2986 ret,
2987 "Contents specified by contents/contents_pillar/"
2988 "contents_grains is not a string or list of strings, and "
2989 "is not binary data. SLS is likely malformed.",
2990 )
2991 contents = ""
2992 for part in validated_contents:
2993 for line in part.splitlines():
2994 contents += line.rstrip("\n").rstrip("\r") + os.linesep
2995 if not contents_newline:
2996 # If contents newline is set to False, strip out the newline
2997 # character and carriage return character
2998 contents = contents.rstrip("\n").rstrip("\r")
2999
3000 except UnicodeDecodeError:
3001 # Either something terrible happened, or we have binary data.
3002 if template:
3003 return _error(
3004 ret,
3005 "Contents specified by contents/contents_pillar/"
3006 "contents_grains appears to be binary data, and"
3007 " as will not be able to be treated as a Jinja"
3008 " template.",
3009 )
3010 contents = use_contents
3011 if template:
3012 contents = __salt__["file.apply_template_on_contents"](
3013 contents,
3014 template=template,
3015 context=context,
3016 defaults=defaults,
3017 saltenv=__env__,
3018 )
3019 if not isinstance(contents, str):
3020 if "result" in contents:
3021 ret["result"] = contents["result"]
3022 else:
3023 ret["result"] = False
3024 if "comment" in contents:
3025 ret["comment"] = contents["comment"]
3026 else:
3027 ret["comment"] = "Error while applying template on contents"
3028 return ret
3029
3030 user = _test_owner(kwargs, user=user)
3031 if salt.utils.platform.is_windows():
3032
3033 # If win_owner not passed, use user
3034 if win_owner is None:
3035 win_owner = user if user else None
3036
3037 # Group isn't relevant to Windows, use win_perms/win_deny_perms
3038 if group is not None:
3039 log.warning(
3040 "The group argument for {} has been ignored as this is "
3041 "a Windows system. Please use the `win_*` parameters to set "
3042 "permissions in Windows.".format(name)
3043 )
3044 group = user
3045
3046 if not create:
3047 if not os.path.isfile(name):
3048 # Don't create a file that is not already present
3049 ret["comment"] = (
3050 "File {} is not present and is not set for " "creation"
3051 ).format(name)
3052 return ret
3053 u_check = _check_user(user, group)
3054 if u_check:
3055 # The specified user or group do not exist
3056 return _error(ret, u_check)
3057 if not os.path.isabs(name):
3058 return _error(ret, "Specified file {} is not an absolute path".format(name))
3059
3060 if os.path.isdir(name):
3061 ret["comment"] = "Specified target {} is a directory".format(name)
3062 ret["result"] = False
3063 return ret
3064
3065 if context is None:
3066 context = {}
3067 elif not isinstance(context, dict):
3068 return _error(ret, "Context must be formed as a dict")
3069 if defaults and not isinstance(defaults, dict):
3070 return _error(ret, "Defaults must be formed as a dict")
3071
3072 if not replace and os.path.exists(name):
3073 ret_perms = {}
3074 # Check and set the permissions if necessary
3075 if salt.utils.platform.is_windows():
3076 ret = __salt__["file.check_perms"](
3077 path=name,
3078 ret=ret,
3079 owner=win_owner,
3080 grant_perms=win_perms,
3081 deny_perms=win_deny_perms,
3082 inheritance=win_inheritance,
3083 reset=win_perms_reset,
3084 )
3085 else:
3086 ret, ret_perms = __salt__["file.check_perms"](
3087 name,
3088 ret,
3089 user,
3090 group,
3091 mode,
3092 attrs,
3093 follow_symlinks,
3094 seuser=seuser,
3095 serole=serole,
3096 setype=setype,
3097 serange=serange,
3098 )
3099 if __opts__["test"]:
3100 if (
3101 isinstance(ret_perms, dict)
3102 and "lmode" in ret_perms
3103 and mode != ret_perms["lmode"]
3104 ):
3105 ret["comment"] = (
3106 "File {} will be updated with permissions "
3107 "{} from its current "
3108 "state of {}".format(name, mode, ret_perms["lmode"])
3109 )
3110 else:
3111 ret["comment"] = "File {} not updated".format(name)
3112 elif not ret["changes"] and ret["result"]:
3113 ret["comment"] = (
3114 "File {} exists with proper permissions. "
3115 "No changes made.".format(name)
3116 )
3117 return ret
3118
3119 accum_data, _ = _load_accumulators()
3120 if name in accum_data:
3121 if not context:
3122 context = {}
3123 context["accumulator"] = accum_data[name]
3124
3125 try:
3126 if __opts__["test"]:
3127 if "file.check_managed_changes" in __salt__:
3128 ret["changes"] = __salt__["file.check_managed_changes"](
3129 name,
3130 source,
3131 source_hash,
3132 source_hash_name,
3133 user,
3134 group,
3135 mode,
3136 attrs,
3137 template,
3138 context,
3139 defaults,
3140 __env__,
3141 contents,
3142 skip_verify,
3143 keep_mode,
3144 seuser=seuser,
3145 serole=serole,
3146 setype=setype,
3147 serange=serange,
3148 verify_ssl=verify_ssl,
3149 **kwargs
3150 )
3151
3152 if salt.utils.platform.is_windows():
3153 try:
3154 ret = __salt__["file.check_perms"](
3155 path=name,
3156 ret=ret,
3157 owner=win_owner,
3158 grant_perms=win_perms,
3159 deny_perms=win_deny_perms,
3160 inheritance=win_inheritance,
3161 reset=win_perms_reset,
3162 )
3163 except CommandExecutionError as exc:
3164 if exc.strerror.startswith("Path not found"):
3165 ret["changes"]["newfile"] = name
3166
3167 if isinstance(ret["changes"], tuple):
3168 ret["result"], ret["comment"] = ret["changes"]
3169 elif ret["changes"]:
3170 ret["result"] = None
3171 ret["comment"] = "The file {} is set to be changed".format(name)
3172 ret["comment"] += (
3173 "\nNote: No changes made, actual changes may\n"
3174 "be different due to other states."
3175 )
3176 if "diff" in ret["changes"] and not show_changes:
3177 ret["changes"]["diff"] = "<show_changes=False>"
3178 else:
3179 ret["result"] = True
3180 ret["comment"] = "The file {} is in the correct state".format(name)
3181
3182 return ret
3183
3184 # If the source is a list then find which file exists
3185 source, source_hash = __salt__["file.source_list"](source, source_hash, __env__)
3186 except CommandExecutionError as exc:
3187 ret["result"] = False
3188 ret["comment"] = "Unable to manage file: {}".format(exc)
3189 return ret
3190
3191 # Gather the source file from the server
3192 try:
3193 sfn, source_sum, comment_ = __salt__["file.get_managed"](
3194 name,
3195 template,
3196 source,
3197 source_hash,
3198 source_hash_name,
3199 user,
3200 group,
3201 mode,
3202 attrs,
3203 __env__,
3204 context,
3205 defaults,
3206 skip_verify,
3207 verify_ssl=verify_ssl,
3208 **kwargs
3209 )
3210 except Exception as exc: # pylint: disable=broad-except
3211 ret["changes"] = {}
3212 log.debug(traceback.format_exc())
3213 return _error(ret, "Unable to manage file: {}".format(exc))
3214
3215 tmp_filename = None
3216
3217 if check_cmd:
3218 tmp_filename = salt.utils.files.mkstemp(suffix=tmp_ext, dir=tmp_dir)
3219
3220 # if exists copy existing file to tmp to compare
3221 if __salt__["file.file_exists"](name):
3222 try:
3223 __salt__["file.copy"](name, tmp_filename)
3224 except Exception as exc: # pylint: disable=broad-except
3225 return _error(
3226 ret,
3227 "Unable to copy file {} to {}: {}".format(name, tmp_filename, exc),
3228 )
3229
3230 try:
3231 ret = __salt__["file.manage_file"](
3232 tmp_filename,
3233 sfn,
3234 ret,
3235 source,
3236 source_sum,
3237 user,
3238 group,
3239 mode,
3240 attrs,
3241 __env__,
3242 backup,
3243 makedirs,
3244 template,
3245 show_changes,
3246 contents,
3247 dir_mode,
3248 follow_symlinks,
3249 skip_verify,
3250 keep_mode,
3251 win_owner=win_owner,
3252 win_perms=win_perms,
3253 win_deny_perms=win_deny_perms,
3254 win_inheritance=win_inheritance,
3255 win_perms_reset=win_perms_reset,
3256 encoding=encoding,
3257 encoding_errors=encoding_errors,
3258 seuser=seuser,
3259 serole=serole,
3260 setype=setype,
3261 serange=serange,
3262 **kwargs
3263 )
3264 except Exception as exc: # pylint: disable=broad-except
3265 ret["changes"] = {}
3266 log.debug(traceback.format_exc())
3267 salt.utils.files.remove(tmp_filename)
3268 if not keep_source:
3269 if not sfn and source and _urlparse(source).scheme == "salt":
3270 # The file would not have been cached until manage_file was
3271 # run, so check again here for a cached copy.
3272 sfn = __salt__["cp.is_cached"](source, __env__)
3273 if sfn:
3274 salt.utils.files.remove(sfn)
3275 return _error(ret, "Unable to check_cmd file: {}".format(exc))
3276
3277 # file being updated to verify using check_cmd
3278 if ret["changes"]:
3279 # Reset ret
3280 ret = {"changes": {}, "comment": "", "name": name, "result": True}
3281
3282 check_cmd_opts = {}
3283 if "shell" in __grains__:
3284 check_cmd_opts["shell"] = __grains__["shell"]
3285
3286 cret = mod_run_check_cmd(check_cmd, tmp_filename, **check_cmd_opts)
3287 if isinstance(cret, dict):
3288 ret.update(cret)
3289 salt.utils.files.remove(tmp_filename)
3290 return ret
3291
3292 # Since we generated a new tempfile and we are not returning here
3293 # lets change the original sfn to the new tempfile or else we will
3294 # get file not found
3295
3296 sfn = tmp_filename
3297
3298 else:
3299 ret = {"changes": {}, "comment": "", "name": name, "result": True}
3300
3301 if comment_ and contents is None:
3302 return _error(ret, comment_)
3303 else:
3304 try:
3305 return __salt__["file.manage_file"](
3306 name,
3307 sfn,
3308 ret,
3309 source,
3310 source_sum,
3311 user,
3312 group,
3313 mode,
3314 attrs,
3315 __env__,
3316 backup,
3317 makedirs,
3318 template,
3319 show_changes,
3320 contents,
3321 dir_mode,
3322 follow_symlinks,
3323 skip_verify,
3324 keep_mode,
3325 win_owner=win_owner,
3326 win_perms=win_perms,
3327 win_deny_perms=win_deny_perms,
3328 win_inheritance=win_inheritance,
3329 win_perms_reset=win_perms_reset,
3330 encoding=encoding,
3331 encoding_errors=encoding_errors,
3332 seuser=seuser,
3333 serole=serole,
3334 setype=setype,
3335 serange=serange,
3336 **kwargs
3337 )
3338 except Exception as exc: # pylint: disable=broad-except
3339 ret["changes"] = {}
3340 log.debug(traceback.format_exc())
3341 return _error(ret, "Unable to manage file: {}".format(exc))
3342 finally:
3343 if tmp_filename:
3344 salt.utils.files.remove(tmp_filename)
3345 if not keep_source:
3346 if not sfn and source and _urlparse(source).scheme == "salt":
3347 # The file would not have been cached until manage_file was
3348 # run, so check again here for a cached copy.
3349 sfn = __salt__["cp.is_cached"](source, __env__)
3350 if sfn:
3351 salt.utils.files.remove(sfn)
3352
3353
3354 _RECURSE_TYPES = ["user", "group", "mode", "ignore_files", "ignore_dirs", "silent"]
3355
3356
3357 def _get_recurse_set(recurse):
3358 """
3359 Converse *recurse* definition to a set of strings.
3360
3361 Raises TypeError or ValueError when *recurse* has wrong structure.
3362 """
3363 if not recurse:
3364 return set()
3365 if not isinstance(recurse, list):
3366 raise TypeError('"recurse" must be formed as a list of strings')
3367 try:
3368 recurse_set = set(recurse)
3369 except TypeError: # non-hashable elements
3370 recurse_set = None
3371 if recurse_set is None or not set(_RECURSE_TYPES) >= recurse_set:
3372 raise ValueError(
3373 'Types for "recurse" limited to {}.'.format(
3374 ", ".join('"{}"'.format(rtype) for rtype in _RECURSE_TYPES)
3375 )
3376 )
3377 if "ignore_files" in recurse_set and "ignore_dirs" in recurse_set:
3378 raise ValueError(
3379 'Must not specify "recurse" options "ignore_files"'
3380 ' and "ignore_dirs" at the same time.'
3381 )
3382 return recurse_set
3383
3384
3385 def _depth_limited_walk(top, max_depth=None):
3386 """
3387 Walk the directory tree under root up till reaching max_depth.
3388 With max_depth=None (default), do not limit depth.
3389 """
3390 for root, dirs, files in salt.utils.path.os_walk(top):
3391 if max_depth is not None:
3392 rel_depth = root.count(os.path.sep) - top.count(os.path.sep)
3393 if rel_depth >= max_depth:
3394 del dirs[:]
3395 yield (str(root), list(dirs), list(files))
3396
3397
3398 def directory(
3399 name,
3400 user=None,
3401 group=None,
3402 recurse=None,
3403 max_depth=None,
3404 dir_mode=None,
3405 file_mode=None,
3406 makedirs=False,
3407 clean=False,
3408 require=None,
3409 exclude_pat=None,
3410 follow_symlinks=False,
3411 force=False,
3412 backupname=None,
3413 allow_symlink=True,
3414 children_only=False,
3415 win_owner=None,
3416 win_perms=None,
3417 win_deny_perms=None,
3418 win_inheritance=True,
3419 win_perms_reset=False,
3420 **kwargs
3421 ):
3422 r"""
3423 Ensure that a named directory is present and has the right perms
3424
3425 name
3426 The location to create or manage a directory, as an absolute path
3427
3428 user
3429 The user to own the directory; this defaults to the user salt is
3430 running as on the minion
3431
3432 group
3433 The group ownership set for the directory; this defaults to the group
3434 salt is running as on the minion. On Windows, this is ignored
3435
3436 recurse
3437 Enforce user/group ownership and mode of directory recursively. Accepts
3438 a list of strings representing what you would like to recurse. If
3439 ``mode`` is defined, will recurse on both ``file_mode`` and ``dir_mode`` if
3440 they are defined. If ``ignore_files`` or ``ignore_dirs`` is included, files or
3441 directories will be left unchanged respectively.
3442 directories will be left unchanged respectively. If ``silent`` is defined,
3443 individual file/directory change notifications will be suppressed.
3444
3445 Example:
3446
3447 .. code-block:: yaml
3448
3449 /var/log/httpd:
3450 file.directory:
3451 - user: root
3452 - group: root
3453 - dir_mode: 755
3454 - file_mode: 644
3455 - recurse:
3456 - user
3457 - group
3458 - mode
3459
3460 Leave files or directories unchanged:
3461
3462 .. code-block:: yaml
3463
3464 /var/log/httpd:
3465 file.directory:
3466 - user: root
3467 - group: root
3468 - dir_mode: 755
3469 - file_mode: 644
3470 - recurse:
3471 - user
3472 - group
3473 - mode
3474 - ignore_dirs
3475
3476 .. versionadded:: 2015.5.0
3477
3478 max_depth
3479 Limit the recursion depth. The default is no limit=None.
3480 'max_depth' and 'clean' are mutually exclusive.
3481
3482 .. versionadded:: 2016.11.0
3483
3484 dir_mode / mode
3485 The permissions mode to set any directories created. Not supported on
3486 Windows.
3487
3488 The default mode for new files and directories corresponds umask of salt
3489 process. For existing files and directories it's not enforced.
3490
3491 file_mode
3492 The permissions mode to set any files created if 'mode' is run in
3493 'recurse'. This defaults to dir_mode. Not supported on Windows.
3494
3495 The default mode for new files and directories corresponds umask of salt
3496 process. For existing files and directories it's not enforced.
3497
3498 makedirs
3499 If the directory is located in a path without a parent directory, then
3500 the state will fail. If makedirs is set to True, then the parent
3501 directories will be created to facilitate the creation of the named
3502 file.
3503
3504 clean
3505 Make sure that only files that are set up by salt and required by this
3506 function are kept. If this option is set then everything in this
3507 directory will be deleted unless it is required.
3508 'clean' and 'max_depth' are mutually exclusive.
3509
3510 require
3511 Require other resources such as packages or files
3512
3513 exclude_pat
3514 When 'clean' is set to True, exclude this pattern from removal list
3515 and preserve in the destination.
3516
3517 follow_symlinks : False
3518 If the desired path is a symlink (or ``recurse`` is defined and a
3519 symlink is encountered while recursing), follow it and check the
3520 permissions of the directory/file to which the symlink points.
3521
3522 .. versionadded:: 2014.1.4
3523
3524 .. versionchanged:: 3001.1
3525 If set to False symlinks permissions are ignored on Linux systems
3526 because it does not support permissions modification. Symlinks
3527 permissions are always 0o777 on Linux.
3528
3529 force
3530 If the name of the directory exists and is not a directory and
3531 force is set to False, the state will fail. If force is set to
3532 True, the file in the way of the directory will be deleted to
3533 make room for the directory, unless backupname is set,
3534 then it will be renamed.
3535
3536 .. versionadded:: 2014.7.0
3537
3538 backupname
3539 If the name of the directory exists and is not a directory, it will be
3540 renamed to the backupname. If the backupname already
3541 exists and force is False, the state will fail. Otherwise, the
3542 backupname will be removed first.
3543
3544 .. versionadded:: 2014.7.0
3545
3546 allow_symlink : True
3547 If allow_symlink is True and the specified path is a symlink, it will be
3548 allowed to remain if it points to a directory. If allow_symlink is False
3549 then the state will fail, unless force is also set to True, in which case
3550 it will be removed or renamed, depending on the value of the backupname
3551 argument.
3552
3553 .. versionadded:: 2014.7.0
3554
3555 children_only : False
3556 If children_only is True the base of a path is excluded when performing
3557 a recursive operation. In case of /path/to/base, base will be ignored
3558 while all of /path/to/base/* are still operated on.
3559
3560 win_owner : None
3561 The owner of the directory. If this is not passed, user will be used. If
3562 user is not passed, the account under which Salt is running will be
3563 used.
3564
3565 .. versionadded:: 2017.7.0
3566
3567 win_perms : None
3568 A dictionary containing permissions to grant and their propagation. For
3569 example: ``{'Administrators': {'perms': 'full_control', 'applies_to':
3570 'this_folder_only'}}`` Can be a single basic perm or a list of advanced
3571 perms. ``perms`` must be specified. ``applies_to`` is optional and
3572 defaults to ``this_folder_subfolder_files``.
3573
3574 .. versionadded:: 2017.7.0
3575
3576 win_deny_perms : None
3577 A dictionary containing permissions to deny and their propagation. For
3578 example: ``{'Administrators': {'perms': 'full_control', 'applies_to':
3579 'this_folder_only'}}`` Can be a single basic perm or a list of advanced
3580 perms.
3581
3582 .. versionadded:: 2017.7.0
3583
3584 win_inheritance : True
3585 True to inherit permissions from the parent directory, False not to
3586 inherit permission.
3587
3588 .. versionadded:: 2017.7.0
3589
3590 win_perms_reset : False
3591 If ``True`` the existing DACL will be cleared and replaced with the
3592 settings defined in this function. If ``False``, new entries will be
3593 appended to the existing DACL. Default is ``False``.
3594
3595 .. versionadded:: 2018.3.0
3596
3597 Here's an example using the above ``win_*`` parameters:
3598
3599 .. code-block:: yaml
3600
3601 create_config_dir:
3602 file.directory:
3603 - name: 'C:\config\'
3604 - win_owner: Administrators
3605 - win_perms:
3606 # Basic Permissions
3607 dev_ops:
3608 perms: full_control
3609 # List of advanced permissions
3610 appuser:
3611 perms:
3612 - read_attributes
3613 - read_ea
3614 - create_folders
3615 - read_permissions
3616 applies_to: this_folder_only
3617 joe_snuffy:
3618 perms: read
3619 applies_to: this_folder_files
3620 - win_deny_perms:
3621 fred_snuffy:
3622 perms: full_control
3623 - win_inheritance: False
3624 """
3625 name = os.path.expanduser(name)
3626 ret = {"name": name, "changes": {}, "result": True, "comment": ""}
3627 if not name:
3628 return _error(ret, "Must provide name to file.directory")
3629 # Remove trailing slash, if present and we're not working on "/" itself
3630 if name[-1] == "/" and name != "/":
3631 name = name[:-1]
3632
3633 if max_depth is not None and clean:
3634 return _error(ret, "Cannot specify both max_depth and clean")
3635
3636 user = _test_owner(kwargs, user=user)
3637 if salt.utils.platform.is_windows():
3638
3639 # If win_owner not passed, use user
3640 if win_owner is None:
3641 win_owner = user if user else salt.utils.win_functions.get_current_user()
3642
3643 # Group isn't relevant to Windows, use win_perms/win_deny_perms
3644 if group is not None:
3645 log.warning(
3646 "The group argument for {} has been ignored as this is "
3647 "a Windows system. Please use the `win_*` parameters to set "
3648 "permissions in Windows.".format(name)
3649 )
3650 group = user
3651
3652 if "mode" in kwargs and not dir_mode:
3653 dir_mode = kwargs.get("mode", [])
3654
3655 if not file_mode:
3656 file_mode = dir_mode
3657
3658 # Make sure that leading zeros stripped by YAML loader are added back
3659 dir_mode = salt.utils.files.normalize_mode(dir_mode)
3660 file_mode = salt.utils.files.normalize_mode(file_mode)
3661
3662 if salt.utils.platform.is_windows():
3663 # Verify win_owner is valid on the target system
3664 try:
3665 salt.utils.win_dacl.get_sid(win_owner)
3666 except CommandExecutionError as exc:
3667 return _error(ret, exc)
3668 else:
3669 # Verify user and group are valid
3670 u_check = _check_user(user, group)
3671 if u_check:
3672 # The specified user or group do not exist
3673 return _error(ret, u_check)
3674
3675 # Must be an absolute path
3676 if not os.path.isabs(name):
3677 return _error(ret, "Specified file {} is not an absolute path".format(name))
3678
3679 # Check for existing file or symlink
3680 if (
3681 os.path.isfile(name)
3682 or (not allow_symlink and os.path.islink(name))
3683 or (force and os.path.islink(name))
3684 ):
3685 # Was a backupname specified
3686 if backupname is not None:
3687 # Make a backup first
3688 if os.path.lexists(backupname):
3689 if not force:
3690 return _error(
3691 ret,
3692 (
3693 ("File exists where the backup target {} should go").format(
3694 backupname
3695 )
3696 ),
3697 )
3698 else:
3699 __salt__["file.remove"](backupname)
3700 os.rename(name, backupname)
3701 elif force:
3702 # Remove whatever is in the way
3703 if os.path.isfile(name):
3704 if __opts__["test"]:
3705 ret["changes"]["forced"] = "File would be forcibly replaced"
3706 else:
3707 os.remove(name)
3708 ret["changes"]["forced"] = "File was forcibly replaced"
3709 elif __salt__["file.is_link"](name):
3710 if __opts__["test"]:
3711 ret["changes"]["forced"] = "Symlink would be forcibly replaced"
3712 else:
3713 __salt__["file.remove"](name)
3714 ret["changes"]["forced"] = "Symlink was forcibly replaced"
3715 else:
3716 if __opts__["test"]:
3717 ret["changes"]["forced"] = "Directory would be forcibly replaced"
3718 else:
3719 __salt__["file.remove"](name)
3720 ret["changes"]["forced"] = "Directory was forcibly replaced"
3721 else:
3722 if os.path.isfile(name):
3723 return _error(
3724 ret, "Specified location {} exists and is a file".format(name)
3725 )
3726 elif os.path.islink(name):
3727 return _error(
3728 ret, "Specified location {} exists and is a symlink".format(name)
3729 )
3730
3731 # Check directory?
3732 if salt.utils.platform.is_windows():
3733 tresult, tcomment, tchanges = _check_directory_win(
3734 name=name,
3735 win_owner=win_owner,
3736 win_perms=win_perms,
3737 win_deny_perms=win_deny_perms,
3738 win_inheritance=win_inheritance,
3739 win_perms_reset=win_perms_reset,
3740 )
3741 else:
3742 tresult, tcomment, tchanges = _check_directory(
3743 name,
3744 user,
3745 group,
3746 recurse or [],
3747 dir_mode,
3748 file_mode,
3749 clean,
3750 require,
3751 exclude_pat,
3752 max_depth,
3753 follow_symlinks,
3754 )
3755
3756 if tchanges:
3757 ret["changes"].update(tchanges)
3758
3759 # Don't run through the reset of the function if there are no changes to be
3760 # made
3761 if __opts__["test"] or not ret["changes"]:
3762 ret["result"] = tresult
3763 ret["comment"] = tcomment
3764 return ret
3765
3766 if not os.path.isdir(name):
3767 # The dir does not exist, make it
3768 if not os.path.isdir(os.path.dirname(name)):
3769 # The parent directory does not exist, create them
3770 if makedirs:
3771 # Everything's good, create the parent Dirs
3772 try:
3773 _makedirs(
3774 name=name,
3775 user=user,
3776 group=group,
3777 dir_mode=dir_mode,
3778 win_owner=win_owner,
3779 win_perms=win_perms,
3780 win_deny_perms=win_deny_perms,
3781 win_inheritance=win_inheritance,
3782 )
3783 except CommandExecutionError as exc:
3784 return _error(ret, "Drive {} is not mapped".format(exc.message))
3785 else:
3786 return _error(ret, "No directory to create {} in".format(name))
3787
3788 if salt.utils.platform.is_windows():
3789 __salt__["file.mkdir"](
3790 path=name,
3791 owner=win_owner,
3792 grant_perms=win_perms,
3793 deny_perms=win_deny_perms,
3794 inheritance=win_inheritance,
3795 reset=win_perms_reset,
3796 )
3797 else:
3798 __salt__["file.mkdir"](name, user=user, group=group, mode=dir_mode)
3799
3800 ret["changes"][name] = "New Dir"
3801
3802 if not os.path.isdir(name):
3803 return _error(ret, "Failed to create directory {}".format(name))
3804
3805 # issue 32707: skip this __salt__['file.check_perms'] call if children_only == True
3806 # Check permissions
3807 if not children_only:
3808 if salt.utils.platform.is_windows():
3809 ret = __salt__["file.check_perms"](
3810 path=name,
3811 ret=ret,
3812 owner=win_owner,
3813 grant_perms=win_perms,
3814 deny_perms=win_deny_perms,
3815 inheritance=win_inheritance,
3816 reset=win_perms_reset,
3817 )
3818 else:
3819 ret, perms = __salt__["file.check_perms"](
3820 name, ret, user, group, dir_mode, None, follow_symlinks
3821 )
3822
3823 errors = []
3824 if recurse or clean:
3825 # walk path only once and store the result
3826 walk_l = list(_depth_limited_walk(name, max_depth))
3827 # root: (dirs, files) structure, compatible for python2.6
3828 walk_d = {}
3829 for i in walk_l:
3830 walk_d[i[0]] = (i[1], i[2])
3831
3832 recurse_set = None
3833 if recurse:
3834 try:
3835 recurse_set = _get_recurse_set(recurse)
3836 except (TypeError, ValueError) as exc:
3837 ret["result"] = False
3838 ret["comment"] = "{}".format(exc)
3839 # NOTE: Should this be enough to stop the whole check altogether?
3840 if recurse_set:
3841 if "user" in recurse_set:
3842 if user or isinstance(user, int):
3843 uid = __salt__["file.user_to_uid"](user)
3844 # file.user_to_uid returns '' if user does not exist. Above
3845 # check for user is not fatal, so we need to be sure user
3846 # exists.
3847 if isinstance(uid, str):
3848 ret["result"] = False
3849 ret["comment"] = (
3850 "Failed to enforce ownership for "
3851 "user {} (user does not "
3852 "exist)".format(user)
3853 )
3854 else:
3855 ret["result"] = False
3856 ret["comment"] = (
3857 "user not specified, but configured as "
3858 "a target for recursive ownership "
3859 "management"
3860 )
3861 else:
3862 user = None
3863 if "group" in recurse_set:
3864 if group or isinstance(group, int):
3865 gid = __salt__["file.group_to_gid"](group)
3866 # As above with user, we need to make sure group exists.
3867 if isinstance(gid, str):
3868 ret["result"] = False
3869 ret["comment"] = (
3870 "Failed to enforce group ownership "
3871 "for group {}".format(group)
3872 )
3873 else:
3874 ret["result"] = False
3875 ret["comment"] = (
3876 "group not specified, but configured "
3877 "as a target for recursive ownership "
3878 "management"
3879 )
3880 else:
3881 group = None
3882
3883 if "mode" not in recurse_set:
3884 file_mode = None
3885 dir_mode = None
3886
3887 if "silent" in recurse_set:
3888 ret["changes"] = {"recursion": "Changes silenced"}
3889
3890 check_files = "ignore_files" not in recurse_set
3891 check_dirs = "ignore_dirs" not in recurse_set
3892
3893 for root, dirs, files in walk_l:
3894 if check_files:
3895 for fn_ in files:
3896 full = os.path.join(root, fn_)
3897 try:
3898 if salt.utils.platform.is_windows():
3899 ret = __salt__["file.check_perms"](
3900 path=full,
3901 ret=ret,
3902 owner=win_owner,
3903 grant_perms=win_perms,
3904 deny_perms=win_deny_perms,
3905 inheritance=win_inheritance,
3906 reset=win_perms_reset,
3907 )
3908 else:
3909 ret, _ = __salt__["file.check_perms"](
3910 full, ret, user, group, file_mode, None, follow_symlinks
3911 )
3912 except CommandExecutionError as exc:
3913 if not exc.strerror.startswith("Path not found"):
3914 errors.append(exc.strerror)
3915
3916 if check_dirs:
3917 for dir_ in dirs:
3918 full = os.path.join(root, dir_)
3919 try:
3920 if salt.utils.platform.is_windows():
3921 ret = __salt__["file.check_perms"](
3922 path=full,
3923 ret=ret,
3924 owner=win_owner,
3925 grant_perms=win_perms,
3926 deny_perms=win_deny_perms,
3927 inheritance=win_inheritance,
3928 reset=win_perms_reset,
3929 )
3930 else:
3931 ret, _ = __salt__["file.check_perms"](
3932 full, ret, user, group, dir_mode, None, follow_symlinks
3933 )
3934 except CommandExecutionError as exc:
3935 if not exc.strerror.startswith("Path not found"):
3936 errors.append(exc.strerror)
3937
3938 if clean:
3939 keep = _gen_keep_files(name, require, walk_d)
3940 log.debug("List of kept files when use file.directory with clean: %s", keep)
3941 removed = _clean_dir(name, list(keep), exclude_pat)
3942 if removed:
3943 ret["changes"]["removed"] = removed
3944 ret["comment"] = "Files cleaned from directory {}".format(name)
3945
3946 # issue 32707: reflect children_only selection in comments
3947 if not ret["comment"]:
3948 if children_only:
3949 ret["comment"] = "Directory {}/* updated".format(name)
3950 else:
3951 if ret["changes"]:
3952 ret["comment"] = "Directory {} updated".format(name)
3953
3954 if __opts__["test"]:
3955 ret["comment"] = "Directory {} not updated".format(name)
3956 elif not ret["changes"] and ret["result"]:
3957 orig_comment = None
3958 if ret["comment"]:
3959 orig_comment = ret["comment"]
3960
3961 ret["comment"] = "Directory {} is in the correct state".format(name)
3962 if orig_comment:
3963 ret["comment"] = "\n".join([ret["comment"], orig_comment])
3964
3965 if errors:
3966 ret["result"] = False
3967 ret["comment"] += "\n\nThe following errors were encountered:\n"
3968 for error in errors:
3969 ret["comment"] += "\n- {}".format(error)
3970
3971 return ret
3972
3973
3974 def recurse(
3975 name,
3976 source,
3977 keep_source=True,
3978 clean=False,
3979 require=None,
3980 user=None,
3981 group=None,
3982 dir_mode=None,
3983 file_mode=None,
3984 sym_mode=None,
3985 template=None,
3986 context=None,
3987 replace=True,
3988 defaults=None,
3989 include_empty=False,
3990 backup="",
3991 include_pat=None,
3992 exclude_pat=None,
3993 maxdepth=None,
3994 keep_symlinks=False,
3995 force_symlinks=False,
3996 win_owner=None,
3997 win_perms=None,
3998 win_deny_perms=None,
3999 win_inheritance=True,
4000 **kwargs
4001 ):
4002 """
4003 Recurse through a subdirectory on the master and copy said subdirectory
4004 over to the specified path.
4005
4006 name
4007 The directory to set the recursion in
4008
4009 source
4010 The source directory, this directory is located on the salt master file
4011 server and is specified with the salt:// protocol. If the directory is
4012 located on the master in the directory named spam, and is called eggs,
4013 the source string is salt://spam/eggs
4014
4015 keep_source : True
4016 Set to ``False`` to discard the cached copy of the source file once the
4017 state completes. This can be useful for larger files to keep them from
4018 taking up space in minion cache. However, keep in mind that discarding
4019 the source file will result in the state needing to re-download the
4020 source file if the state is run again.
4021
4022 .. versionadded:: 2017.7.3
4023
4024 clean
4025 Make sure that only files that are set up by salt and required by this
4026 function are kept. If this option is set then everything in this
4027 directory will be deleted unless it is required.
4028
4029 require
4030 Require other resources such as packages or files
4031
4032 user
4033 The user to own the directory. This defaults to the user salt is
4034 running as on the minion
4035
4036 group
4037 The group ownership set for the directory. This defaults to the group
4038 salt is running as on the minion. On Windows, this is ignored
4039
4040 dir_mode
4041 The permissions mode to set on any directories created.
4042
4043 The default mode for new files and directories corresponds umask of salt
4044 process. For existing files and directories it's not enforced.
4045
4046 .. note::
4047 This option is **not** supported on Windows.
4048
4049 file_mode
4050 The permissions mode to set on any files created.
4051
4052 The default mode for new files and directories corresponds umask of salt
4053 process. For existing files and directories it's not enforced.
4054
4055 .. note::
4056 This option is **not** supported on Windows.
4057
4058 .. versionchanged:: 2016.11.0
4059 This option can be set to ``keep``, and Salt will keep the mode
4060 from the Salt fileserver. This is only supported when the
4061 ``source`` URL begins with ``salt://``, or for files local to the
4062 minion. Because the ``source`` option cannot be used with any of
4063 the ``contents`` options, setting the ``mode`` to ``keep`` is also
4064 incompatible with the ``contents`` options.
4065
4066 sym_mode
4067 The permissions mode to set on any symlink created.
4068
4069 The default mode for new files and directories corresponds umask of salt
4070 process. For existing files and directories it's not enforced.
4071
4072 .. note::
4073 This option is **not** supported on Windows.
4074
4075 template
4076 If this setting is applied, the named templating engine will be used to
4077 render the downloaded file. The following templates are supported:
4078
4079 - :mod:`cheetah<salt.renderers.cheetah>`
4080 - :mod:`genshi<salt.renderers.genshi>`
4081 - :mod:`jinja<salt.renderers.jinja>`
4082 - :mod:`mako<salt.renderers.mako>`
4083 - :mod:`py<salt.renderers.py>`
4084 - :mod:`wempy<salt.renderers.wempy>`
4085
4086 .. note::
4087
4088 The template option is required when recursively applying templates.
4089
4090 replace : True
4091 If set to ``False`` and the file already exists, the file will not be
4092 modified even if changes would otherwise be made. Permissions and
4093 ownership will still be enforced, however.
4094
4095 context
4096 Overrides default context variables passed to the template.
4097
4098 defaults
4099 Default context passed to the template.
4100
4101 include_empty
4102 Set this to True if empty directories should also be created
4103 (default is False)
4104
4105 backup
4106 Overrides the default backup mode for all replaced files. See
4107 :ref:`backup_mode documentation <file-state-backups>` for more details.
4108
4109 include_pat
4110 When copying, include only this pattern, or list of patterns, from the
4111 source. Default is glob match; if prefixed with 'E@', then regexp match.
4112 Example:
4113
4114 .. code-block:: text
4115
4116 - include_pat: hello* :: glob matches 'hello01', 'hello02'
4117 ... but not 'otherhello'
4118 - include_pat: E@hello :: regexp matches 'otherhello',
4119 'hello01' ...
4120
4121 .. versionchanged:: 3001
4122
4123 List patterns are now supported
4124
4125 .. code-block:: text
4126
4127 - include_pat:
4128 - hello01
4129 - hello02
4130
4131 exclude_pat
4132 Exclude this pattern, or list of patterns, from the source when copying.
4133 If both `include_pat` and `exclude_pat` are supplied, then it will apply
4134 conditions cumulatively. i.e. first select based on include_pat, and
4135 then within that result apply exclude_pat.
4136
4137 Also, when 'clean=True', exclude this pattern from the removal
4138 list and preserve in the destination.
4139 Example:
4140
4141 .. code-block:: text
4142
4143 - exclude_pat: APPDATA* :: glob matches APPDATA.01,
4144 APPDATA.02,.. for exclusion
4145 - exclude_pat: E@(APPDATA)|(TEMPDATA) :: regexp matches APPDATA
4146 or TEMPDATA for exclusion
4147
4148 .. versionchanged:: 3001
4149
4150 List patterns are now supported
4151
4152 .. code-block:: text
4153
4154 - exclude_pat:
4155 - APPDATA.01
4156 - APPDATA.02
4157
4158 maxdepth
4159 When copying, only copy paths which are of depth `maxdepth` from the
4160 source path.
4161 Example:
4162
4163 .. code-block:: text
4164
4165 - maxdepth: 0 :: Only include files located in the source
4166 directory
4167 - maxdepth: 1 :: Only include files located in the source
4168 or immediate subdirectories
4169
4170 keep_symlinks
4171 Keep symlinks when copying from the source. This option will cause
4172 the copy operation to terminate at the symlink. If desire behavior
4173 similar to rsync, then set this to True.
4174
4175 force_symlinks
4176 Force symlink creation. This option will force the symlink creation.
4177 If a file or directory is obstructing symlink creation it will be
4178 recursively removed so that symlink creation can proceed. This
4179 option is usually not needed except in special circumstances.
4180
4181 win_owner : None
4182 The owner of the symlink and directories if ``makedirs`` is True. If
4183 this is not passed, ``user`` will be used. If ``user`` is not passed,
4184 the account under which Salt is running will be used.
4185
4186 .. versionadded:: 2017.7.7
4187
4188 win_perms : None
4189 A dictionary containing permissions to grant
4190
4191 .. versionadded:: 2017.7.7
4192
4193 win_deny_perms : None
4194 A dictionary containing permissions to deny
4195
4196 .. versionadded:: 2017.7.7
4197
4198 win_inheritance : None
4199 True to inherit permissions from parent, otherwise False
4200
4201 .. versionadded:: 2017.7.7
4202
4203 """
4204 if "env" in kwargs:
4205 # "env" is not supported; Use "saltenv".
4206 kwargs.pop("env")
4207
4208 name = os.path.expanduser(salt.utils.data.decode(name))
4209
4210 user = _test_owner(kwargs, user=user)
4211 if salt.utils.platform.is_windows():
4212 if group is not None:
4213 log.warning(
4214 "The group argument for {} has been ignored as this "
4215 "is a Windows system.".format(name)
4216 )
4217 group = user
4218 ret = {
4219 "name": name,
4220 "changes": {},
4221 "result": True,
4222 "comment": {}, # { path: [comment, ...] }
4223 }
4224
4225 if "mode" in kwargs:
4226 ret["result"] = False
4227 ret["comment"] = (
4228 "'mode' is not allowed in 'file.recurse'. Please use "
4229 "'file_mode' and 'dir_mode'."
4230 )
4231 return ret
4232
4233 if (
4234 any([x is not None for x in (dir_mode, file_mode, sym_mode)])
4235 and salt.utils.platform.is_windows()
4236 ):
4237 return _error(ret, "mode management is not supported on Windows")
4238
4239 # Make sure that leading zeros stripped by YAML loader are added back
4240 dir_mode = salt.utils.files.normalize_mode(dir_mode)
4241
4242 try:
4243 keep_mode = file_mode.lower() == "keep"
4244 if keep_mode:
4245 # We're not hard-coding the mode, so set it to None
4246 file_mode = None
4247 except AttributeError:
4248 keep_mode = False
4249
4250 file_mode = salt.utils.files.normalize_mode(file_mode)
4251
4252 u_check = _check_user(user, group)
4253 if u_check:
4254 # The specified user or group do not exist
4255 return _error(ret, u_check)
4256 if not os.path.isabs(name):
4257 return _error(ret, "Specified file {} is not an absolute path".format(name))
4258
4259 # expand source into source_list
4260 source_list = _validate_str_list(source)
4261
4262 for idx, val in enumerate(source_list):
4263 source_list[idx] = val.rstrip("/")
4264
4265 for precheck in source_list:
4266 if not precheck.startswith("salt://"):
4267 return _error(
4268 ret,
4269 ("Invalid source '{}' " "(must be a salt:// URI)".format(precheck)),
4270 )
4271
4272 # Select the first source in source_list that exists
4273 try:
4274 source, source_hash = __salt__["file.source_list"](source_list, "", __env__)
4275 except CommandExecutionError as exc:
4276 ret["result"] = False
4277 ret["comment"] = "Recurse failed: {}".format(exc)
4278 return ret
4279
4280 # Check source path relative to fileserver root, make sure it is a
4281 # directory
4282 srcpath, senv = salt.utils.url.parse(source)
4283 if senv is None:
4284 senv = __env__
4285 master_dirs = __salt__["cp.list_master_dirs"](saltenv=senv)
4286 if srcpath not in master_dirs and not any(
4287 x for x in master_dirs if x.startswith(srcpath + "/")
4288 ):
4289 ret["result"] = False
4290 ret["comment"] = (
4291 "The directory '{}' does not exist on the salt fileserver "
4292 "in saltenv '{}'".format(srcpath, senv)
4293 )
4294 return ret
4295
4296 # Verify the target directory
4297 if not os.path.isdir(name):
4298 if os.path.exists(name):
4299 # it is not a dir, but it exists - fail out
4300 return _error(ret, "The path {} exists and is not a directory".format(name))
4301 if not __opts__["test"]:
4302 if salt.utils.platform.is_windows():
4303 win_owner = win_owner if win_owner else user
4304 __salt__["file.makedirs_perms"](
4305 path=name,
4306 owner=win_owner,
4307 grant_perms=win_perms,
4308 deny_perms=win_deny_perms,
4309 inheritance=win_inheritance,
4310 )
4311 else:
4312 __salt__["file.makedirs_perms"](
4313 name=name, user=user, group=group, mode=dir_mode
4314 )
4315
4316 def add_comment(path, comment):
4317 comments = ret["comment"].setdefault(path, [])
4318 if isinstance(comment, str):
4319 comments.append(comment)
4320 else:
4321 comments.extend(comment)
4322
4323 def merge_ret(path, _ret):
4324 # Use the most "negative" result code (out of True, None, False)
4325 if _ret["result"] is False or ret["result"] is True:
4326 ret["result"] = _ret["result"]
4327
4328 # Only include comments about files that changed
4329 if _ret["result"] is not True and _ret["comment"]:
4330 add_comment(path, _ret["comment"])
4331
4332 if _ret["changes"]:
4333 ret["changes"][path] = _ret["changes"]
4334
4335 def manage_file(path, source, replace):
4336 if clean and os.path.exists(path) and os.path.isdir(path) and replace:
4337 _ret = {"name": name, "changes": {}, "result": True, "comment": ""}
4338 if __opts__["test"]:
4339 _ret["comment"] = "Replacing directory {} with a " "file".format(path)
4340 _ret["result"] = None
4341 merge_ret(path, _ret)
4342 return
4343 else:
4344 __salt__["file.remove"](path)
4345 _ret["changes"] = {"diff": "Replaced directory with a " "new file"}
4346 merge_ret(path, _ret)
4347
4348 # Conflicts can occur if some kwargs are passed in here
4349 pass_kwargs = {}
4350 faults = ["mode", "makedirs"]
4351 for key in kwargs:
4352 if key not in faults:
4353 pass_kwargs[key] = kwargs[key]
4354
4355 _ret = managed(
4356 path,
4357 source=source,
4358 keep_source=keep_source,
4359 user=user,
4360 group=group,
4361 mode="keep" if keep_mode else file_mode,
4362 attrs=None,
4363 template=template,
4364 makedirs=True,
4365 replace=replace,
4366 context=context,
4367 defaults=defaults,
4368 backup=backup,
4369 **pass_kwargs
4370 )
4371 merge_ret(path, _ret)
4372
4373 def manage_directory(path):
4374 if os.path.basename(path) == "..":
4375 return
4376 if clean and os.path.exists(path) and not os.path.isdir(path):
4377 _ret = {"name": name, "changes": {}, "result": True, "comment": ""}
4378 if __opts__["test"]:
4379 _ret["comment"] = "Replacing {} with a directory".format(path)
4380 _ret["result"] = None
4381 merge_ret(path, _ret)
4382 return
4383 else:
4384 __salt__["file.remove"](path)
4385 _ret["changes"] = {"diff": "Replaced file with a directory"}
4386 merge_ret(path, _ret)
4387
4388 _ret = directory(
4389 path,
4390 user=user,
4391 group=group,
4392 recurse=[],
4393 dir_mode=dir_mode,
4394 file_mode=None,
4395 makedirs=True,
4396 clean=False,
4397 require=None,
4398 )
4399 merge_ret(path, _ret)
4400
4401 mng_files, mng_dirs, mng_symlinks, keep = _gen_recurse_managed_files(
4402 name, source, keep_symlinks, include_pat, exclude_pat, maxdepth, include_empty
4403 )
4404
4405 for srelpath, ltarget in mng_symlinks:
4406 _ret = symlink(
4407 os.path.join(name, srelpath),
4408 ltarget,
4409 makedirs=True,
4410 force=force_symlinks,
4411 user=user,
4412 group=group,
4413 mode=sym_mode,
4414 )
4415 if not _ret:
4416 continue
4417 merge_ret(os.path.join(name, srelpath), _ret)
4418 for dirname in mng_dirs:
4419 manage_directory(dirname)
4420 for dest, src in mng_files:
4421 manage_file(dest, src, replace)
4422
4423 if clean:
4424 # TODO: Use directory(clean=True) instead
4425 keep.update(_gen_keep_files(name, require))
4426 removed = _clean_dir(name, list(keep), exclude_pat)
4427 if removed:
4428 if __opts__["test"]:
4429 if ret["result"]:
4430 ret["result"] = None
4431 add_comment("removed", removed)
4432 else:
4433 ret["changes"]["removed"] = removed
4434
4435 # Flatten comments until salt command line client learns
4436 # to display structured comments in a readable fashion
4437 ret["comment"] = "\n".join(
4438 "\n#### {} ####\n{}".format(k, v if isinstance(v, str) else "\n".join(v))
4439 for (k, v) in ret["comment"].items()
4440 ).strip()
4441
4442 if not ret["comment"]:
4443 ret["comment"] = "Recursively updated {}".format(name)
4444
4445 if not ret["changes"] and ret["result"]:
4446 ret["comment"] = "The directory {} is in the correct state".format(name)
4447
4448 return ret
4449
4450
4451 def retention_schedule(name, retain, strptime_format=None, timezone=None):
4452 """
4453 Apply retention scheduling to backup storage directory.
4454
4455 .. versionadded:: 2016.11.0
4456
4457 :param name:
4458 The filesystem path to the directory containing backups to be managed.
4459
4460 :param retain:
4461 Delete the backups, except for the ones we want to keep.
4462 The N below should be an integer but may also be the special value of ``all``,
4463 which keeps all files matching the criteria.
4464 All of the retain options default to None,
4465 which means to not keep files based on this criteria.
4466
4467 :most_recent N:
4468 Keep the most recent N files.
4469
4470 :first_of_hour N:
4471 For the last N hours from now, keep the first file after the hour.
4472
4473 :first_of_day N:
4474 For the last N days from now, keep the first file after midnight.
4475 See also ``timezone``.
4476
4477 :first_of_week N:
4478 For the last N weeks from now, keep the first file after Sunday midnight.
4479
4480 :first_of_month N:
4481 For the last N months from now, keep the first file after the start of the month.
4482
4483 :first_of_year N:
4484 For the last N years from now, keep the first file after the start of the year.
4485
4486 :param strptime_format:
4487 A python strptime format string used to first match the filenames of backups
4488 and then parse the filename to determine the datetime of the file.
4489 https://docs.python.org/2/library/datetime.html#datetime.datetime.strptime
4490 Defaults to None, which considers all files in the directory to be backups eligible for deletion
4491 and uses ``os.path.getmtime()`` to determine the datetime.
4492
4493 :param timezone:
4494 The timezone to use when determining midnight.
4495 This is only used when datetime is pulled from ``os.path.getmtime()``.
4496 Defaults to ``None`` which uses the timezone from the locale.
4497
4498 Usage example:
4499
4500 .. code-block:: yaml
4501
4502 /var/backups/example_directory:
4503 file.retention_schedule:
4504 - retain:
4505 most_recent: 5
4506 first_of_hour: 4
4507 first_of_day: 7
4508 first_of_week: 6 # NotImplemented yet.
4509 first_of_month: 6
4510 first_of_year: all
4511 - strptime_format: example_name_%Y%m%dT%H%M%S.tar.bz2
4512 - timezone: None
4513
4514 """
4515 name = os.path.expanduser(name)
4516 ret = {
4517 "name": name,
4518 "changes": {"retained": [], "deleted": [], "ignored": []},
4519 "result": True,
4520 "comment": "",
4521 }
4522 if not name:
4523 return _error(ret, "Must provide name to file.retention_schedule")
4524 if not os.path.isdir(name):
4525 return _error(ret, "Name provided to file.retention must be a directory")
4526
4527 # get list of files in directory
4528 all_files = __salt__["file.readdir"](name)
4529
4530 # if strptime_format is set, filter through the list to find names which parse and get their datetimes.
4531 beginning_of_unix_time = datetime(1970, 1, 1)
4532
4533 def get_file_time_from_strptime(f):
4534 try:
4535 ts = datetime.strptime(f, strptime_format)
4536 ts_epoch = salt.utils.dateutils.total_seconds(ts - beginning_of_unix_time)
4537 return (ts, ts_epoch)
4538 except ValueError:
4539 # Files which don't match the pattern are not relevant files.
4540 return (None, None)
4541
4542 def get_file_time_from_mtime(f):
4543 if f == "." or f == "..":
4544 return (None, None)
4545 lstat = __salt__["file.lstat"](os.path.join(name, f))
4546 if lstat:
4547 mtime = lstat["st_mtime"]
4548 return (datetime.fromtimestamp(mtime, timezone), mtime)
4549 else: # maybe it was deleted since we did the readdir?
4550 return (None, None)
4551
4552 get_file_time = (
4553 get_file_time_from_strptime if strptime_format else get_file_time_from_mtime
4554 )
4555
4556 # data structures are nested dicts:
4557 # files_by_ymd = year.month.day.hour.unixtime: filename
4558 # files_by_y_week_dow = year.week_of_year.day_of_week.unixtime: filename
4559 # http://the.randomengineer.com/2015/04/28/python-recursive-defaultdict/
4560 # TODO: move to an ordered dict model and reduce the number of sorts in the rest of the code?
4561 def dict_maker():
4562 return defaultdict(dict_maker)
4563
4564 files_by_ymd = dict_maker()
4565 files_by_y_week_dow = dict_maker()
4566 relevant_files = set()
4567 ignored_files = set()
4568 for f in all_files:
4569 ts, ts_epoch = get_file_time(f)
4570 if ts:
4571 files_by_ymd[ts.year][ts.month][ts.day][ts.hour][ts_epoch] = f
4572 week_of_year = ts.isocalendar()[1]
4573 files_by_y_week_dow[ts.year][week_of_year][ts.weekday()][ts_epoch] = f
4574 relevant_files.add(f)
4575 else:
4576 ignored_files.add(f)
4577
4578 # This is tightly coupled with the file_with_times data-structure above.
4579 RETAIN_TO_DEPTH = {
4580 "first_of_year": 1,
4581 "first_of_month": 2,
4582 "first_of_day": 3,
4583 "first_of_hour": 4,
4584 "most_recent": 5,
4585 }
4586
4587 def get_first(fwt):
4588 if isinstance(fwt, dict):
4589 first_sub_key = sorted(fwt.keys())[0]
4590 return get_first(fwt[first_sub_key])
4591 else:
4592 return {fwt}
4593
4594 def get_first_n_at_depth(fwt, depth, n):
4595 if depth <= 0:
4596 return get_first(fwt)
4597 else:
4598 result_set = set()
4599 for k in sorted(fwt.keys(), reverse=True):
4600 needed = n - len(result_set)
4601 if needed < 1:
4602 break
4603 result_set |= get_first_n_at_depth(fwt[k], depth - 1, needed)
4604 return result_set
4605
4606 # for each retain criteria, add filenames which match the criteria to the retain set.
4607 retained_files = set()
4608 for retention_rule, keep_count in retain.items():
4609 # This is kind of a hack, since 'all' should really mean all,
4610 # but I think it's a large enough number that even modern filesystems would
4611 # choke if they had this many files in a single directory.
4612 keep_count = sys.maxsize if "all" == keep_count else int(keep_count)
4613 if "first_of_week" == retention_rule:
4614 first_of_week_depth = 2 # year + week_of_year = 2
4615 # I'm adding 1 to keep_count below because it fixed an off-by one
4616 # issue in the tests. I don't understand why, and that bothers me.
4617 retained_files |= get_first_n_at_depth(
4618 files_by_y_week_dow, first_of_week_depth, keep_count + 1
4619 )
4620 else:
4621 retained_files |= get_first_n_at_depth(
4622 files_by_ymd, RETAIN_TO_DEPTH[retention_rule], keep_count
4623 )
4624
4625 deletable_files = list(relevant_files - retained_files)
4626 deletable_files.sort(reverse=True)
4627 changes = {
4628 "retained": sorted(list(retained_files), reverse=True),
4629 "deleted": deletable_files,
4630 "ignored": sorted(list(ignored_files), reverse=True),
4631 }
4632 ret["changes"] = changes
4633
4634 # TODO: track and report how much space was / would be reclaimed
4635 if __opts__["test"]:
4636 ret["comment"] = "{} backups would have been removed from {}.\n".format(
4637 len(deletable_files), name
4638 )
4639 if deletable_files:
4640 ret["result"] = None
4641 else:
4642 for f in deletable_files:
4643 __salt__["file.remove"](os.path.join(name, f))
4644 ret["comment"] = "{} backups were removed from {}.\n".format(
4645 len(deletable_files), name
4646 )
4647 ret["changes"] = changes
4648
4649 return ret
4650
4651
4652 def line(
4653 name,
4654 content=None,
4655 match=None,
4656 mode=None,
4657 location=None,
4658 before=None,
4659 after=None,
4660 show_changes=True,
4661 backup=False,
4662 quiet=False,
4663 indent=True,
4664 create=False,
4665 user=None,
4666 group=None,
4667 file_mode=None,
4668 ):
4669 """
4670 Line-focused editing of a file.
4671
4672 .. versionadded:: 2015.8.0
4673
4674 .. note::
4675
4676 ``file.line`` exists for historic reasons, and is not
4677 generally recommended. It has a lot of quirks. You may find
4678 ``file.replace`` to be more suitable.
4679
4680 ``file.line`` is most useful if you have single lines in a file,
4681 potentially a config file, that you would like to manage. It can
4682 remove, add, and replace lines.
4683
4684 name
4685 Filesystem path to the file to be edited.
4686
4687 content
4688 Content of the line. Allowed to be empty if mode=delete.
4689
4690 match
4691 Match the target line for an action by
4692 a fragment of a string or regular expression.
4693
4694 If neither ``before`` nor ``after`` are provided, and ``match``
4695 is also ``None``, match falls back to the ``content`` value.
4696
4697 mode
4698 Defines how to edit a line. One of the following options is
4699 required:
4700
4701 - ensure
4702 If line does not exist, it will be added. If ``before``
4703 and ``after`` are specified either zero lines, or lines
4704 that contain the ``content`` line are allowed to be in between
4705 ``before`` and ``after``. If there are lines, and none of
4706 them match then it will produce an error.
4707 - replace
4708 If line already exists, it will be replaced.
4709 - delete
4710 Delete the line, if found.
4711 - insert
4712 Nearly identical to ``ensure``. If a line does not exist,
4713 it will be added.
4714
4715 The differences are that multiple (and non-matching) lines are
4716 alloweed between ``before`` and ``after``, if they are
4717 specified. The line will always be inserted right before
4718 ``before``. ``insert`` also allows the use of ``location`` to
4719 specify that the line should be added at the beginning or end of
4720 the file.
4721
4722 .. note::
4723
4724 If ``mode=insert`` is used, at least one of the following
4725 options must also be defined: ``location``, ``before``, or
4726 ``after``. If ``location`` is used, it takes precedence
4727 over the other two options.
4728
4729 location
4730 In ``mode=insert`` only, whether to place the ``content`` at the
4731 beginning or end of a the file. If ``location`` is provided,
4732 ``before`` and ``after`` are ignored. Valid locations:
4733
4734 - start
4735 Place the content at the beginning of the file.
4736 - end
4737 Place the content at the end of the file.
4738
4739 before
4740 Regular expression or an exact case-sensitive fragment of the string.
4741 Will be tried as **both** a regex **and** a part of the line. Must
4742 match **exactly** one line in the file. This value is only used in
4743 ``ensure`` and ``insert`` modes. The ``content`` will be inserted just
4744 before this line, matching its ``indent`` unless ``indent=False``.
4745
4746 after
4747 Regular expression or an exact case-sensitive fragment of the string.
4748 Will be tried as **both** a regex **and** a part of the line. Must
4749 match **exactly** one line in the file. This value is only used in
4750 ``ensure`` and ``insert`` modes. The ``content`` will be inserted
4751 directly after this line, unless ``before`` is also provided. If
4752 ``before`` is not matched, indentation will match this line, unless
4753 ``indent=False``.
4754
4755 show_changes
4756 Output a unified diff of the old file and the new file.
4757 If ``False`` return a boolean if any changes were made.
4758 Default is ``True``
4759
4760 .. note::
4761 Using this option will store two copies of the file in-memory
4762 (the original version and the edited version) in order to generate the diff.
4763
4764 backup
4765 Create a backup of the original file with the extension:
4766 "Year-Month-Day-Hour-Minutes-Seconds".
4767
4768 quiet
4769 Do not raise any exceptions. E.g. ignore the fact that the file that is
4770 tried to be edited does not exist and nothing really happened.
4771
4772 indent
4773 Keep indentation with the previous line. This option is not considered when
4774 the ``delete`` mode is specified. Default is ``True``.
4775
4776 create
4777 Create an empty file if doesn't exist.
4778
4779 .. versionadded:: 2016.11.0
4780
4781 user
4782 The user to own the file, this defaults to the user salt is running as
4783 on the minion.
4784
4785 .. versionadded:: 2016.11.0
4786
4787 group
4788 The group ownership set for the file, this defaults to the group salt
4789 is running as on the minion On Windows, this is ignored.
4790
4791 .. versionadded:: 2016.11.0
4792
4793 file_mode
4794 The permissions to set on this file, aka 644, 0775, 4664. Not supported
4795 on Windows.
4796
4797 .. versionadded:: 2016.11.0
4798
4799 If an equal sign (``=``) appears in an argument to a Salt command, it is
4800 interpreted as a keyword argument in the format of ``key=val``. That
4801 processing can be bypassed in order to pass an equal sign through to the
4802 remote shell command by manually specifying the kwarg:
4803
4804 .. code-block:: yaml
4805
4806 update_config:
4807 file.line:
4808 - name: /etc/myconfig.conf
4809 - mode: ensure
4810 - content: my key = my value
4811 - before: somekey.*?
4812
4813
4814 **Examples:**
4815
4816 Here's a simple config file.
4817
4818 .. code-block:: ini
4819
4820 [some_config]
4821 # Some config file
4822 # this line will go away
4823
4824 here=False
4825 away=True
4826 goodybe=away
4827
4828 And an sls file:
4829
4830 .. code-block:: yaml
4831
4832 remove_lines:
4833 file.line:
4834 - name: /some/file.conf
4835 - mode: delete
4836 - match: away
4837
4838 This will produce:
4839
4840 .. code-block:: ini
4841
4842 [some_config]
4843 # Some config file
4844
4845 here=False
4846 away=True
4847 goodbye=away
4848
4849 If that state is executed 2 more times, this will be the result:
4850
4851 .. code-block:: ini
4852
4853 [some_config]
4854 # Some config file
4855
4856 here=False
4857
4858 Given that original file with this state:
4859
4860 .. code-block:: yaml
4861
4862 replace_things:
4863 file.line:
4864 - name: /some/file.conf
4865 - mode: replace
4866 - match: away
4867 - content: here
4868
4869 Three passes will this state will result in this file:
4870
4871 .. code-block:: ini
4872
4873 [some_config]
4874 # Some config file
4875 here
4876
4877 here=False
4878 here
4879 here
4880
4881 Each pass replacing the first line found.
4882
4883 Given this file:
4884
4885 .. code-block:: text
4886
4887 insert after me
4888 something
4889 insert before me
4890
4891 The following state:
4892
4893 .. code-block:: yaml
4894
4895 insert_a_line:
4896 file.line:
4897 - name: /some/file.txt
4898 - mode: insert
4899 - after: insert after me
4900 - before: insert before me
4901 - content: thrice
4902
4903 If this state is executed 3 times, the result will be:
4904
4905 .. code-block:: text
4906
4907 insert after me
4908 something
4909 thrice
4910 thrice
4911 thrice
4912 insert before me
4913
4914 If the mode is ensure instead, it will fail each time. To succeed, we need
4915 to remove the incorrect line between before and after:
4916
4917 .. code-block:: text
4918
4919 insert after me
4920 insert before me
4921
4922 With an ensure mode, this will insert ``thrice`` the first time and
4923 make no changes for subsequent calls. For something simple this is
4924 fine, but if you have instead blocks like this:
4925
4926 .. code-block:: text
4927
4928 Begin SomeBlock
4929 foo = bar
4930 End
4931
4932 Begin AnotherBlock
4933 another = value
4934 End
4935
4936 And given this state:
4937
4938 .. code-block:: yaml
4939
4940 ensure_someblock:
4941 file.line:
4942 - name: /some/file.conf
4943 - mode: ensure
4944 - after: Begin SomeBlock
4945 - content: this = should be my content
4946 - before: End
4947
4948 This will fail because there are multiple ``End`` lines. Without that
4949 problem, it still would fail because there is a non-matching line,
4950 ``foo = bar``. Ensure **only** allows either zero, or the matching
4951 line present to be present in between ``before`` and ``after``.
4952 """
4953 name = os.path.expanduser(name)
4954 ret = {"name": name, "changes": {}, "result": True, "comment": ""}
4955 if not name:
4956 return _error(ret, "Must provide name to file.line")
4957
4958 managed(name, create=create, user=user, group=group, mode=file_mode, replace=False)
4959
4960 check_res, check_msg = _check_file(name)
4961 if not check_res:
4962 return _error(ret, check_msg)
4963
4964 # We've set the content to be empty in the function params but we want to make sure
4965 # it gets passed when needed. Feature #37092
4966 mode = mode and mode.lower() or mode
4967 if mode is None:
4968 return _error(ret, "Mode was not defined. How to process the file?")
4969
4970 modeswithemptycontent = ["delete"]
4971 if mode not in modeswithemptycontent and content is None:
4972 return _error(
4973 ret,
4974 "Content can only be empty if mode is {}".format(modeswithemptycontent),
4975 )
4976 del modeswithemptycontent
4977
4978 changes = __salt__["file.line"](
4979 name,
4980 content,
4981 match=match,
4982 mode=mode,
4983 location=location,
4984 before=before,
4985 after=after,
4986 show_changes=show_changes,
4987 backup=backup,
4988 quiet=quiet,
4989 indent=indent,
4990 )
4991 if changes:
4992 ret["changes"]["diff"] = changes
4993 if __opts__["test"]:
4994 ret["result"] = None
4995 ret["comment"] = "Changes would be made"
4996 else:
4997 ret["result"] = True
4998 ret["comment"] = "Changes were made"
4999 else:
5000 ret["result"] = True
5001 ret["comment"] = "No changes needed to be made"
5002
5003 return ret
5004
5005
5006 def replace(
5007 name,
5008 pattern,
5009 repl,
5010 count=0,
5011 flags=8,
5012 bufsize=1,
5013 append_if_not_found=False,
5014 prepend_if_not_found=False,
5015 not_found_content=None,
5016 backup=".bak",
5017 show_changes=True,
5018 ignore_if_missing=False,
5019 backslash_literal=False,
5020 ):
5021 r"""
5022 Maintain an edit in a file.
5023
5024 .. versionadded:: 0.17.0
5025
5026 name
5027 Filesystem path to the file to be edited. If a symlink is specified, it
5028 will be resolved to its target.
5029
5030 pattern
5031 A regular expression, to be matched using Python's
5032 :py:func:`re.search`.
5033
5034 .. note::
5035
5036 If you need to match a literal string that contains regex special
5037 characters, you may want to use salt's custom Jinja filter,
5038 ``regex_escape``.
5039
5040 .. code-block:: jinja
5041
5042 {{ 'http://example.com?foo=bar%20baz' | regex_escape }}
5043
5044 repl
5045 The replacement text
5046
5047 count
5048 Maximum number of pattern occurrences to be replaced. Defaults to 0.
5049 If count is a positive integer n, no more than n occurrences will be
5050 replaced, otherwise all occurrences will be replaced.
5051
5052 flags
5053 A list of flags defined in the ``re`` module documentation from the
5054 Python standard library. Each list item should be a string that will
5055 correlate to the human-friendly flag name. E.g., ``['IGNORECASE',
5056 'MULTILINE']``. Optionally, ``flags`` may be an int, with a value
5057 corresponding to the XOR (``|``) of all the desired flags. Defaults to
5058 ``8`` (which equates to ``['MULTILINE']``).
5059
5060 .. note::
5061
5062 ``file.replace`` reads the entire file as a string to support
5063 multiline regex patterns. Therefore, when using anchors such as
5064 ``^`` or ``$`` in the pattern, those anchors may be relative to
5065 the line OR relative to the file. The default for ``file.replace``
5066 is to treat anchors as relative to the line, which is implemented
5067 by setting the default value of ``flags`` to ``['MULTILINE']``.
5068 When overriding the default value for ``flags``, if
5069 ``'MULTILINE'`` is not present then anchors will be relative to
5070 the file. If the desired behavior is for anchors to be relative to
5071 the line, then simply add ``'MULTILINE'`` to the list of flags.
5072
5073 bufsize
5074 How much of the file to buffer into memory at once. The default value
5075 ``1`` processes one line at a time. The special value ``file`` may be
5076 specified which will read the entire file into memory before
5077 processing.
5078
5079 append_if_not_found : False
5080 If set to ``True``, and pattern is not found, then the content will be
5081 appended to the file.
5082
5083 .. versionadded:: 2014.7.0
5084
5085 prepend_if_not_found : False
5086 If set to ``True`` and pattern is not found, then the content will be
5087 prepended to the file.
5088
5089 .. versionadded:: 2014.7.0
5090
5091 not_found_content
5092 Content to use for append/prepend if not found. If ``None`` (default),
5093 uses ``repl``. Useful when ``repl`` uses references to group in
5094 pattern.
5095
5096 .. versionadded:: 2014.7.0
5097
5098 backup
5099 The file extension to use for a backup of the file before editing. Set
5100 to ``False`` to skip making a backup.
5101
5102 show_changes : True
5103 Output a unified diff of the old file and the new file. If ``False``
5104 return a boolean if any changes were made. Returns a boolean or a
5105 string.
5106
5107 .. note:
5108 Using this option will store two copies of the file in memory (the
5109 original version and the edited version) in order to generate the
5110 diff. This may not normally be a concern, but could impact
5111 performance if used with large files.
5112
5113 ignore_if_missing : False
5114 .. versionadded:: 2016.3.4
5115
5116 Controls what to do if the file is missing. If set to ``False``, the
5117 state will display an error raised by the execution module. If set to
5118 ``True``, the state will simply report no changes.
5119
5120 backslash_literal : False
5121 .. versionadded:: 2016.11.7
5122
5123 Interpret backslashes as literal backslashes for the repl and not
5124 escape characters. This will help when using append/prepend so that
5125 the backslashes are not interpreted for the repl on the second run of
5126 the state.
5127
5128 For complex regex patterns, it can be useful to avoid the need for complex
5129 quoting and escape sequences by making use of YAML's multiline string
5130 syntax.
5131
5132 .. code-block:: yaml
5133
5134 complex_search_and_replace:
5135 file.replace:
5136 # <...snip...>
5137 - pattern: |
5138 CentOS \(2.6.32[^\\n]+\\n\s+root[^\\n]+\\n\)+
5139
5140 .. note::
5141
5142 When using YAML multiline string syntax in ``pattern:``, make sure to
5143 also use that syntax in the ``repl:`` part, or you might loose line
5144 feeds.
5145
5146 When regex capture groups are used in ``pattern:``, their captured value is
5147 available for reuse in the ``repl:`` part as a backreference (ex. ``\1``).
5148
5149 .. code-block:: yaml
5150
5151 add_login_group_to_winbind_ssh_access_list:
5152 file.replace:
5153 - name: '/etc/security/pam_winbind.conf'
5154 - pattern: '^(require_membership_of = )(.*)$'
5155 - repl: '\1\2,append-new-group-to-line'
5156
5157 .. note::
5158
5159 The ``file.replace`` state uses Python's ``re`` module.
5160 For more advanced options, see https://docs.python.org/2/library/re.html
5161 """
5162 name = os.path.expanduser(name)
5163
5164 ret = {"name": name, "changes": {}, "result": True, "comment": ""}
5165 if not name:
5166 return _error(ret, "Must provide name to file.replace")
5167
5168 check_res, check_msg = _check_file(name)
5169 if not check_res:
5170 if ignore_if_missing and "file not found" in check_msg:
5171 ret["comment"] = "No changes needed to be made"
5172 return ret
5173 else:
5174 return _error(ret, check_msg)
5175
5176 changes = __salt__["file.replace"](
5177 name,
5178 pattern,
5179 repl,
5180 count=count,
5181 flags=flags,
5182 bufsize=bufsize,
5183 append_if_not_found=append_if_not_found,
5184 prepend_if_not_found=prepend_if_not_found,
5185 not_found_content=not_found_content,
5186 backup=backup,
5187 dry_run=__opts__["test"],
5188 show_changes=show_changes,
5189 ignore_if_missing=ignore_if_missing,
5190 backslash_literal=backslash_literal,
5191 )
5192
5193 if changes:
5194 ret["changes"]["diff"] = changes
5195 if __opts__["test"]:
5196 ret["result"] = None
5197 ret["comment"] = "Changes would have been made"
5198 else:
5199 ret["result"] = True
5200 ret["comment"] = "Changes were made"
5201 else:
5202 ret["result"] = True
5203 ret["comment"] = "No changes needed to be made"
5204
5205 return ret
5206
5207
5208 def keyvalue(
5209 name,
5210 key=None,
5211 value=None,
5212 key_values=None,
5213 separator="=",
5214 append_if_not_found=False,
5215 prepend_if_not_found=False,
5216 search_only=False,
5217 show_changes=True,
5218 ignore_if_missing=False,
5219 count=1,
5220 uncomment=None,
5221 key_ignore_case=False,
5222 value_ignore_case=False,
5223 ):
5224 """
5225 Key/Value based editing of a file.
5226
5227 .. versionadded:: 3001
5228
5229 This function differs from ``file.replace`` in that it is able to search for
5230 keys, followed by a customizable separator, and replace the value with the
5231 given value. Should the value be the same as the one already in the file, no
5232 changes will be made.
5233
5234 Either supply both ``key`` and ``value`` parameters, or supply a dictionary
5235 with key / value pairs. It is an error to supply both.
5236
5237 name
5238 Name of the file to search/replace in.
5239
5240 key
5241 Key to search for when ensuring a value. Use in combination with a
5242 ``value`` parameter.
5243
5244 value
5245 Value to set for a given key. Use in combination with a ``key``
5246 parameter.
5247
5248 key_values
5249 Dictionary of key / value pairs to search for and ensure values for.
5250 Used to specify multiple key / values at once.
5251
5252 separator : "="
5253 Separator which separates key from value.
5254
5255 append_if_not_found : False
5256 Append the key/value to the end of the file if not found. Note that this
5257 takes precedence over ``prepend_if_not_found``.
5258
5259 prepend_if_not_found : False
5260 Prepend the key/value to the beginning of the file if not found. Note
5261 that ``append_if_not_found`` takes precedence.
5262
5263 show_changes : True
5264 Show a diff of the resulting removals and inserts.
5265
5266 ignore_if_missing : False
5267 Return with success even if the file is not found (or not readable).
5268
5269 count : 1
5270 Number of occurrences to allow (and correct), default is 1. Set to -1 to
5271 replace all, or set to 0 to remove all lines with this key regardsless
5272 of its value.
5273
5274 .. note::
5275 Any additional occurrences after ``count`` are removed.
5276 A count of -1 will only replace all occurrences that are currently
5277 uncommented already. Lines commented out will be left alone.
5278
5279 uncomment : None
5280 Disregard and remove supplied leading characters when finding keys. When
5281 set to None, lines that are commented out are left for what they are.
5282
5283 .. note::
5284 The argument to ``uncomment`` is not a prefix string. Rather; it is a
5285 set of characters, each of which are stripped.
5286
5287 key_ignore_case : False
5288 Keys are matched case insensitively. When a value is changed the matched
5289 key is kept as-is.
5290
5291 value_ignore_case : False
5292 Values are checked case insensitively, trying to set e.g. 'Yes' while
5293 the current value is 'yes', will not result in changes when
5294 ``value_ignore_case`` is set to True.
5295
5296 An example of using ``file.keyvalue`` to ensure sshd does not allow
5297 for root to login with a password and at the same time setting the
5298 login-gracetime to 1 minute and disabling all forwarding:
5299
5300 .. code-block:: yaml
5301
5302 sshd_config_harden:
5303 file.keyvalue:
5304 - name: /etc/ssh/sshd_config
5305 - key_values:
5306 permitrootlogin: 'without-password'
5307 LoginGraceTime: '1m'
5308 DisableForwarding: 'yes'
5309 - separator: ' '
5310 - uncomment: '# '
5311 - key_ignore_case: True
5312 - append_if_not_found: True
5313
5314 The same example, except for only ensuring PermitRootLogin is set correctly.
5315 Thus being able to use the shorthand ``key`` and ``value`` parameters
5316 instead of ``key_values``.
5317
5318 .. code-block:: yaml
5319
5320 sshd_config_harden:
5321 file.keyvalue:
5322 - name: /etc/ssh/sshd_config
5323 - key: PermitRootLogin
5324 - value: without-password
5325 - separator: ' '
5326 - uncomment: '# '
5327 - key_ignore_case: True
5328 - append_if_not_found: True
5329
5330 .. note::
5331 Notice how the key is not matched case-sensitively, this way it will
5332 correctly identify both 'PermitRootLogin' as well as 'permitrootlogin'.
5333
5334 """
5335 name = os.path.expanduser(name)
5336
5337 # default return values
5338 ret = {
5339 "name": name,
5340 "changes": {},
5341 "result": None,
5342 "comment": "",
5343 }
5344
5345 if not name:
5346 return _error(ret, "Must provide name to file.keyvalue")
5347 if key is not None and value is not None:
5348 if type(key_values) is dict:
5349 return _error(
5350 ret, "file.keyvalue can not combine key_values with key and value"
5351 )
5352 key_values = {str(key): value}
5353
5354 elif not isinstance(key_values, dict) or not key_values:
5355 msg = "is not a dictionary"
5356 if not key_values:
5357 msg = "is empty"
5358 return _error(
5359 ret, "file.keyvalue key and value not supplied and key_values " + msg,
5360 )
5361
5362 # try to open the file and only return a comment if ignore_if_missing is
5363 # enabled, also mark as an error if not
5364 file_contents = []
5365 try:
5366 with salt.utils.files.fopen(name, "r") as fd:
5367 file_contents = fd.readlines()
5368 except OSError:
5369 ret["comment"] = "unable to open {n}".format(n=name)
5370 ret["result"] = True if ignore_if_missing else False
5371 return ret
5372
5373 # used to store diff combinations and check if anything has changed
5374 diff = []
5375 # store the final content of the file in case it needs to be rewritten
5376 content = []
5377 # target format is templated like this
5378 tmpl = "{key}{sep}{value}" + os.linesep
5379 # number of lines changed
5380 changes = 0
5381 # keep track of number of times a key was updated
5382 diff_count = {k: count for k in key_values.keys()}
5383
5384 # read all the lines from the file
5385 for line in file_contents:
5386 test_line = line.lstrip(uncomment)
5387 did_uncomment = True if len(line) > len(test_line) else False
5388
5389 if key_ignore_case:
5390 test_line = test_line.lower()
5391
5392 for key, value in key_values.items():
5393 test_key = key.lower() if key_ignore_case else key
5394 # if the line starts with the key
5395 if test_line.startswith(test_key):
5396 # if the testline got uncommented then the real line needs to
5397 # be uncommented too, otherwhise there might be separation on
5398 # a character which is part of the comment set
5399 working_line = line.lstrip(uncomment) if did_uncomment else line
5400
5401 # try to separate the line into its' components
5402 line_key, line_sep, line_value = working_line.partition(separator)
5403
5404 # if separation was unsuccessful then line_sep is empty so
5405 # no need to keep trying. continue instead
5406 if line_sep != separator:
5407 continue
5408
5409 # start on the premises the key does not match the actual line
5410 keys_match = False
5411 if key_ignore_case:
5412 if line_key.lower() == test_key:
5413 keys_match = True
5414 else:
5415 if line_key == test_key:
5416 keys_match = True
5417
5418 # if the key was found in the line and separation was successful
5419 if keys_match:
5420 # trial and error have shown it's safest to strip whitespace
5421 # from values for the sake of matching
5422 line_value = line_value.strip()
5423 # make sure the value is an actual string at this point
5424 test_value = str(value).strip()
5425 # convert test_value and line_value to lowercase if need be
5426 if value_ignore_case:
5427 line_value = line_value.lower()
5428 test_value = test_value.lower()
5429
5430 # values match if they are equal at this point
5431 values_match = True if line_value == test_value else False
5432
5433 # in case a line had its comment removed there are some edge
5434 # cases that need considderation where changes are needed
5435 # regardless of values already matching.
5436 needs_changing = False
5437 if did_uncomment:
5438 # irrespective of a value, if it was commented out and
5439 # changes are still to be made, then it needs to be
5440 # commented in
5441 if diff_count[key] > 0:
5442 needs_changing = True
5443 # but if values did not match but there are really no
5444 # changes expected anymore either then leave this line
5445 elif not values_match:
5446 values_match = True
5447 else:
5448 # a line needs to be removed if it has been seen enough
5449 # times and was not commented out, regardless of value
5450 if diff_count[key] == 0:
5451 needs_changing = True
5452
5453 # then start checking to see if the value needs replacing
5454 if not values_match or needs_changing:
5455 # the old line always needs to go, so that will be
5456 # reflected in the diff (this is the original line from
5457 # the file being read)
5458 diff.append("- {}".format(line))
5459 line = line[:0]
5460
5461 # any non-zero value means something needs to go back in
5462 # its place. negative values are replacing all lines not
5463 # commented out, positive values are having their count
5464 # reduced by one every replacement
5465 if diff_count[key] != 0:
5466 # rebuild the line using the key and separator found
5467 # and insert the correct value.
5468 line = str(
5469 tmpl.format(key=line_key, sep=line_sep, value=value)
5470 )
5471
5472 # display a comment in case a value got converted
5473 # into a string
5474 if not isinstance(value, str):
5475 diff.append(
5476 "+ {} (from {} type){}".format(
5477 line.rstrip(), type(value).__name__, os.linesep
5478 )
5479 )
5480 else:
5481 diff.append("+ {}".format(line))
5482 changes += 1
5483 # subtract one from the count if it was larger than 0, so
5484 # next lines are removed. if it is less than 0 then count is
5485 # ignored and all lines will be updated.
5486 if diff_count[key] > 0:
5487 diff_count[key] -= 1
5488 # at this point a continue saves going through the rest of
5489 # the keys to see if they match since this line already
5490 # matched the current key
5491 continue
5492 # with the line having been checked for all keys (or matched before all
5493 # keys needed searching), the line can be added to the content to be
5494 # written once the last checks have been performed
5495 content.append(line)
5496 # finally, close the file
5497 fd.close()
5498
5499 # if append_if_not_found was requested, then append any key/value pairs
5500 # still having a count left on them
5501 if append_if_not_found:
5502 tmpdiff = []
5503 for key, value in key_values.items():
5504 if diff_count[key] > 0:
5505 line = tmpl.format(key=key, sep=separator, value=value)
5506 tmpdiff.append("+ {}".format(line))
5507 content.append(line)
5508 changes += 1
5509 if tmpdiff:
5510 tmpdiff.insert(0, "- <EOF>" + os.linesep)
5511 tmpdiff.append("+ <EOF>" + os.linesep)
5512 diff.extend(tmpdiff)
5513 # only if append_if_not_found was not set should prepend_if_not_found be
5514 # considered, benefit of this is that the number of counts left does not
5515 # mean there might be both a prepend and append happening
5516 elif prepend_if_not_found:
5517 did_diff = False
5518 for key, value in key_values.items():
5519 if diff_count[key] > 0:
5520 line = tmpl.format(key=key, sep=separator, value=value)
5521 if not did_diff:
5522 diff.insert(0, " <SOF>" + os.linesep)
5523 did_diff = True
5524 diff.insert(1, "+ {}".format(line))
5525 content.insert(0, line)
5526 changes += 1
5527
5528 # if a diff was made
5529 if changes > 0:
5530 # return comment of changes if test
5531 if __opts__["test"]:
5532 ret["comment"] = "File {n} is set to be changed ({c} lines)".format(
5533 n=name, c=changes
5534 )
5535 if show_changes:
5536 # For some reason, giving an actual diff even in test=True mode
5537 # will be seen as both a 'changed' and 'unchanged'. this seems to
5538 # match the other modules behaviour though
5539 ret["changes"]["diff"] = "".join(diff)
5540
5541 # add changes to comments for now as well because of how
5542 # stateoutputter seems to handle changes etc.
5543 # See: https://github.com/saltstack/salt/issues/40208
5544 ret["comment"] += "\nPredicted diff:\n\r\t\t"
5545 ret["comment"] += "\r\t\t".join(diff)
5546 ret["result"] = None
5547
5548 # otherwise return the actual diff lines
5549 else:
5550 ret["comment"] = "Changed {c} lines".format(c=changes)
5551 if show_changes:
5552 ret["changes"]["diff"] = "".join(diff)
5553 else:
5554 ret["result"] = True
5555 return ret
5556
5557 # if not test=true, try and write the file
5558 if not __opts__["test"]:
5559 try:
5560 with salt.utils.files.fopen(name, "w") as fd:
5561 # write all lines to the file which was just truncated
5562 fd.writelines(content)
5563 fd.close()
5564 except OSError:
5565 # return an error if the file was not writable
5566 ret["comment"] = "{n} not writable".format(n=name)
5567 ret["result"] = False
5568 return ret
5569 # if all went well, then set result to true
5570 ret["result"] = True
5571
5572 return ret
5573
5574
5575 def blockreplace(
5576 name,
5577 marker_start="#-- start managed zone --",
5578 marker_end="#-- end managed zone --",
5579 source=None,
5580 source_hash=None,
5581 template="jinja",
5582 sources=None,
5583 source_hashes=None,
5584 defaults=None,
5585 context=None,
5586 content="",
5587 append_if_not_found=False,
5588 prepend_if_not_found=False,
5589 backup=".bak",
5590 show_changes=True,
5591 append_newline=None,
5592 insert_before_match=None,
5593 insert_after_match=None,
5594 ):
5595 """
5596 Maintain an edit in a file in a zone delimited by two line markers
5597
5598 .. versionadded:: 2014.1.0
5599 .. versionchanged:: 2017.7.5,2018.3.1
5600 ``append_newline`` argument added. Additionally, to improve
5601 idempotence, if the string represented by ``marker_end`` is found in
5602 the middle of the line, the content preceding the marker will be
5603 removed when the block is replaced. This allows one to remove
5604 ``append_newline: False`` from the SLS and have the block properly
5605 replaced if the end of the content block is immediately followed by the
5606 ``marker_end`` (i.e. no newline before the marker).
5607
5608 A block of content delimited by comments can help you manage several lines
5609 entries without worrying about old entries removal. This can help you
5610 maintaining an un-managed file containing manual edits.
5611
5612 .. note::
5613 This function will store two copies of the file in-memory (the original
5614 version and the edited version) in order to detect changes and only
5615 edit the targeted file if necessary.
5616
5617 Additionally, you can use :py:func:`file.accumulated
5618 <salt.states.file.accumulated>` and target this state. All accumulated
5619 data dictionaries' content will be added in the content block.
5620
5621 name
5622 Filesystem path to the file to be edited
5623
5624 marker_start
5625 The line content identifying a line as the start of the content block.
5626 Note that the whole line containing this marker will be considered, so
5627 whitespace or extra content before or after the marker is included in
5628 final output
5629
5630 marker_end
5631 The line content identifying the end of the content block. As of
5632 versions 2017.7.5 and 2018.3.1, everything up to the text matching the
5633 marker will be replaced, so it's important to ensure that your marker
5634 includes the beginning of the text you wish to replace.
5635
5636 content
5637 The content to be used between the two lines identified by
5638 ``marker_start`` and ``marker_end``
5639
5640 source
5641 The source file to download to the minion, this source file can be
5642 hosted on either the salt master server, or on an HTTP or FTP server.
5643 Both HTTPS and HTTP are supported as well as downloading directly
5644 from Amazon S3 compatible URLs with both pre-configured and automatic
5645 IAM credentials. (see s3.get state documentation)
5646 File retrieval from Openstack Swift object storage is supported via
5647 swift://container/object_path URLs, see swift.get documentation.
5648 For files hosted on the salt file server, if the file is located on
5649 the master in the directory named spam, and is called eggs, the source
5650 string is salt://spam/eggs. If source is left blank or None
5651 (use ~ in YAML), the file will be created as an empty file and
5652 the content will not be managed. This is also the case when a file
5653 already exists and the source is undefined; the contents of the file
5654 will not be changed or managed.
5655
5656 If the file is hosted on a HTTP or FTP server then the source_hash
5657 argument is also required.
5658
5659 A list of sources can also be passed in to provide a default source and
5660 a set of fallbacks. The first source in the list that is found to exist
5661 will be used and subsequent entries in the list will be ignored.
5662
5663 .. code-block:: yaml
5664
5665 file_override_example:
5666 file.blockreplace:
5667 - name: /etc/example.conf
5668 - source:
5669 - salt://file_that_does_not_exist
5670 - salt://file_that_exists
5671
5672 source_hash
5673 This can be one of the following:
5674 1. a source hash string
5675 2. the URI of a file that contains source hash strings
5676
5677 The function accepts the first encountered long unbroken alphanumeric
5678 string of correct length as a valid hash, in order from most secure to
5679 least secure:
5680
5681 .. code-block:: text
5682
5683 Type Length
5684 ====== ======
5685 sha512 128
5686 sha384 96
5687 sha256 64
5688 sha224 56
5689 sha1 40
5690 md5 32
5691
5692 See the ``source_hash`` parameter description for :mod:`file.managed
5693 <salt.states.file.managed>` function for more details and examples.
5694
5695 template : jinja
5696 Templating engine to be used to render the downloaded file. The
5697 following engines are supported:
5698
5699 - :mod:`cheetah <salt.renderers.cheetah>`
5700 - :mod:`genshi <salt.renderers.genshi>`
5701 - :mod:`jinja <salt.renderers.jinja>`
5702 - :mod:`mako <salt.renderers.mako>`
5703 - :mod:`py <salt.renderers.py>`
5704 - :mod:`wempy <salt.renderers.wempy>`
5705
5706 context
5707 Overrides default context variables passed to the template
5708
5709 defaults
5710 Default context passed to the template
5711
5712 append_if_not_found : False
5713 If markers are not found and this option is set to ``True``, the
5714 content block will be appended to the file.
5715
5716 prepend_if_not_found : False
5717 If markers are not found and this option is set to ``True``, the
5718 content block will be prepended to the file.
5719
5720 insert_before_match
5721 If markers are not found, this parameter can be set to a regex which will
5722 insert the block before the first found occurrence in the file.
5723
5724 .. versionadded:: Sodium
5725
5726 insert_after_match
5727 If markers are not found, this parameter can be set to a regex which will
5728 insert the block after the first found occurrence in the file.
5729
5730 .. versionadded:: Sodium
5731
5732 backup
5733 The file extension to use for a backup of the file if any edit is made.
5734 Set this to ``False`` to skip making a backup.
5735
5736 dry_run : False
5737 If ``True``, do not make any edits to the file and simply return the
5738 changes that *would* be made.
5739
5740 show_changes : True
5741 Controls how changes are presented. If ``True``, the ``Changes``
5742 section of the state return will contain a unified diff of the changes
5743 made. If False, then it will contain a boolean (``True`` if any changes
5744 were made, otherwise ``False``).
5745
5746 append_newline
5747 Controls whether or not a newline is appended to the content block. If
5748 the value of this argument is ``True`` then a newline will be added to
5749 the content block. If it is ``False``, then a newline will *not* be
5750 added to the content block. If it is unspecified, then a newline will
5751 only be added to the content block if it does not already end in a
5752 newline.
5753
5754 .. versionadded:: 2017.7.5,2018.3.1
5755
5756 Example of usage with an accumulator and with a variable:
5757
5758 .. code-block:: jinja
5759
5760 {% set myvar = 42 %}
5761 hosts-config-block-{{ myvar }}:
5762 file.blockreplace:
5763 - name: /etc/hosts
5764 - marker_start: "# START managed zone {{ myvar }} -DO-NOT-EDIT-"
5765 - marker_end: "# END managed zone {{ myvar }} --"
5766 - content: 'First line of content'
5767 - append_if_not_found: True
5768 - backup: '.bak'
5769 - show_changes: True
5770
5771 hosts-config-block-{{ myvar }}-accumulated1:
5772 file.accumulated:
5773 - filename: /etc/hosts
5774 - name: my-accumulator-{{ myvar }}
5775 - text: "text 2"
5776 - require_in:
5777 - file: hosts-config-block-{{ myvar }}
5778
5779 hosts-config-block-{{ myvar }}-accumulated2:
5780 file.accumulated:
5781 - filename: /etc/hosts
5782 - name: my-accumulator-{{ myvar }}
5783 - text: |
5784 text 3
5785 text 4
5786 - require_in:
5787 - file: hosts-config-block-{{ myvar }}
5788
5789 will generate and maintain a block of content in ``/etc/hosts``:
5790
5791 .. code-block:: text
5792
5793 # START managed zone 42 -DO-NOT-EDIT-
5794 First line of content
5795 text 2
5796 text 3
5797 text 4
5798 # END managed zone 42 --
5799 """
5800 name = os.path.expanduser(name)
5801
5802 ret = {"name": name, "changes": {}, "result": False, "comment": ""}
5803 if not name:
5804 return _error(ret, "Must provide name to file.blockreplace")
5805
5806 if sources is None:
5807 sources = []
5808 if source_hashes is None:
5809 source_hashes = []
5810
5811 (ok_, err, sl_) = _unify_sources_and_hashes(
5812 source=source,
5813 source_hash=source_hash,
5814 sources=sources,
5815 source_hashes=source_hashes,
5816 )
5817 if not ok_:
5818 return _error(ret, err)
5819
5820 check_res, check_msg = _check_file(name)
5821 if not check_res:
5822 return _error(ret, check_msg)
5823
5824 accum_data, accum_deps = _load_accumulators()
5825 if name in accum_data:
5826 accumulator = accum_data[name]
5827 # if we have multiple accumulators for a file, only apply the one
5828 # required at a time
5829 deps = accum_deps.get(name, [])
5830 filtered = [
5831 a for a in deps if __low__["__id__"] in deps[a] and a in accumulator
5832 ]
5833 if not filtered:
5834 filtered = [a for a in accumulator]
5835 for acc in filtered:
5836 acc_content = accumulator[acc]
5837 for line in acc_content:
5838 if content == "":
5839 content = line
5840 else:
5841 content += "\n" + line
5842
5843 if sl_:
5844 tmpret = _get_template_texts(
5845 source_list=sl_, template=template, defaults=defaults, context=context
5846 )
5847 if not tmpret["result"]:
5848 return tmpret
5849 text = tmpret["data"]
5850
5851 for index, item in enumerate(text):
5852 content += str(item)
5853
5854 try:
5855 changes = __salt__["file.blockreplace"](
5856 name,
5857 marker_start,
5858 marker_end,
5859 content=content,
5860 append_if_not_found=append_if_not_found,
5861 prepend_if_not_found=prepend_if_not_found,
5862 insert_before_match=insert_before_match,
5863 insert_after_match=insert_after_match,
5864 backup=backup,
5865 dry_run=__opts__["test"],
5866 show_changes=show_changes,
5867 append_newline=append_newline,
5868 )
5869 except Exception as exc: # pylint: disable=broad-except
5870 log.exception("Encountered error managing block")
5871 ret["comment"] = (
5872 "Encountered error managing block: {}. "
5873 "See the log for details.".format(exc)
5874 )
5875 return ret
5876
5877 if changes:
5878 ret["changes"]["diff"] = changes
5879 if __opts__["test"]:
5880 ret["result"] = None
5881 ret["comment"] = "Changes would be made"
5882 else:
5883 ret["result"] = True
5884 ret["comment"] = "Changes were made"
5885 else:
5886 ret["result"] = True
5887 ret["comment"] = "No changes needed to be made"
5888
5889 return ret
5890
5891
5892 def comment(name, regex, char="#", backup=".bak"):
5893 """
5894 Comment out specified lines in a file.
5895
5896 name
5897 The full path to the file to be edited
5898 regex
5899 A regular expression used to find the lines that are to be commented;
5900 this pattern will be wrapped in parenthesis and will move any
5901 preceding/trailing ``^`` or ``$`` characters outside the parenthesis
5902 (e.g., the pattern ``^foo$`` will be rewritten as ``^(foo)$``)
5903 Note that you _need_ the leading ^, otherwise each time you run
5904 highstate, another comment char will be inserted.
5905 char : ``#``
5906 The character to be inserted at the beginning of a line in order to
5907 comment it out
5908 backup : ``.bak``
5909 The file will be backed up before edit with this file extension
5910
5911 .. warning::
5912
5913 This backup will be overwritten each time ``sed`` / ``comment`` /
5914 ``uncomment`` is called. Meaning the backup will only be useful
5915 after the first invocation.
5916
5917 Set to False/None to not keep a backup.
5918
5919 Usage:
5920
5921 .. code-block:: yaml
5922
5923 /etc/fstab:
5924 file.comment:
5925 - regex: ^bind 127.0.0.1
5926
5927 .. versionadded:: 0.9.5
5928 """
5929 name = os.path.expanduser(name)
5930
5931 ret = {"name": name, "changes": {}, "result": False, "comment": ""}
5932 if not name:
5933 return _error(ret, "Must provide name to file.comment")
5934
5935 check_res, check_msg = _check_file(name)
5936 if not check_res:
5937 return _error(ret, check_msg)
5938
5939 # remove (?i)-like flags, ^ and $
5940 unanchor_regex = re.sub(r"^(\(\?[iLmsux]\))?\^?(.*?)\$?$", r"\2", regex)
5941
5942 comment_regex = char + unanchor_regex
5943
5944 # Make sure the pattern appears in the file before continuing
5945 if not __salt__["file.search"](name, regex, multiline=True):
5946 if __salt__["file.search"](name, comment_regex, multiline=True):
5947 ret["comment"] = "Pattern already commented"
5948 ret["result"] = True
5949 return ret
5950 else:
5951 return _error(ret, "{}: Pattern not found".format(unanchor_regex))
5952
5953 if __opts__["test"]:
5954 ret["changes"][name] = "updated"
5955 ret["comment"] = "File {} is set to be updated".format(name)
5956 ret["result"] = None
5957 return ret
5958 with salt.utils.files.fopen(name, "rb") as fp_:
5959 slines = fp_.read()
5960 slines = slines.decode(__salt_system_encoding__)
5961 slines = slines.splitlines(True)
5962
5963 # Perform the edit
5964 __salt__["file.comment_line"](name, regex, char, True, backup)
5965
5966 with salt.utils.files.fopen(name, "rb") as fp_:
5967 nlines = fp_.read()
5968 nlines = nlines.decode(__salt_system_encoding__)
5969 nlines = nlines.splitlines(True)
5970
5971 # Check the result
5972 ret["result"] = __salt__["file.search"](name, unanchor_regex, multiline=True)
5973
5974 if slines != nlines:
5975 if not __utils__["files.is_text"](name):
5976 ret["changes"]["diff"