"Fossies" - the Fresh Open Source Software Archive 
Member "salt-3002.2/salt/pillar/file_tree.py" (18 Nov 2020, 18416 Bytes) of package /linux/misc/salt-3002.2.tar.gz:
As a special service "Fossies" has tried to format the requested source page into HTML format using (guessed) Python source code syntax highlighting (style:
standard) with prefixed line numbers.
Alternatively you can here
view or
download the uninterpreted source code file.
For more information about "file_tree.py" see the
Fossies "Dox" file reference documentation and the latest
Fossies "Diffs" side-by-side code changes report:
3002.1_vs_3002.2.
1 """
2 The ``file_tree`` external pillar allows values from all files in a directory
3 tree to be imported as Pillar data.
4
5 .. note::
6
7 This is an external pillar and is subject to the :ref:`rules and
8 constraints <external-pillars>` governing external pillars.
9
10 .. versionadded:: 2015.5.0
11
12 In this pillar, data is organized by either Minion ID or Nodegroup name. To
13 setup pillar data for a specific Minion, place it in
14 ``<root_dir>/hosts/<minion_id>``. To setup pillar data for an entire
15 Nodegroup, place it in ``<root_dir>/nodegroups/<node_group>`` where
16 ``<node_group>`` is the Nodegroup's name.
17
18 Example ``file_tree`` Pillar
19 ============================
20
21 Master Configuration
22 --------------------
23
24 .. code-block:: yaml
25
26 ext_pillar:
27 - file_tree:
28 root_dir: /srv/ext_pillar
29 follow_dir_links: False
30 keep_newline: True
31
32 The ``root_dir`` parameter is required and points to the directory where files
33 for each host are stored. The ``follow_dir_links`` parameter is optional and
34 defaults to False. If ``follow_dir_links`` is set to True, this external pillar
35 will follow symbolic links to other directories.
36
37 .. warning::
38 Be careful when using ``follow_dir_links``, as a recursive symlink chain
39 will result in unexpected results.
40
41 .. versionchanged:: 2018.3.0
42 If ``root_dir`` is a relative path, it will be treated as relative to the
43 :conf_master:`pillar_roots` of the environment specified by
44 :conf_minion:`pillarenv`. If an environment specifies multiple
45 roots, this module will search for files relative to all of them, in order,
46 merging the results.
47
48 If ``keep_newline`` is set to ``True``, then the pillar values for files ending
49 in newlines will keep that newline. The default behavior is to remove the
50 end-of-file newline. ``keep_newline`` should be turned on if the pillar data is
51 intended to be used to deploy a file using ``contents_pillar`` with a
52 :py:func:`file.managed <salt.states.file.managed>` state.
53
54 .. versionchanged:: 2015.8.4
55 The ``raw_data`` parameter has been renamed to ``keep_newline``. In earlier
56 releases, ``raw_data`` must be used. Also, this parameter can now be a list
57 of globs, allowing for more granular control over which pillar values keep
58 their end-of-file newline. The globs match paths relative to the
59 directories named for minion IDs and nodegroups underneath the ``root_dir``
60 (see the layout examples in the below sections).
61
62 .. code-block:: yaml
63
64 ext_pillar:
65 - file_tree:
66 root_dir: /path/to/root/directory
67 keep_newline:
68 - files/testdir/*
69
70 .. note::
71 In earlier releases, this documentation incorrectly stated that binary
72 files would not affected by the ``keep_newline`` configuration. However,
73 this module does not actually distinguish between binary and text files.
74
75 .. versionchanged:: 2017.7.0
76 Templating/rendering has been added. You can now specify a default render
77 pipeline and a black- and whitelist of (dis)allowed renderers.
78
79 ``template`` must be set to ``True`` for templating to happen.
80
81 .. code-block:: yaml
82
83 ext_pillar:
84 - file_tree:
85 root_dir: /path/to/root/directory
86 render_default: jinja|yaml
87 renderer_blacklist:
88 - gpg
89 renderer_whitelist:
90 - jinja
91 - yaml
92 template: True
93
94 Assigning Pillar Data to Individual Hosts
95 -----------------------------------------
96
97 To configure pillar data for each host, this external pillar will recursively
98 iterate over ``root_dir``/hosts/``id`` (where ``id`` is a minion ID), and
99 compile pillar data with each subdirectory as a dictionary key and each file
100 as a value.
101
102 For example, the following ``root_dir`` tree:
103
104 .. code-block:: text
105
106 ./hosts/
107 ./hosts/test-host/
108 ./hosts/test-host/files/
109 ./hosts/test-host/files/testdir/
110 ./hosts/test-host/files/testdir/file1.txt
111 ./hosts/test-host/files/testdir/file2.txt
112 ./hosts/test-host/files/another-testdir/
113 ./hosts/test-host/files/another-testdir/symlink-to-file1.txt
114
115 will result in the following pillar tree for minion with ID ``test-host``:
116
117 .. code-block:: text
118
119 test-host:
120 ----------
121 apache:
122 ----------
123 config.d:
124 ----------
125 00_important.conf:
126 <important_config important_setting="yes" />
127 20_bob_extra.conf:
128 <bob_specific_cfg has_freeze_ray="yes" />
129 corporate_app:
130 ----------
131 settings:
132 ----------
133 common_settings:
134 // This is the main settings file for the corporate
135 // internal web app
136 main_setting: probably
137 bob_settings:
138 role: bob
139
140 .. note::
141
142 The leaf data in the example shown is the contents of the pillar files.
143 """
144
145 import fnmatch
146 import logging
147 import os
148
149 import salt.loader
150 import salt.template
151 import salt.utils.dictupdate
152 import salt.utils.files
153 import salt.utils.minions
154 import salt.utils.path
155 import salt.utils.stringio
156 import salt.utils.stringutils
157
158 # Set up logging
159 log = logging.getLogger(__name__)
160
161
162 def _on_walk_error(err):
163 """
164 Log salt.utils.path.os_walk() error.
165 """
166 log.error("%s: %s", err.filename, err.strerror)
167
168
169 def _check_newline(prefix, file_name, keep_newline):
170 """
171 Return a boolean stating whether or not a file's trailing newline should be
172 removed. To figure this out, first check if keep_newline is a boolean and
173 if so, return its opposite. Otherwise, iterate over keep_newline and check
174 if any of the patterns match the file path. If a match is found, return
175 False, otherwise return True.
176 """
177 if isinstance(keep_newline, bool):
178 return not keep_newline
179 full_path = os.path.join(prefix, file_name)
180 for pattern in keep_newline:
181 try:
182 if fnmatch.fnmatch(full_path, pattern):
183 return False
184 except TypeError:
185 if fnmatch.fnmatch(full_path, str(pattern)):
186 return False
187 return True
188
189
190 def _construct_pillar(
191 top_dir,
192 follow_dir_links,
193 keep_newline=False,
194 render_default=None,
195 renderer_blacklist=None,
196 renderer_whitelist=None,
197 template=False,
198 ):
199 """
200 Construct pillar from file tree.
201 """
202 pillar = {}
203 renderers = salt.loader.render(__opts__, __salt__)
204
205 norm_top_dir = os.path.normpath(top_dir)
206 for dir_path, dir_names, file_names in salt.utils.path.os_walk(
207 top_dir, topdown=True, onerror=_on_walk_error, followlinks=follow_dir_links
208 ):
209 # Find current path in pillar tree
210 pillar_node = pillar
211 norm_dir_path = os.path.normpath(dir_path)
212 prefix = os.path.relpath(norm_dir_path, norm_top_dir)
213 if norm_dir_path != norm_top_dir:
214 path_parts = []
215 head = prefix
216 while head:
217 head, tail = os.path.split(head)
218 path_parts.insert(0, tail)
219 while path_parts:
220 pillar_node = pillar_node[path_parts.pop(0)]
221
222 # Create dicts for subdirectories
223 for dir_name in dir_names:
224 pillar_node[dir_name] = {}
225
226 # Add files
227 for file_name in file_names:
228 file_path = os.path.join(dir_path, file_name)
229 if not os.path.isfile(file_path):
230 log.error("file_tree: %s: not a regular file", file_path)
231 continue
232
233 contents = b""
234 try:
235 with salt.utils.files.fopen(file_path, "rb") as fhr:
236 buf = fhr.read(__opts__["file_buffer_size"])
237 while buf:
238 contents += buf
239 buf = fhr.read(__opts__["file_buffer_size"])
240 if contents.endswith(b"\n") and _check_newline(
241 prefix, file_name, keep_newline
242 ):
243 contents = contents[:-1]
244 except OSError as exc:
245 log.error("file_tree: Error reading %s: %s", file_path, exc.strerror)
246 else:
247 data = contents
248 if template is True:
249 data = salt.template.compile_template_str(
250 template=salt.utils.stringutils.to_unicode(contents),
251 renderers=renderers,
252 default=render_default,
253 blacklist=renderer_blacklist,
254 whitelist=renderer_whitelist,
255 )
256 if salt.utils.stringio.is_readable(data):
257 pillar_node[file_name] = data.getvalue()
258 else:
259 pillar_node[file_name] = data
260
261 return pillar
262
263
264 def ext_pillar(
265 minion_id,
266 pillar,
267 root_dir=None,
268 follow_dir_links=False,
269 debug=False,
270 keep_newline=False,
271 render_default=None,
272 renderer_blacklist=None,
273 renderer_whitelist=None,
274 template=False,
275 ):
276 """
277 Compile pillar data from the given ``root_dir`` specific to Nodegroup names
278 and Minion IDs.
279
280 If a Minion's ID is not found at ``<root_dir>/host/<minion_id>`` or if it
281 is not included in any Nodegroups named at
282 ``<root_dir>/nodegroups/<node_group>``, no pillar data provided by this
283 pillar module will be available for that Minion.
284
285 .. versionchanged:: 2017.7.0
286 Templating/rendering has been added. You can now specify a default
287 render pipeline and a black- and whitelist of (dis)allowed renderers.
288
289 ``template`` must be set to ``True`` for templating to happen.
290
291 .. code-block:: yaml
292
293 ext_pillar:
294 - file_tree:
295 root_dir: /path/to/root/directory
296 render_default: jinja|yaml
297 renderer_blacklist:
298 - gpg
299 renderer_whitelist:
300 - jinja
301 - yaml
302 template: True
303
304 :param minion_id:
305 The ID of the Minion whose pillar data is to be collected
306
307 :param pillar:
308 Unused by the ``file_tree`` pillar module
309
310 :param root_dir:
311 Filesystem directory used as the root for pillar data (e.g.
312 ``/srv/ext_pillar``)
313
314 .. versionchanged:: 2018.3.0
315 If ``root_dir`` is a relative path, it will be treated as relative to the
316 :conf_master:`pillar_roots` of the environment specified by
317 :conf_minion:`pillarenv`. If an environment specifies multiple
318 roots, this module will search for files relative to all of them, in order,
319 merging the results.
320
321 :param follow_dir_links:
322 Follow symbolic links to directories while collecting pillar files.
323 Defaults to ``False``.
324
325 .. warning::
326
327 Care should be exercised when enabling this option as it will
328 follow links that point outside of ``root_dir``.
329
330 .. warning::
331
332 Symbolic links that lead to infinite recursion are not filtered.
333
334 :param debug:
335 Enable debug information at log level ``debug``. Defaults to
336 ``False``. This option may be useful to help debug errors when setting
337 up the ``file_tree`` pillar module.
338
339 :param keep_newline:
340 Preserve the end-of-file newline in files. Defaults to ``False``.
341 This option may either be a boolean or a list of file globs (as defined
342 by the `Python fnmatch package
343 <https://docs.python.org/library/fnmatch.html>`_) for which end-of-file
344 newlines are to be kept.
345
346 ``keep_newline`` should be turned on if the pillar data is intended to
347 be used to deploy a file using ``contents_pillar`` with a
348 :py:func:`file.managed <salt.states.file.managed>` state.
349
350 .. versionchanged:: 2015.8.4
351 The ``raw_data`` parameter has been renamed to ``keep_newline``. In
352 earlier releases, ``raw_data`` must be used. Also, this parameter
353 can now be a list of globs, allowing for more granular control over
354 which pillar values keep their end-of-file newline. The globs match
355 paths relative to the directories named for Minion IDs and
356 Nodegroup namess underneath the ``root_dir``.
357
358 .. code-block:: yaml
359
360 ext_pillar:
361 - file_tree:
362 root_dir: /srv/ext_pillar
363 keep_newline:
364 - apache/config.d/*
365 - corporate_app/settings/*
366
367 .. note::
368 In earlier releases, this documentation incorrectly stated that
369 binary files would not affected by the ``keep_newline``. However,
370 this module does not actually distinguish between binary and text
371 files.
372
373
374 :param render_default:
375 Override Salt's :conf_master:`default global renderer <renderer>` for
376 the ``file_tree`` pillar.
377
378 .. code-block:: yaml
379
380 render_default: jinja
381
382 :param renderer_blacklist:
383 Disallow renderers for pillar files.
384
385 .. code-block:: yaml
386
387 renderer_blacklist:
388 - json
389
390 :param renderer_whitelist:
391 Allow renderers for pillar files.
392
393 .. code-block:: yaml
394
395 renderer_whitelist:
396 - yaml
397 - jinja
398
399 :param template:
400 Enable templating of pillar files. Defaults to ``False``.
401 """
402 # Not used
403 del pillar
404
405 if not root_dir:
406 log.error("file_tree: no root_dir specified")
407 return {}
408
409 if not os.path.isabs(root_dir):
410 pillarenv = __opts__["pillarenv"]
411 if pillarenv is None:
412 log.error("file_tree: root_dir is relative but pillarenv is not set")
413 return {}
414 log.debug("file_tree: pillarenv = %s", pillarenv)
415
416 env_roots = __opts__["pillar_roots"].get(pillarenv, None)
417 if env_roots is None:
418 log.error(
419 "file_tree: root_dir is relative but no pillar_roots are specified "
420 " for pillarenv %s",
421 pillarenv,
422 )
423 return {}
424
425 env_dirs = []
426 for env_root in env_roots:
427 env_dir = os.path.normpath(os.path.join(env_root, root_dir))
428 # don't redundantly load consecutively, but preserve any expected precedence
429 if env_dir not in env_dirs or env_dir != env_dirs[-1]:
430 env_dirs.append(env_dir)
431 dirs = env_dirs
432 else:
433 dirs = [root_dir]
434
435 result_pillar = {}
436 for root in dirs:
437 dir_pillar = _ext_pillar(
438 minion_id,
439 root,
440 follow_dir_links,
441 debug,
442 keep_newline,
443 render_default,
444 renderer_blacklist,
445 renderer_whitelist,
446 template,
447 )
448 result_pillar = salt.utils.dictupdate.merge(
449 result_pillar, dir_pillar, strategy="recurse"
450 )
451 return result_pillar
452
453
454 def _ext_pillar(
455 minion_id,
456 root_dir,
457 follow_dir_links,
458 debug,
459 keep_newline,
460 render_default,
461 renderer_blacklist,
462 renderer_whitelist,
463 template,
464 ):
465 """
466 Compile pillar data for a single root_dir for the specified minion ID
467 """
468 log.debug("file_tree: reading %s", root_dir)
469
470 if not os.path.isdir(root_dir):
471 log.error(
472 "file_tree: root_dir %s does not exist or is not a directory", root_dir
473 )
474 return {}
475
476 if not isinstance(keep_newline, (bool, list)):
477 log.error(
478 "file_tree: keep_newline must be either True/False or a list "
479 "of file globs. Skipping this ext_pillar for root_dir %s",
480 root_dir,
481 )
482 return {}
483
484 ngroup_pillar = {}
485 nodegroups_dir = os.path.join(root_dir, "nodegroups")
486 if os.path.exists(nodegroups_dir) and len(__opts__.get("nodegroups", ())) > 0:
487 master_ngroups = __opts__["nodegroups"]
488 ext_pillar_dirs = os.listdir(nodegroups_dir)
489 if len(ext_pillar_dirs) > 0:
490 for nodegroup in ext_pillar_dirs:
491 if os.path.isdir(nodegroups_dir) and nodegroup in master_ngroups:
492 ckminions = salt.utils.minions.CkMinions(__opts__)
493 _res = ckminions.check_minions(
494 master_ngroups[nodegroup], "compound"
495 )
496 match = _res["minions"]
497 if minion_id in match:
498 ngroup_dir = os.path.join(nodegroups_dir, str(nodegroup))
499 ngroup_pillar = salt.utils.dictupdate.merge(
500 ngroup_pillar,
501 _construct_pillar(
502 ngroup_dir,
503 follow_dir_links,
504 keep_newline,
505 render_default,
506 renderer_blacklist,
507 renderer_whitelist,
508 template,
509 ),
510 strategy="recurse",
511 )
512 else:
513 if debug is True:
514 log.debug(
515 "file_tree: no nodegroups found in file tree directory %s, skipping...",
516 ext_pillar_dirs,
517 )
518 else:
519 if debug is True:
520 log.debug("file_tree: no nodegroups found in master configuration")
521
522 host_dir = os.path.join(root_dir, "hosts", minion_id)
523 if not os.path.exists(host_dir):
524 if debug is True:
525 log.debug(
526 "file_tree: no pillar data for minion %s found in file tree directory %s",
527 minion_id,
528 host_dir,
529 )
530 return ngroup_pillar
531
532 if not os.path.isdir(host_dir):
533 log.error("file_tree: %s exists, but is not a directory", host_dir)
534 return ngroup_pillar
535
536 host_pillar = _construct_pillar(
537 host_dir,
538 follow_dir_links,
539 keep_newline,
540 render_default,
541 renderer_blacklist,
542 renderer_whitelist,
543 template,
544 )
545 return salt.utils.dictupdate.merge(ngroup_pillar, host_pillar, strategy="recurse")