vinegar.data_source.yaml_target

YAML-based source for configuration data, using pattern-based targeting.

This data source uses a flexible and powerful targeting mechanism. It works on a directory tree where the file top.yaml in the root of the tree defines targeting rules that specify which systems receive which configuration data.

Due to this flexible architecture, this data source cannot identify a system given a key and an associated value, so its find_system method will always return None.

File syntax

For example, the file top.yaml might look like this:

# Files applied to all systems:
'*':
    - common.file1
    - common.file2

# Files applied to systems that have an ID starting with "mysys-"
'mysys-*':
    - example

# Files applied to all systems that have an ID that ends with
# ".a.example.com" or ".b.example.com".
'*.a.example.com or *.b.example.com':
    - other.example

In this example, a system named mysys-x.b.example.com would receive data from the following files in the directory tree:

  • common/file1.yaml

  • common/file2.yaml

  • example.yaml

  • other/example.yaml

A system with the ID my.example.com, on the other hand, would only receive data from the following files:

  • common/file1.yaml

  • common/file2.yaml

In order to make the file structure cleaner, instead of creating a file in a directory, one can create a sub-directory with a file called init.yaml.

For example, the reference to example would be resolved to example/init.yaml if example.yaml does not exist.

Keys in top.yaml patterns matching system IDs or data from preceding data sources, but they can also be combinations of several such patterns using logical expressions. Please refer to the documentation of the vinegar.utils.system_matcher for details. The dict passed to the matcher is a SmartLookupDict, so nested keys may be used in matching expressions.

Each data file (e.g. common/file1.yaml in the example above) is a simple YAML file that provides configuration data.

Such a file might look like this:

boot_files:
  kernel: vmlinuz-4.4.0-148-generic
  initrd: initrd.img-4.4.0-148-generic

A data file can include other data files by listing the under the include key:

include:
  - some.otherfile
  - example.more

This has the same effect as if the content of that file was pasted at the position of the include, with one difference: Duplicate keys will not cause a parsing errors. Instead, they are going to be merged (see below).

Included files can also be specified in a relative fashion. The following example includes a file called other.yaml that is in the same directory as the file where the include is specified:

include:
  - .other

This also works for files in parent directories (and parent directories of parent directories, etc.):

include:
  - ..some
  - ...file

This example includes the file some.yaml in the parent directory of the directoy where the current file is located and file.yaml in the parent directory of that directory.

Merging multiple data files

More than one data file can apply to a single system through several ways:

  • A key in the top.yaml file can list more than one file.

  • A system ID might match multiple keys (patterns) in top.yaml.

  • A data file that is included through top.yaml might itself include other data files.

In all these cases, the data provided by the different files is merged. When merging data, values from files that are listed later, take precedence over files that are listed earlier.

When using include: in a data file, data from the included files overrides data that precedes the include: block, but not data that follows the block.

Dictionaries that are part of the data tree are merged. By default, lists are not merged but replaced, however this can be changed through the merge_lists configuration option.

Please note that this data source does never merge the data passed to its get_data method (through the preceding_data argument) into the resulting data. If this is desired, a composite data source (see vinegar.data_source.get_composite_data_source) should be used.

Using Templating

The YAML files that are used by this data source can contain template code. By default, the jinja template engine is used. Please refer to the documentation of that engine for details about the syntax.

Another template engine can be selected through the template configuration option, or templating can be disabled completely by setting that option to None.

The data source provides two context objects to the template engine: The id object contains the system ID (as a str) and the data objects contains the data that has been passed to the get_data method as preceding_data. The data object is passed as a SmartLookupDict to make it easier to get nested values.

Configuration options

This data source has several configuration options that can be used to control its behavior. Of all these options, only the root_dir option must be specified. All other options have default values.

root_dir:

path to the directory that contains top.yaml (as a str). All other files are also resolved relative to this directory.

allow_empty_top:

If set to True having a top.yaml file that is empty does not result in an exception being raised. This can be useful when templating code is used to selectively generate content in top.yaml. The default is False which means that an exception is raised if top.yaml does not contain at least one key-value pair.

cache_size:

Maximum number of data trees that are cached. This data source uses an LRUCache so that the process of compiling the data for a specific system does not have to be repeated for every call to get_data. By default, this cache stores up to 64 entries. If set to zero, the cache is disabled completely. Please note that this will not disable the cache of the template engine that is used. Please refer to the documentation for the template engine in use to see whether it uses a cache and how it can be disabled.

file_suffix:

The file name suffix that is used when constructing the names of the YAML files (e.g. top.yaml). The default is .yaml. Changing this to something else can be useful when using a template engine. In this case, using a file extension specific to the template engine (e.g. .yaml.jinja) might help editors to use the correct kind of syntax highlighting for the files.

merge_lists:

If True, lists are merged when merging data from different data files. If False (the default), lists are not merged, but replaced. Please refer to the documentation for merge_data_trees for details about the effects of this option.

merge_sets:

If True (the default), sets are merged when merging data from different data files. If False, sets are not merged, but replaced. Please refer to the documentation for merge_data_trees for details about the effects of this option.

template:

name of the template engine (as a str) that shall be used for rending the top.yaml and the data files. The default is jinja. This name is passed to get_template_engine in order to retrieve the template engine. If set to None templating is disabled.

template_config:

configuration for the template engine. The default is an empty dictionary ({}). This configuration is passed on to the template engine as is.

class vinegar.data_source.yaml_target.YamlTargetSource(config: Mapping[Any, Any])

Data source that constructs a configuration tree through a flexible targeting mechanism.

For information about the configuration options supported by this data source, please refer to the module documentation.

find_system(lookup_key: str, lookup_value: Any) → str | None

Find a system given the specified key and value.

If no system can be found, the data source returns None.

Parameters:

  • lookup_key – key for which to look. The interpretation of the key is up to the data source. Some data sources might use a flat structure, while others might support hierarchical data-structures. In the latter case, the use of the colon (:) as a hierarchy separator in the key is encouraged, but not required.

  • lookup_value – value for which to look. The interpreation of the value is up to the data source.

Returns:

system identifier or None if no system could be identified using the specified key and value.

get_data(system_id: str, preceding_data: Mapping[Any, Any], preceding_data_version: str) → Tuple[Mapping[Any, Any], str]

Return data associated with the specified system.

If the data source does not have any information associated with the specified system ID, it should return an empty dictionary.

The return value of this method is in fact a tuple of the configuration data and a version string. The version string can be used by the calling code to decide whether the data has changed and thus caches have to be discarded. For example, the results of rendering a template might be cached and the cached version might be used as long as the version string returned by this method does not change. This means that implementations have to be careful to never return the same version string when the data for a system has changed. The vinegar.utils.version provides utility functions for generating version strings in a way that makes accidental collisions unlikely.

Please note that it is not the job of a data source to merge the preceding_data with the data provided by itself. The calling code takes care of this. Code wanting to use multiple data sources in a chain can use the get_composite_data_source function.

Implementations are encouraged to use caching to improve performance when this method is repeatedly called for the same systems.

Parameters:

  • system_id – ID of the system for which data is requested.

  • preceding_data – Data provided by the data source(s) that come earlier in the chain. This may be empty if there are no preceding data sources or if they did not provide any data for the system.

  • preceding_data_version – Version of the preceding_data. This is an arbitrary string (typically a hash) that can be used to detect when the data provided by the preceding sources has changed.

Returns:

tuple where the first element is the data associated with the specified system and the second element is a version string that changes whenver the returned data changes (for the same system).

vinegar.data_source.yaml_target.get_instance(config: Mapping[Any, Any]) → YamlTargetSource

Create a YAML data source supporting targeting.

For information about the configuration options supported by that source, please refer to the module documentation.

Parameters:

config – configuration for the data source.

Returns:

YAML data source using the specified configuration.