vinegar.data_source

Sub modules

`sqlite`	Data source backed by an SQLite database.
`text_file`	Data source backed by a text file.
`yaml_target`	YAML-based source for configuration data, using pattern-based targeting.

Module API

Sources providing configuration information associated with each system.

Different sources can be used to fill retrieve configuration information for systems. A very flexible one is vinegar.data_source.yaml_target, which fills the configuration tree by parsing a central YAML file specifies target information (which files apply to which system) and the YAML files specified by this central file in order to retrieve the actual data.

Multiple data sources can easily be chained by using the get_composite_data_source function.

All source implementations have in common that they must specify a get_instance function that takes a dict with configuration data as its only parameter. This function must return an instance of data_source. The key name in that configuration dict is reserved for use by the calling code and should be ignored by the data source.

Data sources are thread safe.

class vinegar.data_source.DataSource

Source that provides configuration information for a system.

The information provided can (and typically will) be different for each targeted system.

This information is then passed to the templating mechanism when generating files requested by a client. This way, different files can be generated for each system.

Often, it can be useful to have more than one data source. In this case, the data returned by each of the sources should be merged with the data returned by the other sources. The get_composite_data_source function provides a convenient tool for such a setup.

If possible, data sources should preserve the key order in dictionaries.

Each data source has to implement the get_data method. This method is used to collect the data for a system with a known identifier. It also has to implement the find_system method. This method is used to find the system identifier using a key and an associated value.

It is perfectly legal for a data source to be able to provide data for a system, but not be able to find the system ID given the data. For example, a data source may provide the same data to a group of systems and in that case it it is impossible to identify a specific system using the data.

Data source have to be implemented in a thread-safe manner, so that get_data and find_system can safely be used by different threads.

abstract find_system(lookup_key: str, lookup_value: Any) → str | None

Find a system given the specified key and value.

If no system can be found, the data source returns None.

Parameters:

lookup_key – key for which to look. The interpretation of the key is up to the data source. Some data sources might use a flat structure, while others might support hierarchical data-structures. In the latter case, the use of the colon (:) as a hierarchy separator in the key is encouraged, but not required.
lookup_value – value for which to look. The interpreation of the value is up to the data source.

Returns:

system identifier or None if no system could be identified using the specified key and value.

abstract get_data(system_id: str, preceding_data: Mapping[Any, Any], preceding_data_version: str) → Tuple[Mapping[Any, Any], str]

Return data associated with the specified system.

If the data source does not have any information associated with the specified system ID, it should return an empty dictionary.

The return value of this method is in fact a tuple of the configuration data and a version string. The version string can be used by the calling code to decide whether the data has changed and thus caches have to be discarded. For example, the results of rendering a template might be cached and the cached version might be used as long as the version string returned by this method does not change. This means that implementations have to be careful to never return the same version string when the data for a system has changed. The vinegar.utils.version provides utility functions for generating version strings in a way that makes accidental collisions unlikely.

Please note that it is not the job of a data source to merge the preceding_data with the data provided by itself. The calling code takes care of this. Code wanting to use multiple data sources in a chain can use the get_composite_data_source function.

Implementations are encouraged to use caching to improve performance when this method is repeatedly called for the same systems.

Parameters:

system_id – ID of the system for which data is requested.
preceding_data – Data provided by the data source(s) that come earlier in the chain. This may be empty if there are no preceding data sources or if they did not provide any data for the system.
preceding_data_version – Version of the preceding_data. This is an arbitrary string (typically a hash) that can be used to detect when the data provided by the preceding sources has changed.

Returns:

tuple where the first element is the data associated with the specified system and the second element is a version string that changes whenver the returned data changes (for the same system).

class vinegar.data_source.DataSourceAware

Marker interface indicating that a component needs a DataSource.

An object can implement this interface in order to indicate to the creating code that it wants a data source to be injected into it through the set_data_source method.

This is useful when there are several different implementations of a component and some of them require a data source and others do not. The container creating the components can decouple the logic (and configuration) for creating them from the logic that injects the data source.

In general, a component that implements this interface should still try to provide as much of its functionality as reasonably possible, if no data source has been injected into it.

Code wanting to inject a data source into a component that might possibly need it can use the inject_data_source helper function.

set_data_source(data_source: DataSource) → None

Set the data source to be used by this component.

Calling inject_data_source might be preferable to calling this method directly.

In general, code using this method should not assume that this method is thread safe, even if the object that implements it is considered safe in general. This means that this method should typically be called only once, directly after creating an object.

Parameters:: data_source – data source to be injected.

vinegar.data_source.get_composite_data_source(data_sources: Sequence[Tuple[str, Mapping[Any, Any]] | DataSource], merge_lists: bool = False, merge_sets: bool = True) → DataSource

Return a data source that is a composite of the specified data sources.

The returned data source takes the provided initial data and passes it to the first data source in the chain. It then takes the result of that data source, merges it with the initial data, and passes the merged data to the next data source in the chain. This process goes on until the last data source has been reached. At this point, the result of the last data source is merged with the result of the previous data source and returned.

The data source returned by this function internally uses merge_data_trees to merge the data returned by a source with the preceding data. The merge_lists option is passed on to that function and defines whether lists are also merged. By default, only dictionaries are merged.

Parameters:

data_sources – sequence of data sources that are chained together. Each item in the sequence can either be an instance of DataSource or a tuple. If it is a tuple, the first element must be the name of the data source (as passed to get_data_source) and the second one must be the corresponding configuration.
merge_lists – defines whether lists are merged when merging data or whether a list in one dictionary replaces the list in the other dictionary. Please refer to the documentation for merge_data_trees for details.
merge_sets – defines whether sets are merged when merging data or whether a set in one dictionary replaces the set in the other dictionary. Please refer to the documentation for merge_data_trees for details.

Returns:

composite data source that chains the specified data sources together.

vinegar.data_source.get_data_source(name: str, config: Mapping[Any, Any]) → DataSource

Create the an instance of the data source with the specified name, using the specified configuration.

Parameters:: name – name of the data source. If the name contains a dot, it is treated as an absolute module name. Otherwise it is treated as a name of one of the modules inside the vinegar.data_source module.
Param:: config: configuration data for the data source. The meaning of that data is up to the implementation of the data source.
Returns:: newly created data source.

vinegar.data_source.inject_data_source(obj: Any, data_source: DataSource) → None

Inject a data source into an object.

This data source is only injected if obj is an instance of DataSourceAware, so it is safe to call this function for any object.

In general, code using this function should not assume that it is thread safe, even if the object that is the target of the injection is considered thread safe in general. This means that this function should typically be called only once, directly after creating an object.

Parameters:

obj – object that might or might not be an instance of DataSourceAware.
data_source – data source to be injected. It is only injected if obj is DataSourceAware.

vinegar.data_source.merge_data_trees(tree1: Mapping[Any, Any], tree2: Mapping[Any, Any], merge_lists: bool = False, merge_sets: bool = True) → Mapping[Any, Any]

Merge two mappings, returning the resulting dictionary.

In general, the resulting dictionary is formed by taking the key-value pairs from both mappings and putting them into a single dictionary. If the same key is present in both dictionaries, the value from the second dictionary takes precedence.

If the value is itself a mapping, the merge process is applied recursively.

If the value is a sequence, the process depends on the merge_lists option. If it is set to True, the resulting list is created by first adding all elements from the first sequence and then appending all elements from the second sequence, except for those elements that were already present in the first sequence. If merge_lists is set to False, the second sequence simply replaces the first one (like for non-sequence types).

If the value is a set, the process depends on the merge_sets option. If it is set to True (the default), the resulting set is created by calculating the union of both sets (set1 | set2). If it is set to False, the second set simply replaces the first one (like for non-set types).

In this context, the str, bytes, bytearray, and memoryview types are not treated as sequences. Values of this type always replace each other and are not merged.

If the value associated with a key is a mapping in one mapping, but not in the other one, an exception is thrown. The same applies when merge_lists is True and the value associated with a key is a sequence in one mapping, but not in the other one.

The resulting dictionary preserves key order. This means that it first contains all keys from the first mapping and then those keys from the second mapping that were not also present in the first mapping.

Parameters:

tree1 – mapping that shall be used as a base for the merge process.
tree2 – mapping that is merged into the data from tree1, taking precedence in case of key collisions.
merge_lists – True if sequences in the mappings shall be merged, too, False if they shall replace each other. The default is False.
merge_sets – True if sets in the mappings shall be merged, too, False if they shall replace each other. The default is True.

Returns:

insertion-order preserving dictionary that contains the merged data from tree1 and tree2.