git and configuration files

Unfortunately, that's it. The version control system should not include configuration files that will not run on other hosts, therefore, as a rule, the software is distributed without real configs, and the initial configuration command that can generate this file is added to console commands.

Adding required configuration options on the go is a bad idea, at least between major versions. They should have their default value at which the application continues to behave the same as before. All these -webkit-something-tralala in CSS appeared exactly from the same place - let's add this thing, but let's not shove it as a ready-made option when we are ready for implementation - we will introduce it so that it will not be renamed and not changed later (specifically in CSS the names are given by the standard, but the general idea should be clear).

However, I found one hack for myself - the entire dev-env is thrust into the vagranate, where you can freely write any configs and change them on the go, as a result, you can play freely in the development team with a test configuration.

Another thing that helps is “parallel” files in which values ​​are redefined: configuration.yml contains some configuration, and configuration.local.yml contains only a couple of options that “take over” over similar options from configuration.yml .

The authors of the manifesto of 12 factors spoke on this topic.

Here is what they write about the configuration :

The litmus test of whether the configuration and application code are properly separated is the fact that the application code base can be freely available at any time without compromising any private data.

Further:

Another approach to configuration is to use configuration files that are not saved to the version control system, for example, config/database.yml in Rails. This is a huge improvement before using constants that are saved in the code, but there are still some disadvantages to this method: it is easy to save the configuration file by mistake to the repository; There is a tendency when configuration files are scattered in different places and in different formats, because of this it becomes difficult to view and manage all settings in one place.

This is the way described in the question. The authors of the manifesto offer this solution:

An appendix of twelve factors stores configuration in environment variables (often abbreviated to env vars or env). Environment variables are easy to change between deployments without changing the code; unlike configuration files, it is less likely to accidentally save them to the code repository; and unlike user configuration files or other configuration mechanisms, such as Java System Properties, they are a language and operating system independent standard.

Naturally, all this applies mainly to web applications. For desktop and mobile applications, these rules are no longer applicable.

From myself I will add that Azure-applications ASP.NET and ASP.NET WebAPI are now configured exactly this way: in the application panel on the Application settings tab, you can specify environment variables. Same in Heroku.

Example

In our project is required to send emails and SMS. Naturally, on the local contours of the developers and on the general contour of the development, you do not need to send anything, but you also need access to the content of the notifications. That is, developers should see that the notification service has worked, and see what exactly will be sent to the client on the combat circuit.

Since we use dependency injection , we have made several implementations of the classes that send notifications. The product implementation sends the letters, and the developers implementation writes notifications to the file. As an IoC library, we use Autofac, which allows us to register dependencies in a configuration file, so a notification service for developers was registered in our Web.config .

MSBuild can transform ASP.NET project configuration files during deployment. If your folder contains the Web.config and Web.Release.config , when you deploy the project in the Release configuration, MSBuild will apply the transformations from Web.Release.config to Web.config . You can change the attributes of sections, add and delete subsections. We changed the registered class, so that on the Release circuit we started the real notification service instead of the debugging one.

Then we are satisfied with this decision.

For a while, everything worked well, but then the 4th version of Autofac came out, which became compatible with the new .NET configuration system. At the same time, Autofac developers have cut support for the old method, that is, good old Web.config and App.config . However, MSBuild cannot automatically transform new configuration files.

I had to redo the scheme. Now for each contour we began to store our version of the IoC configuration in the files IoC.Dev.json , IoC.Stage.json , IoC.Release.json . Loading the desired configuration file was carried out as follows:

Startup.cs

 var configName = Environment.GetEnvironmentVariable("APPSETTING_CONFIG_NAME") ?? "Dev"; var config = new ConfigurationBuilder().AddJsonFile($"IoC.{configName}.json", optional: true, reloadOnChange: true); var module = new ConfigurationModule(config.Build()); builder.RegisterModule(module); 

It was a spontaneous solution that we found in Google and were able to apply to our Azure-project. However, this is not ideal in terms of a 12-factor application. Part of the configuration is indeed rendered into the environment variable, but part is in the IoC.*.json files.

What's wrong? The system administrator can decide that the files IoC.Dev.json , IoC.Stage.json , IoC.Release.json can be easily changed, although in fact we are rather severe about them. We do not need versatility and flexibility here, we would like to limit the setting to two options: a) send notifications; b) we put the notice in a secret place.

So we can hard-code these two strategies and at the start of the application choose the one that is specified in the environment variable:

 var notifyStrategy = Environment.GetEnvironmentVariable("APPSETTING_NOTIFY_STRATEGY"); switch (notifyStrategy) { case "send": builder.RegisterType<ReadNotifier>().As<INotifier>(); break; case "save": builder.RegisterType<FakeNotifier>().As<INotifier>(); break; default: throw new ArgumentException("Ну всё теперь.", nameof(notifyStrategy)); } 

Now the application administrator can configure it at his dive level. It will not break anything important in the wilds of XML / JSON IoC. As a result, we managed to completely get rid of the configuration files at this level and get closer to the ideals of 12-factor applications.

Based on this, I would advise:

1. All secret settings, including database connection strings, logins and passwords for sending emails, etc., are taken from the environment variables. Deploying an application is reduced to running a single command (for .NET it is MSBuild). Environment variables and the deployment process are described in the README.md .

2. Implement dependency injection directly in code, providing several strategies that the administrator will manage. He will thank you if he doesn’t have to study the details of the application, and he can, with one setting, implement a completely different set of three to five to ten types without understanding their interrelationships.

3. Some configuration files are in fact the declarative part of the code and do not need to be modified after deployment. Details depend on the language, I think that it is more often found in interpreted languages. We should not consider such files as true configuration files and can leave them in the project as it is.

4. The configuration that the administrator can change and which can be saved between deployments can be transferred to the database. It is possible and in the file, but in this case, it can be destroyed with inaccurate deployment. The values ​​in the database or in the file, if they are not there when the system is first started, the standard ones from the code are written.

I try to do as follows. First, the program tries to use the configuration specialized for the current host from the config-hostname.xml file. All sensitive information is stored in it, and it does not get into git due to ignoring by the mask config-*.xml . If there is no specialized configuration, the default one is used from the config.xml file, which is stored in the repository. According to the history of changes to this file, it is very convenient to track when certain features appeared. Often it is more convenient than to search by changelog.

In those cases where it is impossible to come up with a meaningful default value, but at the same time I don’t want to remove it from the default configuration at all (it should be clear that it exists), I prescribe some deliberately invalid special value ( -1 or the string "must be customized"). The program, when working with such a parameter, understands this value, issues an appropriate diagnostics and exits.

I just keep sample versions of config files. Purely in order to read an example of how it might look. These configs are collected using ansible.

It is he who substitutes critical information such as passwords.

The benefits are obvious: a centralized description and storage of configurations, all files are marked.

In some cases, to simplify life, I use the following trick: the dev configuration passwords are md5 from production passwords.

Something like this:

 [mysql] host = "{{ databases_mysql[0].dbhost }}" dbname = "{{ databases_mysql[0].dbname }}" user = "{{ db_users_mysql[0].name }}" password = "{{ mask_pwd | ternary(db_users_mysql[0].password|hash("md5"),db_users_mysql[0].password) }}" 

where mask_pwd is a Boolean variable, whether or not to mask the password (set depending on the type of environment).

Everything described below refers to the spring application, the first paragraph can be attributed to other technologies, if it is possible to substitute data from environment variables .

• There are many ways to configure settings . But there really are certain sensitive data (passwords, credentials, etc.) that cannot be stored in the public domain. In this case, you can use the option of storing the skeleton settings in the property files in git `s, but with substitutions from enviroment variables. In this case, sensitive data will be recorded in the OC.
• The first item has one major drawback - the synchronization of such data between different servers. This situation may occur with microservice and SOA architectures (one or more applications use the same credentials). In this case, there is spring cloud vault , which allows you to restrict access to such data through application authorization. The official description of the project.

  Spring Cloud Vault Config. For all types of applications, across the environments. Credentials for external services such as MySQL, PostgreSQL, Apache Cassandra, MongoDB, Consul, AWS and more.

As a way, probably, to solve this problem using continuous integration systems that you yourself will create a configuration with an environment variable.
The environment variable is set in the continuous integration system itself.
TeamCity may, in other CI I think there is such an opportunity. Also there is the ability to add in addition to variable environments and more parameters, system properties which is very convenient.
To all this, the convenience of writing scripts assembly.
An example of generating a configuration using the js file example

 file=$(cat <<EOF export const SERVER_URL = '%dev.server_url%'; export const PREFIX = '%dev.api_prefix%'; export const API_URL = '%dev.api_url%'; export const imgPath = '%dev.server_url%'; export const DefaultLanguage = '%dev.default_lang%'; export const GoogleMapsKey = '%dev.google_maps_key%'; export const ConfigAuth = { vkontakte: {client_id: '%dev.social.auth.vkontakte%'}, facebook: {client_id: '%dev.social.auth.facebook%'}, google: {client_id: '%dev.social.auth.google_client_id%', apiKey: '%dev.social.auth.google_api_key%' } } EOF ) 

%dev.server_url% - variable environment as a template, TeamCity parses this line with %property.name%

The result will be output already in the created configuration file export const SERVER_URL = 'http://localhost:8000/';

And in addition to all this, the configuration example in the repository

I store the default configuration in the .env file in the project root.

 ENV_database_host=127.0.0.1 

This file is the same for all developers, it gets into the git repository. The file is loaded when docker-compose up -d started and values ​​get into the container through the environment block in the docker service description

 services: php: environment: ENV_database_host: "${ENV_database_host}" 

Then this value is loaded into the parameters of the application on symfony in the parameters.yml file

 parameters: database_host: "%env(ENV_database_host)%" 

And already the database_host parameter is used in the DI services of the symfony container "as usual"

On the remote server, the values ​​from the .env file must be overlapped by others. To do this, in the Pipeline CI\CD settings in the GitLab repository, variables are created for each of the environments with its own suffix. For example, ENV_database_host_MASTER for staging and ENV_database_host_PRODUCTION for production.

So that the correct configuration gets to the remote server, the values ​​of the variables ⁠-⁠ with ⁠-⁠ suffix are transferred to the variables without the suffix, the file docker-compose.yml compiled with the values ​​already overlapped and the result is copied to the server

 docker-compose -f docker-compose-deploy.yml config > build/docker-compose.yml