Deploy the code directly to the docker container. Or how not to prokrasinirovat after each commit

Task WEB-12982 arrived
Create a branch web-12982 in the repository
While the branch is going, you read TZ and drink coffee
You start directly to develop

git commit, git push
While the branch is being re-compiled
git commit, git push
While the branch is being re-compiled you are flipping through Twitter
git commit, git push
...
Surrender a 50-commit branch to the review

You understand that 50 commits are exactly 50 minutes of pure time, which is collected in fragments, because the segments of 1 minute are too small to do anything other than procrastination and basic needs.

Familiar with the situation? In my company, the development infrastructure is organized in the following way:

• There are many project repositories in Hitlab.
• In order to provide development convenience when creating a new branch, dockers automatically create their own sandbox at a unique address, a complete copy of the parent branch with all the necessary environments.
• Everything you need is ready - just write the code and test-see-evaluate the result after each commit, it is very convenient!

But, slowly ... If this situation is close to you, welcome under cat.

• The essence of the problem
• Solution options
• Solution implementation

The essence of the problem

tldr: The edits made to the code require container reassembly and spend time (more than a minute, strongly depends on the project, especially if CI \ CD is configured), which in fact cannot be spent with benefit, but which totally takes a decent piece of the programmer’s work time.

I have previously published an article on a related topic , but related to the debugging process (By the way, thanks for the informative comments and positive feedback). However, the problem with the cut has not really disappeared anywhere, we also continue to wait until the branch is rebuilt.

Even if we skip the additional stages and leave only build-dev and deploy-dev, the waiting time is insignificant for any useful actions, but significantly in the total time spent, especially when it comes to CI \ CD, for example from Gitlab.

Of course, you can speed up the process by collecting a project locally, provided that the programmer has a relatively powerful computer, gitlab-runner is configured, the same environment is configured as on the remote server, and the corresponding tags are added to gitlab-ci.yml, but I doubt that the build speed will be the same as the automatic deployment of the code via FTP after the ctrl + s keys.

Special burns furious exhausting when in the development process you make typos / errors that generally do not affect the operation of the application, but which can not be left just so you notice them only when you look at the result after the assembly.

What is required and solutions

tldr: Find a way to see the result of edits as quickly and easily as possible. In order not to commit every time and not wait for rebuilding the branch. I selected rsync from a local computer to a remote server folder mounted with a container folder.

I did not find a completely ready solution on the Internet, but in fact there are several options:

• Configure ftp \ ssh directly into the container with the code base, include it in the image, connect via FTP directly to the container itself
  
  • Personally, this option seemed to me somewhat complicated, and too “crutchy”, although here all the options are crutches.
• (for local use) use docker cp to load the code directly into the container
  
  • The option is not suitable for working with a remote server.
  • Docker cp has extremely limited functionality, given the vendor folders that should not be copied each time, and the copy algorithm itself, it will be quite slow.
• (for remote \ local use) Mount the required folder of the container to the external folder of the host. Download local files directly to the mounted folder of the host. There are already a lot of implementation options:
  
  • Use docker-machine and specifically docker-machine scp
    
    • Again, you need to configure the environment, configure the docker-machine, maybe it makes sense when you constantly work with different servers
  • In the IDE, configure an FTP connection with the desired host folder
    
    • Each new branch requires creating a new connection or changing the mapping.
  • Use scp or rsync to upload files to the desired host folder, for this, use a small bash-script and hang it on hotkeys
    
    • At first it seems that it is unnecessarily difficult, but in fact it is not. The script itself is as simple as possible and is needed to automate the process so that you do not have to reconfigure the mappings every time.

The solution itself: rsync + volumes

tldr:

• Need ssh access to remote server and rsync on local machine
• The remote server must have a write-free folder.
• Mount the project folder in the container to the external host folder
• Add a small bash script to the project root to synchronize files with a remote server
• Configuring the hotkey for synchronization script execution

It is worth noting that the solution provided is excluded for the development and test environment

In this case, I’ll look at the docker-compose and gitlab-ci environment, while docker-compose uses environment variables from gitlab-ci.

Create the path to the final folder in gitlab-ci and export this path to docker-compose.yml:

before_script: - export SHARED_DIR_BASE='/var/www/builds' # папка на удаленном сервере, в которой будем хранить проекты - export SHARED_BRANCH_DIR=${SHARED_DIR_BASE}/${PROJECT_GROUP}/${PROJECT_NAME}/${CI_COMMIT_REF_NAME} # допустим мы создаем ветку web-123 в проекте my_group/my_project, конечный путь для записи будет /var/shared/my_group/my_project/web-123 Deploy dev: stage: deploy_dev script: # непосредственно создаем папку, переносим туда проект и настраиваем права на запись - mkdir -p ${SHARED_BRANCH_DIR} - rsync -r --exclude-from=.gitignore --exclude-from=.dockerignore . ${SHARED_BRANCH_DIR} - find ${SHARED_BRANCH_DIR} -type d -exec setfacl -d -mo:rwx {} \; - find ${SHARED_BRANCH_DIR} -type d -exec setfacl -mo:rwx {} \; - find ${SHARED_BRANCH_DIR} -type f -exec setfacl -mo:rwx {} \; - envsubst < docker-compose.tmpl > docker-compose.yml # Переносим переменные окружения из gitlab-ci.yml в docker-compose.yml, шаблоном является docker-compose.tmpl - docker-compose up -d 

Next, we need to mount the project folders to the external host folder in docker-compose, since we use variables in docker-compose, we need the docker-compose.tmpl template in which we will use these variables.

 version: '2.3' services: web: ... volumes: - ${SHARED_BRANCH_DIR}:/app/:rw # Непосредственная привязка внешней папки хоста к внутренней папке контейнера с самим приложением # Строки ниже уже индивидуальны в зависимости от проекта. Общая суть в том, что нам нужно смонтировать общую папку, но при этом НЕ МОНТИРОВАТЬ вендорные или динамические файлы и папки. Иначе при загрузке файлов из локальной машины в смонтированную нужно переносить и эти файлы, которые как правило довольно громоздкие, либо они могут быть удалены при синхронизации - /app/protected/vendor/ 

The current configuration is enough, now when building a branch on the host server, the folder / var / www / builds / GROUP_NAME / PROJECT_NAME / NAME_WINKLE will be created and the project itself is transferred there except for those files and folders specified in .gitignore and .dockerignore, then you can simply set up FTP-mappings, but we will go a little further and make the process a bit more automated.

In fact, in order to synchronize files, we need to run something like this:

 rsync -r -u \ --delete-after \ --exclude-from=.gitignore \ --exclude-from=.dockerignore \ . $sshUserName@$sshHost:$sharedBaseDir 

In fact, on small-medium projects, this command will be completed faster than you can commit and push edits to the repository. It remains to bring this script to a more complete form and bind its execution to hotkeys.

 #!/usr/bin/env bash # Присваиваем переданные данные while [ -n "$1" ] do case "$1" in --sshUserName=*) sshUserName=${1#*=} ;; --sshHost=*) sshHost=${1#*=} ;; esac shift done # Определяем shared папку для ветки gitBranch=$(git branch | grep \* | cut -d ' ' -f2) gitProject=$(git config --local remote.origin.url|sed -n 's#.*/\([^.]*\)\.git#\1#p') gitGroup=$(git config --local remote.origin.url|sed -n 's#.*/\([^.]*\)/.*\.git#\1#p') sharedBaseDir=/var/www/builds/$gitGroup/$gitProject/$gitBranch # Синхронизируем папку проекта с папкой на удаленном хосте rsync -r -u \ --delete-after \ --exclude-from=.gitignore \ --exclude-from=.dockerignore \ . $sshUserName@$sshHost:$sharedBaseDir echo "done" 

It should be noted that the script should be located in the root of the project folder of the repository or indicate in which directory it should work.

It remains only to tie the execution of this script to specific hotkeys and set the sshUserName and sshHost parameters (it is assumed that you already have access to the remote server via ssh). How to do this, I will give an example of PHPstorm.

• Go to File -> Settings
• In the settings window in the left menu, expand Tools , select External Tools
• Register the path to the script and specify the real sshUserName and sshHost in the arguments
  
  
• Next, go to Keymap and look for the name of our External Tools, install the necessary combination
  
  

That's all. Now when you press the desired combination, all project files are synchronized with the remote folder that is mounted with the project folder inside the container. That is, any changes will be visible almost immediately.

I do not pretend to the "ideality" of this decision, for sure there are better options, I will be glad if I find out about them in the comments. Thank!