Hosting Node.js https server with auto-updated SSL in the cloud and how I set up the development cycle (+ git, react)

Foreword

To begin with, once I wanted to create an application. This desire arose from the fact that I love to read, and there are simply no normal book aggregators in the vast Russian Internet. Actually, from the pain of searching for something to read and trying to recall the name of the book that I read recently and on which chapter I stopped, the desire was born to make a web application in which all this would be possible and convenient. It should be noted that no experience in development, programming, etc. I did not have, my work is not connected with it at all. Nevertheless, desire overcame laziness and turned into concrete actions, a kind of hobby.

I will not tell you how I studied javascript, node.js, react, html, css, etc., let's go to what I have come to at the moment, what I would like to share with you and, of course, listen to constructive criticism of experts.

Like many, I trained on my own PC on localhost: 3000, created front / back-ends, installed, worked with api, etc., but I was always worried about how to transfer all this to hosting? Will it work? Do I need to rewrite the code because of this? And most importantly, is it possible to set everything up so that I can work on the application from any PC and easily transfer everything to hosting on production? I'll tell you about it.

Hosting selection

I was ready to spend $ 10 a month for my hobby, so I chose the hosting I planned to stay with in the future. As I said, before that I had 0 experience, including hosting sites. I tried and refused the following:

Jelastic : beautiful and user-friendly interface, everything seems to be intuitive, scalable and understandable. Nevertheless, I encountered difficulties in setting up (for some reason, nginx from vps did not want to work, only with their own module) and connecting SSL (and automatic updating) to the Russian-language domain using standard tools (they promised to fix it, but I don’t want to wait)

REG.RU cloud hosting : I have the domain right there, so the solution seemed logical, but they didn’t have a separately configured PostgreSQL, and since I didn’t want to get involved with the administration of the database, I began to search further.

AWS and Google clouds : I tried, everything seems to be fine, but I remembered our "wonderful" laws and the requirement to stir user data on servers in the Russian Federation. Unfortunately, these guys did not have servers in the Russian Federation. Not a lawyer, but from sin I decided to look for clouds with servers in the Russian Federation. If your application is unlikely to have problems with the law, then a good choice.

Although there were clouds with servers in the Russian Federation, I still wanted something that would save me from having to dive into the administration of PostgreSQL. A gust stumbled a bit on Yandex.Oblaka , which had recently become available, tried it, everything seemed simple and convenient, so I stopped on them for now. It should be noted that PostgreSQL hosting at them immediately comes from 1core and 4GB of RAM, which is about 2k rubles per month, so for the development time and low load I plan to run PostgreSQL on VPS for ~ 300r, and with increasing load, transfer the base and let Yandex administers and updates.

Configuring Yandex. Cloud

Virtual Private Cloud

1) Create a directory for your site:

2) Create a Virtual Private Cloud:

The main thing that he gives for me at the current stage is the IP for accessing the created resource from the outside. I got acquainted with subnets, zones, isolation and fault tolerance superficially, if necessary, I will make it up.

3) Create a subnet and assign it an internal IP (as I understand it, this is about how a local network)

4) Go to the IP tab and reserve yourself a static IP.

According to it, we will connect from home and other places. It is probably possible to work with dynamic, but I did not understand when it changes.

Compute cloud

Here we will have calculations :) That is, we will create a Linux virtual machine (I chose ubuntu 18.04), install the node.js application and postgreSQL.

We press to create a VM, we unscrew all the settings to a minimum, since there will be no load during development (when our application comes out, then we will tighten up more, well, we will monitor them on schedules).

Ssh

The problem point I encountered at this stage is SSH:

What it is and why I had no idea, so I went to study. It turned out - this is just an access method, but not by the password, but by the generated SSH key. To actually generate it, download and install Putty as we are advised.

Run C: \ Program Files \ PuTTY \ puttygen.exe

Press the Generate button and move the mouse to give randomness to the generated key (as I understood it). Next, copy the appeared line starting with ssh-rsa somewhere in a text file and click Save private key, Save public key. The key copied to a text file is inserted into the SSH key of the Yandex open page. Login is root, otherwise you will not have access when working with the graphic file system of the application via which you will connect to the cloud from home / work (perhaps there is a way, but I did not understand).
As noted by andreymal, it is better not to use root, so that Chinese bots do not pick up the password to your cloud, but since Yandex.Cloud access is only via SSH, you can live like.

The application on the hosting is worth running exclusively non-root user, so as not to allow attackers to run malicious code through possible vulnerabilities in your application.

Connect to the cloud from a PC and choose a free SSH client

Standard Putty allows you to work only with the command line, and since my windows user is unusual, I started looking for a client with a pseudo-conductor. At first, I tried Mobaxterm, but after some idle time it turns off, the explorer hangs altogether, so now I work with bitvise ssh and I don’t observe any problems like Mobaxterm.

Setting up bitvise ssh

Here in the Server> Host field we indicate our external IP clouds. Port 22. Click the client key manager> import and specify in it the generated Putty previously private key. You may still need a key phrase, choose something that you will not forget. We close this window and in the authentication field we specify username: root, method publick key, client key - select the one imported earlier. We press login and if we did everything correctly, then connect to the cloud:

Install Node.js

Here I recommend using digitalocean.com instructions, they are very detailed and many are in Russian. I usually google “digitalocean ubuntu 18.04 node.js” or whatever you want to install or configure.

How to install Node.js can be read here .

If it is short, then we come on nodesource (here the latest versions of node.js can be installed), we look through here:

Copy and run the following commands in turn:

curl -sL https://deb.nodesource.com/setup_11.x | sudo -E bash - sudo apt-get install -y nodejs

Checking how the team installed

 nodejs -v

Show us the version of node.js

 npm -v

It will show us the version of the package manager for node.js.

Next, go to the folder / opt / mysuperapp (my_super_app_name - this folder you must create). The opt directory was chosen as the location of the application after long googling “where it’s appropriate to put the application's node.js files in ubuntu”.

Finally, we create the server.js file, which will be the entry point of the application, and paste the code of the simple server on node.js there:

 const http = require('http'); const hostname = 'localhost'; const port = 80; const server = http.createServer((req, res) => { res.statusCode = 200; res.setHeader('Content-Type', 'text/plain'); res.end('Hello World!\n'); }); server.listen(port, hostname, () => { console.log(`Server running at http://${hostname}:${port}/`); });

Port 80 is for http requests, 443 is for https. While we have a server on http.

Save everything and run the command:

 node server.js

The line 'Server running at localhost : 80 /' should appear in the console

Now you can open the browser, enter the external IP (the one in your Yandex cloud with your ubuntu VM) and we will see “Hello World!”

We do everything conveniently or the development cycle with git

Everything seems to work, but we will not work all the time connecting to the cloud. In addition, all of a sudden we will work in the future not alone.

Github

Github is the place where our application code will be located. In short, the principle of work for one person is as follows:

On the home PC we develop our application.
We save and unload the code on Github in one click.
On a hosting or on another PC, download our application from github, reboot the server (if it is hosting) and the new version of our web application is available on the world wide web.

Everything is fast, simple and convenient.

Actually register on Github and create a private repository for our application (it will be available only to us):

Copy the line github.com/ReTWi/mysuperapp.git to download the application.

We return to the bitvise command line, stop the application by pressing ctrl + c (if it still works).
Go to the directory / opt and delete the folder we created with the application

Git is what we will use to download our application on github, and from there to a hosting or other PC. Git is a separate topic for discussion, so for now let's stop at this.
Install git on hosting with commands:

 sudo apt update sudo apt install git

Check whether everything is well established:

 git --version

The git version should appear.

We fill in the git data (and did not understand why, but apparently there may be some boring warnings).

 git config --global user.name "Your Name" git config --global user.email "youremail@domain.com"

Finally, we upload our application to the hosting with the command:
(there should be a link to your application)

 git clone https://github.com/ReTWi/mysuperapp.git

A new mysuperapp will appear in the / opt directory, where our application files downloaded from github will be located.

Now it's time to repeat the same thing for the PC and close the PC chain (different) -> Github -> Hosting

Install node.js on the PC .

Visual studio code

To begin, select the source code editor, where we will work. I chose the Visual studio code, it is so simple, convenient, there are a lot of plug-ins in it and you can set up the synchronization of settings if you work from several devices. Actually download, install, run, select a shared application folder, as git clone will create your own.

The plugins I use are the following:

Install git for PC .
Open the console in VScode with ctrl + shift + `or terminal> new terminal

Retreat:

In the windows console, it’s bad with Russian characters and so that there are no cracks you need to open file> preferences> settings, type in terminal.integrated.shellArgs.windows, click

And add the line "terminal.integrated.shellArgs.windows": ["-NoExit", "/ c", "chcp 65001"],

Repeat the command to download files from github:

 git clone https://github.com/ReTWi/mysuperapp.git

In VScode, click File> Open Folder and open the folder of our application.

Create a server.js file with the same simple server code:

 const http = require('http'); const hostname = 'localhost'; const port = 80; const server = http.createServer((req, res) => { res.statusCode = 200; res.setHeader('Content-Type', 'text/plain'); res.end('Hello World!\n'); }); server.listen(port, hostname, () => { console.log(`Server running at http://${hostname}:${port}/`); });

Install nodemon to automatically reboot the server when code changes:

 npm i nodemon -g

i - short for install
g - global installation (to be available in the console), and not just for our application.

Run the command:

 nodemon server.js

Open localhost : 80 / or just localhost: 80 in the browser and see Hello World.

Now it's time to check our PC chain> Github> Hosting.

Download the Github desktop for more convenience, connect your github account, then click the file add local repository and specify the directory of our application.

In the application, we see the changes we made compared to the version loaded with Github (we added server.js):

Click “commit to master”> “push origin”, thus uploading files from a PC to Github.

Go to our github account in the browser and see the uploaded server.js file:

Let's practice a little more, in VScode we replace the string “res.end ('Hello World! \ N');” with “res.end ('OmNomNom');”. We will see that the server itself has rebooted:

Check in the browser and see the OmNomNom changes we made.

Desktop github will also show us that we changed the line:

Again, click “commit to master”> “push origin” to send the files to github.

Switch to hosting command line.

Stop our application if it is still running (ctrl + c).

Download our updated application with the commands:

 git config credential.helper store git pull

The first will save our data in order not to constantly enter a login and password. In the future, git pull will suffice.

Install pm2 - something like nodemon, only for hosting:

 npm i pm2 -g

Let's start the application using pm2, which next time git pull on the hosting will restart our server:

 pm2 start server.js --watch

Open the browser on our external IP cloud and see our “OmNomNom”.

Thus, we have closed the chain of work with the application and its rapid deployment on the hosting.

Create temporary SSL certificates for HTTPS on localhost and hosting

Go to the site zerossl.com

In the domains, ip ... field, first enter localhost, click generate and download 2 files by the button:

We keep them in our project in the ssl / localhost folder.

Repeat the procedure for external IP Clouds and save to ssl / myapp.

We start more complicated https server node.js

Application structure:

client - our front-end will lie here. I have react.
logs - logs on hosting will fall here
node_modules - node.js modules
private - your private files, I keep SSH access to the cloud there
server is your backend
ssl - ssl certificates for https to work on localhost and on hosting
.babelrc - webpack'om application's react assembly settings (allows using more modern JS in the development of frontend)
.gitignore - files that will not be moved to github (git doesn't seem to see them)
client.js - entry point for generating react assemblies
package.json - node_modeles and various command snippets you use.
package-lock.json - changes in the modules (as I understand it, the file will check for the same modules installed on the hosting and on the PC).
pm2-watch.json - startup settings for pm2
README.md - cover for github
server.js - the launch point of our backend server Node.js
webpack.config.js - react assembly configuration

.gitignore

Here we indicate those files / folders that we do not want to upload to github. They will only be on this device and git will not monitor / show their changes. Open and paste:

 /node_modules/ /logs/* # exception to the rule !logs/.gitkeep /public/react_bundle.js /public/isProd.js

Since github does not upload empty folders, you can put something inside, for example, an empty .gitkeep file. Save the file and close.

package.json

Open and paste the following (after // added comments)

 { "name": "myapp", // название вашего приложения "version": "1.0.0", "description": "OmNomNom", "main": "server.js", "scripts": { "server": "pm2 start pm2-watch.json", // командой npm run server можно запустить этот скрипт "client": "webpack -w --mode development", // командой npm client можно запустить этот скрипт. Собирает приложение реакт и отслеживает изменения в коде, автоматически обновляя сборку. "client-prod": "webpack --mode production", // собирает сжатый вариант для production "client-analyze": "webpack --mode production --analyze true" // собирает сжатый вариант для production и позволяет посмотреть размеры разных модулей приложения. Полезно для оптимизации }, "repository": { "type": "git", "url": "git+https://github.com/myapp/knigam.git" // ссылка на ваш репозиторий github }, "author": "rtw", "license": "UNLICENSED", // запрет на любое использование (личное приложение) "bugs": { "url": https://github.com/myapp/knigam.git" }, "homepage": "https://github.com/myapp/knigam.git#readme", "dependencies": { "@babel/core": "^7.2.2", // Современный js для frontend "@babel/plugin-transform-runtime": "^7.2.0", // Современный js для frontend "@babel/preset-env": "^7.3.1", // Современный js для frontend "@babel/preset-react": "^7.0.0", // Современный js для frontend "ajv": "^6.8.1", // Проверка типов переменных "babel-loader": "^8.0.5", // Современный js для frontend "babel-plugin-styled-components": "^1.10.0", // Работа со styled-components "css-loader": "^2.1.0", // Для сборки webpack'om css "fastify": "^2.0.0-rc.6", // аналог express, более живой и активно развивающийся "fastify-cookie": "^2.1.6", // Работа с куки "fastify-static": "^2.2.0", // Работа со статичными файлами "moment": "^2.24.0", // Работа со временем "pg": "^7.8.0", // Работа со временем "pino": "^5.11.1", // Работа с postgreSQL из node.js "pino-pretty": "^2.5.0", // Читаемые логи в консоли "react": "^16.8.1", // Frontend библиотека. Выбор был между ней и Vue.js, но второй больше фрэймворк. В реакте больше нужно делать руками, что полезно для обучения "react-dom": "^16.8.1", // React для работы с браузером "style-loader": "^0.23.1", // Для сборки webpack'om стилей, уже не помню "styled-components": "^4.1.3", // CSS in JS, очень удобно для динамических стилей и локализации стилей в компонентах "webpack": "^4.29.3", // Сборщик реакт приложения "webpack-bundle-analyzer": "^3.0.3", // Анализатор размеров модулей собранного реакт приложения "webpack-cli": "^3.2.3" // Консоль сборщик реакт приложения, не помню уже зачем } }

I’ll dwell on two main frameworks / libraries chosen for the application:
Fastify was chosen as an alternative to express.js, since the first one already has experimental support for htpp2, it is actively developing and it seems to me that it has more future than express.js, which has become very clumsy and somehow develops. On the other hand, express.js has been working for a long time and it will be easier for you to find information on it.

React was chosen because it was easier for me to work, understand and try everything with my own hands. Vue - already seemed something with its own rules, direction. Although in Vue you may have to write something less with your own hands, but since the priority was placed on training, it went somehow easier for a person who had not previously programmed react.

Save the package.json file and install all modules specified in dependencies with the command:

 npm i

We will have a node_modules folder, in which there will be all modules for our application.

client - while an empty folder
logs - inside there is a .gitkeep file, so that the folder migrates to the hosting and the logs are successfully dropped there. When developing we will display everything in the console.

public

Here the static files of our site will lie, the images there, favicons, etc.
Separately dwell on two files:
index.html:

 <!DOCTYPE html> <html> <head> <base href="/" /> <meta charset="UTF-8" /> <title>MyApp</title> </head> <body> <div id="cookies">Этот текст заменится содержимым react_bundle после его загрузки</div> <noscript >Система: онлайн подключение возможно только при наличии Javscript</noscript > <script src="react_bundle.js"></script> </body> </html>

- here we load the react-frontend and render it to the tag by its id.

isProd.js contains a single line “module.exports = false”
Since it is in exceptions .gitignore, it is not transferred. Accordingly, on the PC we set it to false, and on the hosting to true. Then we use this file to understand what kind of environment we are in now (development / production). It seemed to me the most convenient, besides, you can partially change the code during development and test the operation of the modules in production.

ssl - there are previously saved certificates in the localhost and myapp folders

.babelrc

 { "presets": [ [ "@babel/preset-env", { "targets": { "browsers": [">0.25%", "not ie 11", "not op_mini all"] } } ], "@babel/preset-react" ], "plugins": [ "babel-plugin-styled-components", "@babel/plugin-transform-runtime" ] }

Settings to create our react_bundle with support for browsers used by more than> 0.25% of users.

client.js

 import React from 'react' import { render } from 'react-dom' render(<div>Реакт!!</div>, document.getElementById('cookies'))

Render our frontend in a div with a cookies tag.

pm2-watch.json - allows hosting the “npm run server” command to start the server with tracking of changes in the code and automatic reboot.

webpack.config.js

Reactor application collector:

 const webpack = require('webpack'), path = require('path'), BundleAnalyzerPlugin = require('webpack-bundle-analyzer').BundleAnalyzerPlugin module.exports = (env, argv) => { let prod = argv.mode == 'production' let config = { entry: './client.js', output: { path: path.resolve('./public'), filename: 'react_bundle.js' }, module: { rules: [ { test: /\.(js|jsx)$/, exclude: /node_modules/, loader: 'babel-loader' }, { test: /\.css$/, use: ['style-loader', 'css-loader'] } ] }, resolve: { alias: { client: path.resolve('./client/shared'), public: path.resolve('./public') } }, plugins: [ argv.analyze ? new BundleAnalyzerPlugin() : false, prod ? new webpack.optimize.AggressiveMergingPlugin() : false, new webpack.ContextReplacementPlugin(/moment[\/\\]locale$/, /ru/) ].filter(Boolean), optimization: { minimize: prod ? true : false }, performance: { hints: false } } return config }

In short, it opens the client.js file and everything inside it, collecting react_bundle and placing it in the public folder, from which it will be loaded via the open index.html.

server.js

 const isProd = require('./public/isProd'), fs = require('fs'), log = require('./server/logger'), path = require('path') // Ошибки среды node.js, чтобы приложение никогда не падало process.on('unhandledRejection', (reason, promise) => { log.error({ reason, promise }, 'серверный процесс unhandledRejection') }) process.on('uncaughtException', err => { log.error({ err }, 'серверный процесс uncaughtException') }) // Redirect server from http port 80 to https 443 const fastifyHttp = require('fastify')({ logger: log, ignoreTrailingSlash: true }) fastifyHttp.listen(80, '::', (err, address) => { if (err) { log.error({ err, address }, 'Ошибка при запуске HTTP сервера') } else { log.warn('Http cервер запущен') } }) // Let's Encrypt challenge fastifyHttp.get('/.well-known/acme-challenge/:file', (req, res) => { let stream = fs.createReadStream( path.join(__dirname + '/ssl/.well-known/acme-challenge/' + req.params.file) ) res.type('text/html').send(stream) }) fastifyHttp.get('/*', (req, res) => { res.redirect(301, 'https://' + req.headers.host + req.raw.url) }) fastifyHttp.get('/', (req, res) => { res.redirect(301, 'https://' + req.headers.host + req.raw.url) }) // Сервер let fastifyOptions = { logger: log, ignoreTrailingSlash: true, http2: true } fastifyOptions.https = isProd ? { allowHTTP1: true, key: fs.readFileSync('./ssl/myapp/key.txt'), cert: fs.readFileSync('./ssl/myapp/crt.txt') } : { allowHTTP1: true, key: fs.readFileSync('./ssl/localhost/cert.key'), cert: fs.readFileSync('./ssl/localhost/cert.pem') } const fastify = require('fastify')(fastifyOptions) fastify.listen(443, '::', (err, address) => { if (err) { log.error({ err, address }, 'Ошибка при запуске сервера') } else { log.warn( `Сервер запущен в ${ isProd ? 'продакшен' : 'режиме разработки' }` ) } }) // Валидатор fastify.setSchemaCompiler(schema => { return ajv.compile(schema) }) // Ошибки fastify fastify.setErrorHandler((err, req, res) => { log.error({ err, req }, 'fastify errorHandler') // Ошибка валидации данных запроса if (err.validation) { return res.send({ error: 'Ошибка валидации данных запроса' }) } else { return res.send({ error: 'Ошибка errorHandler' }) } }) // Статические файлы fastify.register(require('fastify-static'), { root: path.join(__dirname, './public') }) // Куки fastify.register(require('fastify-cookie'), err => { if (err) log.error({ err }, 'fastify-cookie') }) // Ответ на любой запрос исключая апи / несуществующая страница // Тут мы на любой запрос отдаём index.html, по сути наш фронэнд // Запросы фронтэнда у нас принимаются с префиксом api, то есть GET /api/userdata fastify.setNotFoundHandler((req, res) => { res.sendFile('index.html') }) // Routes fastify.register( async openRoutes => { // Пути доступные всем openRoutes.register(require('./server/api/open')) openRoutes.register(async withSession => { // Пути доступные только после авторизации и проверки сессии // Проверяем прямо тут хуком, пример: ///withSession.addHook('preHandler', async (req, res) => { // if (!(await sessionManagerIsOk(req, res))) return // }) withSession.register(require('./server/api/with_session')) }) }, { prefix: '/api' } // префикс всех путей )

Folder server

Here lies on the backend and all the way.
logger.js - depending on the environment, isProd logs either to the console or to errors.log

 'use strict' const pino = require('pino'), isProd = require('../public/isProd') let logOptions = isProd ? { level: 'warn', // уровень логирования timestamp: () => { return ',"time":"' + new Date() + '"' } } : { level: 'warn', prettifier: require('pino-pretty'), prettyPrint: { levelFirst: true, translateTime: true } } let dest = isProd ? pino.destination('./logs/errors.log') : pino.destination(1) let log = pino(logOptions, dest) module.exports = log

server / api /
open.js - here we add our paths.

 'use strict' module.exports = function(fastify, options, next) { fastify.route({ method: 'GET', url: '/', handler: async (req, res) => { res.send('api / route') } }) fastify.route({ method: 'GET', url: '/hi', handler: async (req, res) => { res.send('api / route hi') } }) next() }

After setting up and checking everything on Localhost, we simply upload everything to github, and from there git pull to the hosting. All that needs to be done on the hosting is to install the node.js modules with the “npm i” command and create the isProd.js file

Automatically updated SSL

When you buy a domain for yourself and link it to the cloud's IP, sample instructions for REG.RU , you can install on the server an automatically updated free SSL for the operation of the site via https.

Our server works without nginx. Perhaps we will need it in the future as a load balancer or a faster HTTP server for distributing static files, but so far I don’t see a need for it. We do not need load balancing yet, but I didn’t find any comparison of the statics distribution speed.

Before installing in the ssl folder, create a .well-known folder, and in it acme-challenge. It will be /opt/myapp/ssl/.well-known/acme-challenge

To install on the server with node.js without nginx automatically updated SSL click on the link . Take turns executing commands in the hosting console:

 sudo apt-get update sudo apt-get install software-properties-common sudo add-apt-repository universe sudo add-apt-repository ppa:certbot/certbot sudo apt-get update sudo apt-get install certbot sudo certbot certonly

We choose the second verification method, which will place a certain file in the /opt/myapp/ssl/.well-known/acme-challenge folder, and after confirming the owner of the server it will delete it.

We specify our domain on request, for example: “example.com” and the path to the ssl folder of our application (the server is configured to give the file created by the bot) "/ opt / myapp / ssl".

The bot itself will set up a cron task to renew the certificate before its expiration within 90 days.

I did not think that it would take so much time to write everything, by 4 o'clock in the morning I could have missed something: /

The opinion of habravchan and specialists who mastered this canvas or read some particular moments is interesting. How is your development cycle? Are there any moments in which I am mistaken or am not acting in the best way?