📜 ⬆️ ⬇️

Practical application of transformation of AST-trees on the example of Putout

Introduction


Every day when working on the code, on the way to implementing useful functionality for the user, there are forced (inevitable, or simply desirable) changes to the code. This may be refactoring, updating a library or framework to a new major version, updating JavaScript syntax (which is not uncommon recently). Even if the library is part of a working draft - changes are inevitable. Most of these changes are routine. There is nothing interesting for the developer on the one hand, on the other it does not bring anything to the business, and on the third, in the update process, you need to be very careful not to break the wood and break the functionality. Thus, we come to the conclusion that it is better to shift such a routine onto the shoulders of the programs, what would they all do themselves, and the person, in turn, would control whether everything was properly done. That is what this article is about.


AST


For programmatic processing of the code, it is necessary to translate it into a special presentation with which it would be convenient for the programs to work. Such a representation exists, it is called Abstract Syntax Tree (AST).
In order to get it, use parsers. The resulting AST can be transformed as you like, and then to save the result you need a code generator. Let us consider in more detail each of the steps. Let's start with the parser.


Parser


And so we have the code:


a + b

Usually parsers are divided into two parts:



Splits the code into tokens, each of which describes a part of the code:


 [{ "type": "Identifier", "value": "a" }, { "type": "Punctuator", "value": "+", }, { "type": "Identifier", "value": "b" }]


Builds a syntax tree of tokens:


 { "type": "BinaryExpression", "left": { "type": "Identifier", "name": "a" }, "operator": "+", "right": { "type": "Identifier", "name": "b" } }

And here we already have the very idea with which you can work programmatically. It should be clarified that there are a large number of JavaScript parsers, here are some of them:



There is a standard JavaScript parsers, it is called ESTree and determines which nodes should parse as it should.
For a more detailed analysis of the implementation process of the parser (as well as the transformer and generator), you can read super-tiny-compiler .


Transformer


In order to convert an AST tree, you can use the Visitor pattern, for example, using the @ babel / traverse library. The following code will output the names of all the JavaScript code identifiers from the code variable.


 import * as parser from "@babel/parser"; import traverse from "@babel/traverse"; const code = `function square(n) { return n * n; }`; const ast = parser.parse(code); traverse(ast, { Identifier(path) { console.log(path.node.name); } });

Generator


You can generate code, for example, using @ babel / generator , thus:


 import {parse} from '@babel/parser'; import generate from '@babel/generator'; const code = 'class Example {}'; const ast = parse(code); const output = generate(ast, code);

And so, at this stage, the reader had to get a basic idea of ​​what is needed to transform JavaScript code, and with what tools this is implemented.


It is necessary to add such an online tool as astexplorer , it combines a large number of parsers, transformers and generators.


Putout


Putout is a code transformer with plug-in support. In fact, it is a cross between eslint and babel , combining the advantages of both tools.


As eslint putout shows problem areas in the code, but unlike eslint putout changes the behavior of the code, that is, it is able to correct all errors that it can find.


Like babel putout transforms the code, but tries to change it as little as possible, so it can be used to work with code that is stored in the repository.


Another thing worth mentioning is the prettier , it is a formatting tool, and it differs radically.


Jscodeshift is not very far from putout , but it does not support plugins, does not show error messages, and also uses ast-types instead of @ babel / types .


Appearance history


In the process of work, eslint helps me a lot with my hints. But sometimes you want more from him. For example, what would he remove the debugger , fix test.only , and also delete unused variables. The last point formed the basis of putout , in the development process, it became clear that this is not very simple and many other transformations are much easier to implement. Thus, putout smoothly evolved from one function to a plugin system. Removing unused variables is still the most difficult process, but it doesn’t interfere with the development and maintenance of many other equally useful transformations.


How Putout Works from the Inside


Work putout can be divided into two parts: the engine and plugins. This architecture allows you to not be distracted by the transformation when working with the engine, and when working on plug-ins you will focus on their purpose.


Built-in plugins


The work putout is built on the plugin system. Each plugin represents one rule. Using the built-in rules, you can do the following:



  // было import one from 'one'; // станет const one = require('one'); 


 // было const name = user.name; // станет const {name} = user;

  1. Combine unstructure properties:

 // было const {name} = user; const {password} = user; // станет const { name, password } = user;

Each plug-in is built according to the Unix Philosophy , that is, they are as simple as possible, each performs one action, making them easy to combine, because they, in essence, are filters.


For example, having the following code:


 const name = user.name; const password = user.password;

It is first converted into the following using apply-destructuring :


 const {name} = user; const {password} = user;

After that, using merge-destructuring-properties is converted to:


 const { name, password } = user;

Thus, plug-ins can work both separately and together. When creating your own plugins, it is recommended to adhere to this rule, and implement a plug-in with minimal functionality that does only what you need, and the plug-in and user plug-ins take care of the rest.


Usage example


After we have familiarized ourselves with the built-in rules, we can consider an example of using putout .
Create an example.js file with the following contents:


 const x = 1, y = 2; const name = user.name; const password = user.password; console.log(name, password);

Now, run putout , passing example.js as an argument:


 coderaiser@cloudcmd:~/example$ putout example.js /home/coderaiser/example/example.js 1:6 error "x" is defined but never used remove-unused-variables 1:13 error "y" is defined but never used remove-unused-variables 6:0 error Unexpected "console" call remove-console 1:0 error variables should be declared separately split-variable-declarations 3:6 error Object destructuring should be used apply-destructuring 4:6 error Object destructuring should be used apply-destructuring 6 errors in 1 files fixable with the `--fix` option

We will receive information containing 6 errors, discussed in more detail above, now we will correct them, and see what happened:


 coderaiser@cloudcmd:~/example$ putout example.js --fix coderaiser@cloudcmd:~/example$ cat example.js const { name, password } = user;

As a result of the correction, unused variables and console.log calls were removed, and destructuring was also applied.


Settings


The default settings may not always be suitable for everyone, therefore putout supports the configuration file .putout.json , it consists of the following sections:



Rules

The rules section contains a rule system. The rules, by default, are set as follows:


 { "rules": { "remove-unused-variables": true, "remove-debugger": true, "remove-only": true, "remove-skip": true, "remove-process-exit": false, "remove-console": true, "split-variable-declarations": true, "remove-empty": true, "remove-empty-pattern": true, "convert-esm-to-commonjs": false, "apply-destructuring": true, "merge-destructuring-properties": true } }

In order to enable remove-process-exit enough to set it to true in the .putout.json file:


 { "rules": { "remove-process-exit": true } }

This will be enough to report all the process.exit calls found in the code, and remove them if the --fix parameter is --fix .


Ignore

If you need to add some folders to the list of exceptions, just add the ignore section:


 { "ignore": [ "test/fixture" ] }

Match

In case of need of an extensive system of rules, for example, enable process.exit for the bin directory, just use the match section:


 { "match": { "bin": { "remove-process-exit": true, } } }

Plugins

In the case of using plugins that are not embedded and have the putout-plugin- , they must be included in the plugins section before being activated in the rules section. For example, to connect the putout-plugin-add-hello-world and enable the add-hello-world rule, it’s enough to specify:


 { "rules": { "add-hello-world": true }, "plugins": [ "add-hello-world" ] }

Putout engine


The putout engine is a command line tool that reads the settings, parses the files, loads and launches the plugins, and then writes the result of the plugins.


It uses the recast library, which helps to accomplish a very important task: after parsing and transformation, collect the code in a state as close as possible to the previous one.


For parsing, ESTree compatible parser (currently babel with an estree plugin, but changes are possible in the future), and for transformation, babel tools. Why exactly babel ? It's simple. The fact is that this is a very popular product, much more popular than other similar tools, and it develops much more rapidly. Each new proposal in the standard EcmaScript can not do without a babel-plugin . Babel also has a Babel Handbook book in which all the features and tools for circumventing and transforming an AST tree are described very well.


Own plugin for Putout


The putout plugin system is quite simple, and very similar to the eslint plugins , as well as the babel plugins . True, instead of a single function, the putout plugin should export 3. This is done to increase the reuse of the code, because duplicating the functionality in 3 functions is not very convenient, it is much easier to put it into separate functions and just call it in the right places.


Plugin structure

So Putout plugin consists of 3 functions:



The main point to remember when creating a plugin for putout is its name, it must begin with putout-plugin- . Next can be the name of the operation that the plugin performs, for example, the plug-in remove-wrong should be called like this: putout-plugin-remove-wrong .


You should also add the words: putout and putout-plugin in package.json , in the keywords section, and specify "putout": ">=3.10" in peerDependencies "putout": ">=3.10" , or the version that will be the last one at the time of writing the plugin.


Example plugin for Putout

Let's write, for example, a plugin that will remove the word debugger from the code. Such a plugin is already there, it is @ putout / plugin-remove-debugger and it is simple enough to consider it now.


It looks like this:


 // возвращаем ошибку соответствующую каждому из найденых узлов module.exports.report = () => 'Unexpected "debugger" statement'; // в этой функции ищем узлы, содержащией debugger с помощью паттерна Visitor module.exports.find = (ast, {traverse}) => { const places = []; traverse(ast, { DebuggerStatement(path) { places.push(path); } }); return places; }; // удаляем код, найденный в предыдущем шаге module.exports.fix = (path) => { path.remove(); };

If a remove-debugger rule is included in .putout.json , the @putout/plugin-remove-debugger will be loaded. First, the find function is called which, using the traverse function, traverse nodes of the AST tree and saves all the necessary places.


The next step putout will turn to report for the desired message.


If the --fix flag is used, the plug-in fix function will be called and the transformation will be performed, in this case, the node is deleted.


Sample plugin test

In order to simplify the testing of plugins, the @ putout / test tool was written. At its core, this is nothing more than a wrapper over a tape , with several methods for convenience and ease of testing.


The test for the remove-debugger plugin can look like this:


 const removeDebugger = require('..'); const test = require('@putout/test')(__dirname, { 'remove-debugger': removeDebugger, }); // проверяем что бы сообщение было именно таким test('remove debugger: report', (t) => { t.reportCode('debugger', 'Unexpected "debugger" statement'); t.end(); }); // проверяем результат трансформации test('remove debugger: transformCode', (t) => { t.transformCode('debugger', ''); t.end(); });

Codemods

Not any transformation needs to be used every day, for one-time transformations, it is enough to do everything the same, only instead of publishing to npm place it in the ~/.putout . When launching, putout will look in this folder, pick up and start transformations.


Here is an example of a transformation that replaces the tape connection and try-to-tape connection with a supertape call: convert-tape-to-supertape .


eslint-plugin-putout


Finally, it’s worth adding one thing: putout tries to change the code minimally, but if it happens to a friend that some formatting rules break, eslint --fix is ​​always ready to eslint --fix , and for this purpose there is a special plugin eslint-plugin-putout . It can brighten up many formatting errors, and of course it can be configured in accordance with the preferences of developers on a particular project. Connect it easily:


 { "extends": [ "plugin:putout/recommended", ], "plugins": [ "putout" ] }

So far there is only one rule in it: one-line-destructuring , it does the following:


 // было const { one } = hello; // станет const {one} = hello;

There are still many included eslint rules that you can read in more detail .


Conclusion


I want to thank the reader for paying attention to this text. I sincerely hope that the topic of AST transformations will become more popular, and articles about this fascinating process will appear more often. I would be very grateful for any comments and suggestions related to the future direction of putout development. Create an issue , send a pool of requests , test, write what rules you would like to see, and how to transform your code programmatically, we will work together to improve the AST transformation tool.



Source: https://habr.com/ru/post/439564/

More articles: