📜 ⬆️ ⬇️

Transferring 30,000 lines of code from Flow to TypeScript

We recently moved 30 thousand lines of JavaScript from our MemSQL Studio system from Flow to TypeScript. In this article I will explain why we ported the code base, how it happened and what happened.

Disclaimer: my goal is not a criticism of Flow at all. I admire the project and think that there is enough room in the JavaScript community for both types of type checking. In the end, everyone will choose what suits him best. I sincerely hope that the article will help in this choice.

First I will bring you up to date. We at MemSQL are big fans of static and strong JavaScript typing to avoid common problems with dynamic and weak typing.

Speech about common problems:

  1. Errors of type in runtime due to the fact that different parts of the code are not consistent with implicit types.
  2. Too much time is spent writing tests for such trivial things as checking type parameters (checking in runtime also increases the size of the package).
  3. There is not enough editor / IDE integration, because without static typing it is much more difficult to implement the Jump to Definition function, mechanical refactoring and other functions.
  4. There is no possibility to write code around data models, that is, first to design data types, and then the code basically “writes itself”.

These are just some of the benefits of static typing, even more listed in a recent article on Flow .

At the beginning of 2016, we implemented tcomb to implement some type security in the runtime of one of our internal JavaScript projects (a disclaimer: I was not involved in this project). Although runtime checking is sometimes useful, it doesn’t even offer all the advantages of static typing (combining static typing with type checking in runtime may be suitable for certain cases, io-ts allows you to do this using tcomb and TypeScript, although I have never tried ). Understanding this, we decided to implement Flow for another project that we started in 2016. At that time, Flow seemed like a great choice:


When we started working on MemSQL Studio at the end of 2017, we were going to cover the entire application with types (it is written entirely in JavaScript: both the frontend and the backend are executed in the browser). We took Flow as a tool that was successfully used in the past.

But my attention was attracted by Babel 7 with TypeScript support . This release meant that switching to TypeScript no longer requires switching to the entire TypeScript ecosystem and you can continue to use Babel for JavaScript. More importantly, we could use TypeScript only for type checking , and not as a full-fledged "language".

Personally, I think that separating type checking from code generator is a more elegant way to use static (and strong) typing in JavaScript, because:

  1. We share the problems of code and typing. This reduces the stops on type checking and speeds up development: if for some reason type checking is slow, the code will still be generated correctly (if you use tsc with Babel, you can adjust it to the same behavior).
  2. Babel has great plugins and functions that the TypeScript generator doesn't have. For example, Babel allows you to specify supported browsers and automatically generate a code for them. This is a very complex function and it makes no sense to support it in parallel in two different projects.
  3. I like JavaScript as a programming language (apart from the lack of static typing), and I have no idea how long TypeScript will exist while I believe in ECMAScript for many years. Therefore, I prefer to write and “think” in JavaScript (note that I say “use Flow” or “use TypeScript” instead of “write on Flow” or “on TypeScript”, because I always present them with tools, not programming languages).

Of course, this approach has some drawbacks:

  1. The TypeScript compiler can theoretically perform type-based optimizations, but here we lose this opportunity.
  2. The configuration of the project is a little more complicated with an increase in the number of tools and dependencies. I think this is a relatively weak argument: a bunch of Babel and Flow never let us down.

TypeScript as an alternative to Flow


I noticed a growing interest in TypeScript in the JavaScript community: both online and with other developers. Therefore, as soon as I learned that Babel 7 supports TypeScript, I immediately began to explore potential transition options. In addition, we encountered some flaws in Flow:

  1. Lower quality of the integration of the editor / IDE (compared to TypeScript). Nuclide - Facebook's own IDE with the best integration - is outdated.
  2. There is a smaller community, which means fewer type definitions for different libraries, and they are of lower quality (at the moment the GitHub has a DefinitelyTyped 19,682 stars, and only 3070 has a type typed repository).
  3. Lack of a public development plan and poor interaction between Flow Team on Facebook and the community. You can read this employee comment on Facebook to understand the situation.
  4. Large memory consumption and frequent leaks — at some of our developers, Flow sometimes took up almost 10 GB of RAM.

Of course, we had to study how TypeScript suits us. This is a very difficult question: studying the topic involved thorough reading of the documentation, which helped to understand that for every Flow function there is an equivalent of TypeScript. Then I explored the open-source TypeScript development plan, and I really liked the features that are planned for the future (for example, partial derivation of the type arguments that we used in Flow).

Transfer of more than 30 thousand lines of code from Flow to TypeScript


To begin with, Babel had to be updated from 6 to 7. This simple task took 16 man-hours, because we decided to upgrade Webpack 3 to 4 at the same time. The task was complicated by some outdated dependencies in our code. The vast majority of JavaScript projects will not have such problems.

After that, we replaced Babel's Flow settings set with a new TypeScript settings set, and then for the first time launched the TypeScript compiler on all our source codes written with Flow. The result is 8245 syntax errors (tsc CLI does not show real errors for the project until all syntax errors have been corrected).

At first, this number frightened us (very), but we quickly realized that most of the errors are due to the fact that TypeScript does not support .js files. After studying the topic, I learned that TypeScript files must end with either .ts or .tsx (if they have JSX). It seems to me a clear inconvenience. In order not to think about the presence / absence of JSX, I simply renamed all the files to .tsx.

There are about 4,000 syntax errors left. Most of them are related to the type of import , which with the help of TypeScript can be replaced simply with import, as well as the difference in the designation of objects ( {||} instead of {} ). Quickly applying a pair of regular expressions, we left 414 syntax errors. Everything else had to be corrected manually:


After correcting all the syntax errors, tsc finally said how many real type errors in our code base are about 1300 in total. Now we had to sit down and decide whether to continue or not. In the end, if the migration takes weeks, then it is better to stay on Flow. However, we decided that transferring the code would require less than one week of one engineer’s work, which is quite acceptable.

Please note that at the time of the migration, we had to stop all the work on this code base. Nevertheless, in parallel, you can start new projects - but you have to keep in mind potentially hundreds of type errors in the existing code, which is not easy.

What kind of mistakes?


TypeScript and Flow in many ways handle JavaScript code differently. So, Flow is stricter with respect to some things, and TypeScript - with respect to others. A deep comparison of the two systems will be very long, so let's just study some examples.

Note: all references to the TypeScript sandbox assume "strict" parameters. Unfortunately, when you share a link, these parameters are not stored in the URL. Therefore, they must be set manually after opening any link to the sandbox from this article.

invariant.js


The invariant function turned out to be very common in our source code. Just to quote the documentation:

 var invariant = require('invariant'); invariant(someTruthyVal, 'This will not throw'); // No errors invariant(someFalseyVal, 'This will throw an error with this message'); // Error raised: Invariant Violation: This will throw an error with this message

The idea is clear: a simple function that gives an error on some condition. Let's see how to implement and use it on Flow:

 type Maybe<T> = T | void; function invariant(condition: boolean, message: string) { if (!condition) { throw new Error(message); } } function f(x: Maybe<number>, c: number) { if (c > 0) { invariant(x !== undefined, "When c is positive, x should never be undefined"); (x + 1); // works because x has been refined to "number" } }

Now load the same snippet into TypeScript . As you can see from the link, TypeScript gives an error, because it cannot understand that x guaranteed not to remain undefined after the last line. This is actually a known problem - TypeScript (so far) does not know how to do such an inference through a function. However, this is a very common pattern in our code base, so we had to manually replace each invariant instance (over 150 pieces) with another code, which immediately gives an error:

 type Maybe<T> = T | void; function f(x: Maybe<number>, c: number) { if (c > 0) { if (x === undefined) { throw new Error("When c is positive, x should never be undefined"); } (x + 1); // works because x has been refined to "number" } }

Not much compared to the invariant , but not such an important issue.

$ ExpectError vs. @ ts-ignore


Flow has a very interesting function, similar to @ts-ignore , except that it gives an error if the next line is not an error. This is very useful for writing “tests for types” that ensure that type checking (whether TypeScript or Flow) finds certain type errors.

Unfortunately, there is no such function in TypeScript, so our tests have lost some value. I look forward to implementing this function on TypeScript .

Common type errors and type inference


Often TypeScript allows more explicit code than Flow, as in this example:

 type Leaf = { host: string; port: number; type: "LEAF"; }; type Aggregator = { host: string; port: number; type: "AGGREGATOR"; } type MemsqlNode = Leaf | Aggregator; function f(leaves: Array<Leaf>, aggregators: Array<Aggregator>): Array<MemsqlNode> { // The next line errors because you cannot concat aggregators to leaves. return leaves.concat(aggregators); }

Flow prints the leaves.concat (aggregators) type as an Array <Leaf | Aggregator> , which can then be Array<MemsqlNode> to Array<MemsqlNode> . I think this is a good example where Flow is a little smarter, and TypeScript needs a little help: in this case we can apply a type assertion, but this is dangerous and should be done very carefully.

Although I have no formal evidence, but I believe that Flow is far superior to TypeScript in the derivation of types. I really hope that TypeScript will reach Flow level, since the language is developing very actively, and many recent improvements have been made in this particular area. In many places our code had to help TypeScript a little through annotations or type statements, although we avoided the latter as much as possible). Consider another example (we had more than 200 such errors):

 type Player = { name: string; age: number; position: "STRIKER" | "GOALKEEPER", }; type F = () => Promise<Array<Player>>; const f1: F = () => { return Promise.all([ { name: "David Gomes", age: 23, position: "GOALKEEPER", }, { name: "Cristiano Ronaldo", age: 33, position: "STRIKER", } ]); };

TypeScript will not allow you to write this, because it will not allow to declare { name: "David Gomes", age: 23, type: "GOALKEEPER" } as an object of type Player (see the exact sandbox for an exact error). This is another case where I find TypeScript not smart enough (at least compared to Flow, which understands this code).

There are several options for fixing this:


Here is another example (TypeScript), where Flow is again better in type deduction :

 type Connection = { id: number }; declare function getConnection(): Connection; function resolveConnection() { return new Promise(resolve => { return resolve(getConnection()); }) } resolveConnection().then(conn => { // TypeScript errors in the next line because it does not understand // that conn is of type Connection. We have to manually annotate // resolveConnection as Promise<Connection>. (conn.id); });

A very small, but interesting example: Flow considers Array<T>.pop() type T , and TypeScript considers it to be T | void T | void . A point in favor of TypeScript, because it forces double-checking the existence of an element (if the array is empty, then Array.pop returns undefined ). There are several other small examples like this where TypeScript is superior to Flow.

TypeScript definitions for third-party dependencies


Of course, when writing any JavaScript application, you will have at least a few dependencies. They should be typed, otherwise you will lose most of the static type analysis capabilities (as described at the beginning of the article).

Libraries from npm can be supplied with a Flow or TypeScript type definition, with or without both. Very often (small) libraries are not supplied with either one or the other, so you have to write your own type definitions or borrow them from the community. Both Flow and TypeScript support standard definition repositories for third-party JavaScript packages: it is flow-typed and DefinitelyTyped .

I must say that DefinitelyTyped we liked much more. With flow-typed, I had to use the CLI tool to introduce type definitions for various dependencies into a project. DefinitelyTyped integrates this function with the npm CLI tool, sending @types/package-name packages to the npm package repository. This is very cool and has greatly simplified the input of type definitions for our dependencies (jest, react, lodash, react-redux, these are just a few).

In addition, I had a great time replenishing the DefinitelyTyped database (don't think that type definitions are equivalent when porting code from Flow to TypeScript). I have already sent several pull requests , and there were no problems anywhere. Just clone the repository, edit the type definitions, add tests - and send a pull request. The GitHub-bot DefinitelyTyped marks the authors of the definitions you edited. If none of them provides a review within 7 days, then the pull-request comes to the consideration of the maintainer. After merging with the main branch, a new version of the dependency package is sent to npm. For example, when I first updated the @ types / redux-form package, version 7.4.14 was automatically sent to npm. so updating package.json is enough to get new type definitions. If you can’t wait for the pull-request, you can always change the type definitions that are used in your project, as described in one of the previous articles .

In general, the quality of type definitions in DefinitelyTyped is much better due to the larger and more prosperous TypeScript community. In fact, after transferring the project to TypeScript , our type coverage increased from 88% to 96% mainly due to better definitions of third-party dependency types, with fewer any types.

Lintting and tests


  1. We switched from eslint linter to tslint (with eslint for TypeScript it seemed harder to get started).
  2. For tests on TypeScript ts-jest is used . Some of the tests are typed, while others are not (if you type too long, we save them as .js files).

What happened after fixing all typing errors?


After 40 man-hours of work, we reached the last typing error, postponing it for a while using @ts-ignore .

After reviewing the code review comments and fixing a couple of bugs (unfortunately, we had to slightly change the runtime code to correct the logic that TypeScript could not understand) the pull request was gone, and since then we have been using TypeScript. (And yes, we fixed that last @ts-ignore in the next pull request).

In addition to integrating with the editor, working with TypeScript is very similar to working with Flow. Flow server performance is slightly higher, but this is not a big problem, because they issue errors for the current file equally quickly. The only difference in performance is that TypeScript a little later (0.5−1 s) reports new errors after saving the file. The server startup time is about the same (about 2 minutes), but this is not so important. Until now, we have not had any problems with memory consumption. It seems that tsc constantly uses about 600 MB.

It may seem that the type inference function gives a great advantage to Flow, but there are two reasons why this does not really matter:

  1. We converted Flow code base to TypeScript. Obviously, we only got code that Flow can express, but TypeScript is not. If the migration was happening in the opposite direction, I am sure that there would be things that TypeScript better displays / expresses.
  2. Type inference is important in helping to write more concise code. But still, other things are more important, such as a strong community and the availability of type definitions, because weak type inference can be fixed by spending a little more time on typing.

Code statistics


 $ npm run type-coverage # https://github.com/plantain-00/type-coverage 43330 / 45047 96.19% $ cloc # ignoring tests and dependencies -------------------------------------------------------------------------------- Language files blank comment code -------------------------------------------------------------------------------- TypeScript 330 5179 1405 31463

What's next?


We are not finished with improving static type analysis. There are other projects in MemSQL that will eventually switch from Flow to TypeScript (and some JavaScript projects that will start using TypeScript), and we want to make our TypeScript configuration more restrictive. Currently, we have the strictNullChecks option enabled , but noImplicitAny is still disabled. We will also remove a couple of dangerous type assertions from the code.

I am glad to share with you all that I learned during the adventures with JavaScript typing. If any particular topic is interesting, please let me know .

Source: https://habr.com/ru/post/436554/

More articles: