Selection of technologies, architecture and design in software projects - without bills
Friends! We are continuing a series of “no-bills” publications about project processes, IT-technologies and how to work effectively. Today we will talk about a very painful topic that causes heartburn in the brain - the choice of technologies, programming languages, the role of architects, analysts, timlids and psychics to solve an epic task: to launch a software solution, if possible, within a reasonable time. And we’ll stop separately, well, in order not to get bored at all, on the analysis of the correlation of the sizes of body parts of one part of a team with the performance of the brain of another. Pour coffee and go!
In order to objectively understand why there is still no clarity in the indicated issues and there are so many “religious” disputes with human sacrifices, you need to distract, smile and draw the life course of an IT specialist in the form of a broken line on the table with squares that signify the acquired knowledge with a kitchen knife.
The fun begins with the fact that, usually, even in specialized universities there is not enough knowledge for professional programming and the competence of a specialist directly depends on the spiritual struggle with laziness: take up a book / code for a couple of hours or fill up with a can of beer in front of the TV. Obviously, only a small part can withstand social networks, television series, online games and selflessly learn, but even if, like the monks, constantly and daily sacrifice a part of life, you can understand well one, well, sometimes two industrial programming languages and several close technologies. And truncated, the pension begins and while for the apiary bees are bred.
But with such difficulty, the knowledge obtained for success, it turns out, is still not enough and requires constant specialization in development projects, preferably with hell deadlines, because writing skills right away quickly and correctly, is lost pretty soon.
As a result, only a few squares can be cut with a knife, and the more to the right, the smaller squares, as a rule, are born, for children are born, acquaintances appear, a hobby and you want to live at least a little for yourself at night and play enough tanchiki.
A very clear analogy of an IT expert is probably a martial arts expert who after school left Voronezh to China, spent 10 years in a monastery, stuffed knuckles on his shoulders, cut off with his genitalia and trained 20 hours a day. The fighter will even be nice to know several dialects of Chinese, but one fine day he will face a swarthy guy who has dedicated himself to Capoeira . Most likely, at the level of signs, they will understand each other and will be able to choose technology for a software project, but, unfortunately, most often have to meet with aggressive lovers of cosplay and science fiction - in a Jedi suit with a plastic lightsaber in his hands.
Obviously, only a handful can learn and learn the Truth selflessly and in real life meetings and debates about technologies and programming languages occur mainly between fans of different and fashionable "serials" who do not know, often, the difference between IP and TCP. The problem is exacerbated by aggressive external marketing - most large IT vendors actively “recruit” followers into their ranks, using pumped information channels. You don’t need to go far for examples - Mozilla promotes Rust, Google promotes Go, Oracle is interested (?) In Java promotion, Microsoft develops C #, JetBrains - Kotlin, aliens - Haskell, and the scientific community will not rejoice when under the guise of AI and ML, neuron and datasatanizma, for big money sold ... dusty academic courses on probability theory, line and statistics for ever and ever, amen :-)
And when a Game of Thrones fan starts arguing with a fan of the Clan Soprano about the benefits of aspect-oriented programming , mixins and closures , all that remains is genuinely crazy to laugh and sprinkle debaters with holy sand from Tibet with McDonald's potatoes. There are no useful and objective conclusions to wait and you will get confused even more - and, even worse, you will get sick with viral "pops" and start walking in a striped raincoat.
The fact is that to collect, for example, a car made of iron is quite expensive and long time: you need material, welding, knowledge, a room, a poster of a girl with the correct forms, etc. And to write a program, especially not alone, is quite common. Virtual creativity - blew up and continues to crumple and stretch the world. A sea of digital content, often stupid and primitive, where all writers and artists flooded the web. As a result of the relative simplicity of “self-expression”, in the world, in a relatively short period of time, after World War II, a huge number of not only programs, but also programming languages appeared. It is easy to write a new programming language, it is much more difficult to make it popular. So historically, the languages in which they learned to “write them correctly” (for example C ++, Java, ECMA262) are popular now, and the languages that “made later and, like, better” (for example C #, Python3, Rust, Kotlin) , are much less popular. Choosing a "more correct" language, you risk not finding a sufficient number of developers who know it, or it may happen that the language simply stops developing or ascends into eternity, like Haskell. Therefore, guided by common sense, often choose traditional, so to speak, technologies, even if they are not quite “correct” and consistent, full of crutches and traces of maturation, but they are proven, popular and developed.
In the context of active development graphomania (especially in the direction of the frontend, although it is also fun in npm) I would like to warn against the mindless use of libraries and frameworks . It is important to look at the development history of the framework, the contingent of developers (the simpler the technology, the more surprises) and the duration of version support.
Usually, in open source projects, it is not customary to maintain the framework for a long time, for example, 3-5 years or more - this is trivial, expensive and boring. Therefore, usually, after a set of critical masses of “crutches” is created, the next version is created, after a year or two, and I recommend that users of previous versions move their project ... with their own hands, because the previous version will not be updated anymore and security errors will not be eliminated either. Examples of such "irresponsibility" - just the sea.
Commercial libraries and frameworks, usually , do not suffer from a similar childhood disease (though not all, you need to carefully look into the documentation and study this issue thoroughly) and maintain backward compatibility for 5 years or more. Please pay special attention to this, otherwise you will have to suddenly rewrite your web solution almost from scratch!
In general, on this point, better use the proverb about tit and crane.
Another very popular psychological problem during development is the choice between “doing from scratch” and “take ready”. I will tell you a secret - everyone wants to be gods, novice developers, which, by the way, is normal, they want to pump and get an entry in the resume and choose to “do it from scratch”. But then, usually, they quit due to boredom after six months or a year and, often due to lack of experience, their works have to be rewritten again and again "from scratch". Experienced Shaolin monks usually choose ready-made solutions and libraries to insure against children's mistakes and surprises, and spend the remaining free time developing non-standard, exclusive for this web-project functionality. And it will not be boring!
Choosing a database can be a good association - what to take for a web project, an Oracle database or a MySQL database (yes, I know that they are now being developed by one company). By experience - 99% of web development tasks, even under high loads, are excellent and are solved at high speed in MySQL. And if the result is no different, why ...? :-)
Therefore, usually a software project is now a bunch of ready-made and proven libraries and frameworks, with a good support period and only a small part of the functionality special to this project. And this, IMHO, is correct.
This is also a well-known reason for numerous and endless disputes. In fact, modern IT-technologies and programming languages are divided, roughly, into 2 parts: simple for the majority (no need to delve deeply, several days for understanding) and difficult for the minority (need serious and long specialization, months for understanding). As we saw above, most do not like to study, therefore introducing the correct and reliable technology can be - you will not succeed due to the fact that no one will fully understand it and you will play Russian roulette with a black box. A popular example: the choice between redis or cassandra - products of a different order of complexity. Or, another example: python or C ++.
In simple languages, it is valid to program, often, much, by orders of magnitude, faster — let's compare a python script of 5 lines and similar C code to 100 lines. But, usually, “simple” languages and universal technologies sometimes work much slower and consume much more RAM :-) After all, you have to pay for everything.
Nevertheless, “simple” languages are more and more often used to solve most problems of a software project, where special requirements for super-speed and RAM are not needed. And iron becomes cheaper and cheaper. PHP is now very popular for rapid web development, and python, with its very clear and readable syntax, for common tasks and machine learning. JavaScript is very useful for automating not only the web interface, but also ... creating really fast server applications on Node.js. Go, even with its primitive syntax, allows you to manage system tasks without a more powerful, but much more difficult to understand Java, and you can quickly create a mobile application on orders of simpler and more accessible way for Kotlin. It is possible with much less risk to run a download application on Rust, without delving into the subtleties of memory management in C ++. But no one in their right mind would write a game in Java, recalling the popular minecraft, which can only be played at a distance of 10 meters from the display, squinting :-)
For each task, therefore, it is advisable to pick up a specific tool that has been sharpened for it. The more used tools, the better.
It is ridiculous, but in current realities, not even the choice of language, but the choice of possibilities of a language can have a decisive influence on success. For example, if you have a team that has no object-oriented programming experience and knowledge of design patterns, you can write such a blatant object zoo that it might be easier to solve the problem with 100 functions in one monolithic file that everyone understands than hiding in the office from hundreds of object monsters with a cupboard instead of a head and ears instead of legs. That is why, often, simple languages (such as python, php, javascript, ruby) that do not provide excessive opportunities for development grafomania and perfectionism, focused on clarity, unambiguity (python) and conciseness (php), make it possible to increasingly come to success. It’s not for nothing that stories are so popular when a website started being made in C ++ and then what a nightmare it turned out.
A good analogy here is an example of the expensive and difficult in the "setting" of the knights of the Middle Ages and ... hooligans with firearms. You can train all your life and become a samurai of multi-threaded Java, but it is unexpected to end your life when meeting a schoolboy with a similar and more simple solution on python or Go.
Indeed, 10–15 years ago, it was necessary to study for a long time and read thick books in order to understand the principles of placing objects and their interaction on separate servers and clusters (j2ee, corba, com). Echoes of this boom on architecture and architects can still be found in monstrous creations like Spring . But times change, technologies become more powerful and more accessible. By deploying a couple of free Apache web servers, a MySQL database, a RabbitMQ line somewhere in Amazon , you can solve most of the tasks that were previously available in understandable minority configurations of application servers .
If the task is to support mass network connections, you can raise a cluster of Node.js machines or, rather, machines with Erlang and sleep well than write your own server solution with, for example, Go or C ++.
If you need to do research, constantly, in a stream, selecting one of 10-20 machine learning models and quickly deploying them in the cloud for customer service, then undoubtedly python with a huge number of libraries will become a very convenient and practical solution, and so on.
Of course, in highly specialized projects, deep knowledge of OOP, OP , architectures, Computer Science is of course still required, but more and more often it is enough to know from which blocks to assemble the solution and this will often lead the program system to the goal much faster.
Taking into account the above, it is obvious that the team will most likely develop only within the limited framework of the set of technologies selected for the project. Units will begin to dig related topics, and a small fraction of them read special literature, but these, alas, are exceptional cases. Therefore, it is recommended to select those already experienced in the indicated languages / libraries, which can be checked in the resume and at the interviews. And only those who are really not hit by serials and social networks, can be taken to the "growth", having received a contract previously signed with arterial blood :-)
However, in non-standard projects, you will have to find at least one developer who knows something else and is excellent, except for python, ruby, php, javascript, go, perl, bash, chanson. The chances that the fighter is well aware of algorithms, design patterns, network standards, OOP are greatly increased.
If the project is related to machine learning and you need to dig something in depth, you will have to find a real analyst-mathematician, preferably with a scientific degree and knowledge of python. I am serious - for machine learning, without a core education, you need to study hard for several hours a day (probability theory, linear algebra, least squares method, mathematical statistics, Bayes theorem and its techniques) for several months, if not years.
You can often find long-lived and large software projects written in seemingly "unsuitable" programming languages or sets of libraries. For example, often there are huge control scripts on bash or massive scientific libraries and frameworks for accurate calculations on ... python with razdolbaysky duck typing . The secret is in rigorous processes and practices. Using:
You can create a software solution, in principle, on any technology that will live for years and delight customers. Conversely, working with inflated "pontes" with black boxes, without delving into the details of technology, encouraging negligence and personal interests instead of command ones, you can try to launch the project for months, correcting one mistake and generating dozens - unfortunately, this situation is quite common.
Let's summarize and prioritize:
Good luck to everyone, successful projects, adequate teams and a great mood!
How to become experts
In order to objectively understand why there is still no clarity in the indicated issues and there are so many “religious” disputes with human sacrifices, you need to distract, smile and draw the life course of an IT specialist in the form of a broken line on the table with squares that signify the acquired knowledge with a kitchen knife.
The fun begins with the fact that, usually, even in specialized universities there is not enough knowledge for professional programming and the competence of a specialist directly depends on the spiritual struggle with laziness: take up a book / code for a couple of hours or fill up with a can of beer in front of the TV. Obviously, only a small part can withstand social networks, television series, online games and selflessly learn, but even if, like the monks, constantly and daily sacrifice a part of life, you can understand well one, well, sometimes two industrial programming languages and several close technologies. And truncated, the pension begins and while for the apiary bees are bred.
But with such difficulty, the knowledge obtained for success, it turns out, is still not enough and requires constant specialization in development projects, preferably with hell deadlines, because writing skills right away quickly and correctly, is lost pretty soon.
As a result, only a few squares can be cut with a knife, and the more to the right, the smaller squares, as a rule, are born, for children are born, acquaintances appear, a hobby and you want to live at least a little for yourself at night and play enough tanchiki.
A very clear analogy of an IT expert is probably a martial arts expert who after school left Voronezh to China, spent 10 years in a monastery, stuffed knuckles on his shoulders, cut off with his genitalia and trained 20 hours a day. The fighter will even be nice to know several dialects of Chinese, but one fine day he will face a swarthy guy who has dedicated himself to Capoeira . Most likely, at the level of signs, they will understand each other and will be able to choose technology for a software project, but, unfortunately, most often have to meet with aggressive lovers of cosplay and science fiction - in a Jedi suit with a plastic lightsaber in his hands.
Kopipast, frameworks and science fiction
Obviously, only a handful can learn and learn the Truth selflessly and in real life meetings and debates about technologies and programming languages occur mainly between fans of different and fashionable "serials" who do not know, often, the difference between IP and TCP. The problem is exacerbated by aggressive external marketing - most large IT vendors actively “recruit” followers into their ranks, using pumped information channels. You don’t need to go far for examples - Mozilla promotes Rust, Google promotes Go, Oracle is interested (?) In Java promotion, Microsoft develops C #, JetBrains - Kotlin, aliens - Haskell, and the scientific community will not rejoice when under the guise of AI and ML, neuron and datasatanizma, for big money sold ... dusty academic courses on probability theory, line and statistics for ever and ever, amen :-)
And when a Game of Thrones fan starts arguing with a fan of the Clan Soprano about the benefits of aspect-oriented programming , mixins and closures , all that remains is genuinely crazy to laugh and sprinkle debaters with holy sand from Tibet with McDonald's potatoes. There are no useful and objective conclusions to wait and you will get confused even more - and, even worse, you will get sick with viral "pops" and start walking in a striped raincoat.
Tradition and "correctness"
The fact is that to collect, for example, a car made of iron is quite expensive and long time: you need material, welding, knowledge, a room, a poster of a girl with the correct forms, etc. And to write a program, especially not alone, is quite common. Virtual creativity - blew up and continues to crumple and stretch the world. A sea of digital content, often stupid and primitive, where all writers and artists flooded the web. As a result of the relative simplicity of “self-expression”, in the world, in a relatively short period of time, after World War II, a huge number of not only programs, but also programming languages appeared. It is easy to write a new programming language, it is much more difficult to make it popular. So historically, the languages in which they learned to “write them correctly” (for example C ++, Java, ECMA262) are popular now, and the languages that “made later and, like, better” (for example C #, Python3, Rust, Kotlin) , are much less popular. Choosing a "more correct" language, you risk not finding a sufficient number of developers who know it, or it may happen that the language simply stops developing or ascends into eternity, like Haskell. Therefore, guided by common sense, often choose traditional, so to speak, technologies, even if they are not quite “correct” and consistent, full of crutches and traces of maturation, but they are proven, popular and developed.
In the context of active development graphomania (especially in the direction of the frontend, although it is also fun in npm) I would like to warn against the mindless use of libraries and frameworks . It is important to look at the development history of the framework, the contingent of developers (the simpler the technology, the more surprises) and the duration of version support.
Usually, in open source projects, it is not customary to maintain the framework for a long time, for example, 3-5 years or more - this is trivial, expensive and boring. Therefore, usually, after a set of critical masses of “crutches” is created, the next version is created, after a year or two, and I recommend that users of previous versions move their project ... with their own hands, because the previous version will not be updated anymore and security errors will not be eliminated either. Examples of such "irresponsibility" - just the sea.
Commercial libraries and frameworks, usually , do not suffer from a similar childhood disease (though not all, you need to carefully look into the documentation and study this issue thoroughly) and maintain backward compatibility for 5 years or more. Please pay special attention to this, otherwise you will have to suddenly rewrite your web solution almost from scratch!
In general, on this point, better use the proverb about tit and crane.
New or ... ready
Another very popular psychological problem during development is the choice between “doing from scratch” and “take ready”. I will tell you a secret - everyone wants to be gods, novice developers, which, by the way, is normal, they want to pump and get an entry in the resume and choose to “do it from scratch”. But then, usually, they quit due to boredom after six months or a year and, often due to lack of experience, their works have to be rewritten again and again "from scratch". Experienced Shaolin monks usually choose ready-made solutions and libraries to insure against children's mistakes and surprises, and spend the remaining free time developing non-standard, exclusive for this web-project functionality. And it will not be boring!
Choosing a database can be a good association - what to take for a web project, an Oracle database or a MySQL database (yes, I know that they are now being developed by one company). By experience - 99% of web development tasks, even under high loads, are excellent and are solved at high speed in MySQL. And if the result is no different, why ...? :-)
Therefore, usually a software project is now a bunch of ready-made and proven libraries and frameworks, with a good support period and only a small part of the functionality special to this project. And this, IMHO, is correct.
Difficult and long or ... fast and easy?
This is also a well-known reason for numerous and endless disputes. In fact, modern IT-technologies and programming languages are divided, roughly, into 2 parts: simple for the majority (no need to delve deeply, several days for understanding) and difficult for the minority (need serious and long specialization, months for understanding). As we saw above, most do not like to study, therefore introducing the correct and reliable technology can be - you will not succeed due to the fact that no one will fully understand it and you will play Russian roulette with a black box. A popular example: the choice between redis or cassandra - products of a different order of complexity. Or, another example: python or C ++.
In simple languages, it is valid to program, often, much, by orders of magnitude, faster — let's compare a python script of 5 lines and similar C code to 100 lines. But, usually, “simple” languages and universal technologies sometimes work much slower and consume much more RAM :-) After all, you have to pay for everything.
Nevertheless, “simple” languages are more and more often used to solve most problems of a software project, where special requirements for super-speed and RAM are not needed. And iron becomes cheaper and cheaper. PHP is now very popular for rapid web development, and python, with its very clear and readable syntax, for common tasks and machine learning. JavaScript is very useful for automating not only the web interface, but also ... creating really fast server applications on Node.js. Go, even with its primitive syntax, allows you to manage system tasks without a more powerful, but much more difficult to understand Java, and you can quickly create a mobile application on orders of simpler and more accessible way for Kotlin. It is possible with much less risk to run a download application on Rust, without delving into the subtleties of memory management in C ++. But no one in their right mind would write a game in Java, recalling the popular minecraft, which can only be played at a distance of 10 meters from the display, squinting :-)
For each task, therefore, it is advisable to pick up a specific tool that has been sharpened for it. The more used tools, the better.
It is ridiculous, but in current realities, not even the choice of language, but the choice of possibilities of a language can have a decisive influence on success. For example, if you have a team that has no object-oriented programming experience and knowledge of design patterns, you can write such a blatant object zoo that it might be easier to solve the problem with 100 functions in one monolithic file that everyone understands than hiding in the office from hundreds of object monsters with a cupboard instead of a head and ears instead of legs. That is why, often, simple languages (such as python, php, javascript, ruby) that do not provide excessive opportunities for development grafomania and perfectionism, focused on clarity, unambiguity (python) and conciseness (php), make it possible to increasingly come to success. It’s not for nothing that stories are so popular when a website started being made in C ++ and then what a nightmare it turned out.
A good analogy here is an example of the expensive and difficult in the "setting" of the knights of the Middle Ages and ... hooligans with firearms. You can train all your life and become a samurai of multi-threaded Java, but it is unexpected to end your life when meeting a schoolboy with a similar and more simple solution on python or Go.
Architects & Architects
Indeed, 10–15 years ago, it was necessary to study for a long time and read thick books in order to understand the principles of placing objects and their interaction on separate servers and clusters (j2ee, corba, com). Echoes of this boom on architecture and architects can still be found in monstrous creations like Spring . But times change, technologies become more powerful and more accessible. By deploying a couple of free Apache web servers, a MySQL database, a RabbitMQ line somewhere in Amazon , you can solve most of the tasks that were previously available in understandable minority configurations of application servers .
If the task is to support mass network connections, you can raise a cluster of Node.js machines or, rather, machines with Erlang and sleep well than write your own server solution with, for example, Go or C ++.
If you need to do research, constantly, in a stream, selecting one of 10-20 machine learning models and quickly deploying them in the cloud for customer service, then undoubtedly python with a huge number of libraries will become a very convenient and practical solution, and so on.
Of course, in highly specialized projects, deep knowledge of OOP, OP , architectures, Computer Science is of course still required, but more and more often it is enough to know from which blocks to assemble the solution and this will often lead the program system to the goal much faster.
Team selection
Taking into account the above, it is obvious that the team will most likely develop only within the limited framework of the set of technologies selected for the project. Units will begin to dig related topics, and a small fraction of them read special literature, but these, alas, are exceptional cases. Therefore, it is recommended to select those already experienced in the indicated languages / libraries, which can be checked in the resume and at the interviews. And only those who are really not hit by serials and social networks, can be taken to the "growth", having received a contract previously signed with arterial blood :-)
However, in non-standard projects, you will have to find at least one developer who knows something else and is excellent, except for python, ruby, php, javascript, go, perl, bash, chanson. The chances that the fighter is well aware of algorithms, design patterns, network standards, OOP are greatly increased.
If the project is related to machine learning and you need to dig something in depth, you will have to find a real analyst-mathematician, preferably with a scientific degree and knowledge of python. I am serious - for machine learning, without a core education, you need to study hard for several hours a day (probability theory, linear algebra, least squares method, mathematical statistics, Bayes theorem and its techniques) for several months, if not years.
Processes or technology?
You can often find long-lived and large software projects written in seemingly "unsuitable" programming languages or sets of libraries. For example, often there are huge control scripts on bash or massive scientific libraries and frameworks for accurate calculations on ... python with razdolbaysky duck typing . The secret is in rigorous processes and practices. Using:
- pre-design, stress testing and risk analysis
- automated unit and integration testing (preferably 100% coverage)
- uniform coding standards
- good documentation of system code and components
- maximum transparent communication within the team
- monitoring and analytics of server environment
You can create a software solution, in principle, on any technology that will live for years and delight customers. Conversely, working with inflated "pontes" with black boxes, without delving into the details of technology, encouraging negligence and personal interests instead of command ones, you can try to launch the project for months, correcting one mistake and generating dozens - unfortunately, this situation is quite common.
Results
Let's summarize and prioritize:
- It is increasingly possible to observe the situation when it is the launch speed of a software project that is the decisive success factor. It is long and hard to do something unnecessary worse than to quickly release a solution that is useful to customers and collect feedback for the next breakthrough.
- Quick start is possible only when using the maximum of ready-made components, frameworks and libraries with an adequate support period (taking into account the expected lifetime of your web system)
- Unfortunately, algorithmic and architectural knowledge is in demand less and less. More often, experience in solving similar problems and the practice of using complex tools and the strengths of cloud providers are welcome.
- You do not need to limit yourself and do everything on one technology and only one BEST programming language - in the hands of a child, a child has effectively fired a martial arts samurai who has fought a samurai all conscious and unconscious life. Perfectionism and idealization kill software systems. Choose the right weapon for each task in the project. Take ready and concentrate remaining efforts on non-standard.
- Selected technologies should be simple, understandable and accessible to the majority in the team. Look at what a similar project is basically written on and take this technology stack to yourself. Do not automate hosting with Haskel and do not write websites in C ++ :-) If your project “takes off” and starts to develop, only then you can consider rewriting its small parts on more complex technologies and programming languages. But usually, software projects either do not reach this stage, or reach after a few years.
- The framework allows speeding up the launch of a software project by orders of magnitude. Be sure to check the availability of good documentation and specify the period of support for the framework, so that there is no trouble with a complete rewriting in 2-3 years. This is a very frequent case from our customers.
- Do not believe in the ability of the team to learn - there are too many distractions in life and the situation is only getting worse. Carefully study the resume, check the certificates and exams passed and try to focus as much as possible on the practical result. Put the practice above the theory, at least for the time of creating the first version of the working solution.
- The technology and programming language (s) are not determining success factors. Focus on building the most necessary processes and introducing key development and design practices. The correct processes, as a rule, are guaranteed to lead to the "correct" result. Contacting "sectarian clean technology" or "hitting the science", you will spend a lot of time in vain and you can still not run anything. Remember, 2-3 months is the maximum for launching the next iteration of a software project into battle.
Good luck to everyone, successful projects, adequate teams and a great mood!
Source: https://habr.com/ru/post/436460/