In short, the processor has a list of commands that can only be executed at high priority levels. And the user task is started by an OS at a low priority level. So if you try to get out of your memory directly from the task or when you try to go directly to the ports directly from the task, an exception will be thrown and the control will be transferred back to the operating system.
UPD1:
And yes, an attempt to get into someone else's memory is not limited by the command’s priority, but by the so-called Memory Management Unit (MMU). Each task is allocated a piece of memory and the boundaries of this memory are specified in the MMU. With each access to the memory, the MMU hardware controls these boundaries. When you try to write / read outside the selected bounds, the MMU generates an exception and the control will go back to the OS. And about the commands to access the ports - they can only be run from high-priority tasks.
UPD2:
And if the program needs to read the file, how does this happen? Describe the process
Since the task cannot directly access the ports of the disk controller, it is forced to perform a so-called system request to the operating system. In the case of reading a file, the program accesses the file system driver (which then accesses the disk driver). The disk driver operates at the kernel priority level (or at the driver priority level) and is allowed to access the disk controller's registers. The program also arrives (performs a system request to the OS, the so-called system call) when it needs to display something on the screen or access the network or the USB port. Since control is transferred to the operating system, the operating system can analyze all privilege levels of this program and execute the request or refuse to execute the request if there are not enough privileges or, say, the system file is unavailable at the level of this user. Also, the operating system organizes the separation of data access, for example, if several programs try to simultaneously write to the same file.
UPD3:
And yes, the priority levels in question are not the priority levels of processes and threads in the operating system. These are hardware priority levels in the processor. Different architectures do it in different ways. Sometimes these are two levels - the supervisor and the user / users. Sometimes there are three levels - supervisor, drivers, and user / users. And the priority levels of processes and threads in the OSes exist separately from the hardware priorities that exist in the processor.
UPD4:
Advise literature or other sources where you can learn more about it.
Honestly, I do not know where there are good popular reviews on this issue. But somewhere they certainly have try googling. I myself pretty much programmed controllers at a low level without any operating system, right on the "bare" hardware. To do this, you had to read datasheets for processors. From there I learned this science. However, I cannot recommend this path to others, because a typical datasheet for a processor contains 500-1000-1500 pages in English.
In fact, for an ordinary programmer who does not write drivers or does not port Unix to a new processor, this whole science is rather useless. Having an idea of the internal structure of the system is, of course, useful for general development. But for the actual work of the application programmer, it is enough to know that there is a dedicated code memory, data memory, a heap, and there are requests to the operating system, which are processed and return the result and the error code.
UPD5:
I see the answer is gaining popularity. Well, add another five kopecks. There are processors in which there is no hardware support for memory management (that is, there is no MMU) and there is no separation of operating modes into the supervisor and user modes. These processors are designed to work in the so-called embedded systems (embedded system), in which there is no disk support, there is no hardware support for swapping (which is traditionally performed by the same MMU), there is no loading of new tasks during operation, and all tasks initially work in ROM. These are, for example, Microchip PIC processors, Analog Device signal processors and many others. But there are processors oriented to work under general-purpose OS, such as Windows and Linux with full support for multitasking and multi-use. In these processors, there must be both the MMU and the division into the supervisor / user mode. These processors are primarily x86 from Intel, ARM, beloved by smartphone makers, and many more processors of varying degrees of fame and commercial success.
UPD6:
"In the case of reading a file, the program accesses the disk driver." - there are two mistakes. First, the appeal first goes to the FS driver. Secondly, the program does not choose which driver it accesses ... - Pavel Mayorov 2 hours ago
@Pavel Mayorov There are no errors here because:
In this case, it doesn’t matter that the file system driver works out first, since the file system driver does not access the disk controller. And the question was exactly how the access to the disk controller is divided.
When a program performs a system call, it then chooses which OS driver redirects this call. That is, by choosing the type of system request (to disk, to screen, to USB port), the program thereby selects the final controller driver to be called. Of course, before the controller driver runs, a lot of kilobytes of the OS code itself will be executed, including (in the case of disk access) control of file access rights, control of file occupancy by another program, possibly a compression / decompression code in the case of an archived volume, the network subsystem code in the case of a remote disk and a bunch of miracles including the virtual machine code (if the program is in the virtual machine). But the question was not about all these wonders, but about how, from the hardware point of view, in general, access from many programs to hardware resources that exist in the system exist in a single copy is organized in the OS. And what hardware OS support exists in modern processors.
UPD7:
Then asked to indicate the literature on the subject matter. I remembered one name:
“Micro-processor 80286 architecture” by S. P. Morse, D. D. Albert.
The book is an ancient, 1990 edition. And the processor is ancient. But there it is just described how hardware support for a normal OS is organized in 80286, and not MSDOS, which, as we know, worked in the real mode of the Intel processor. I had this book in paper form. Of course, the 80286 is a bit different from what Intel did in 80386 and beyond, but at least something. And the book, of course, sloppy written. Or maybe this is a translation. But as they say, bezrybe and cancer fish. It is also important to understand that the x86 architecture is not ideal, there are other approaches.
UPD8:
Summing up, we can say that processors that are designed to work under multi-tasking and multi-user disk operating systems should have some peripherals to support such work, in particular, there should be a MMU. Also, such processors must have different hardware priority levels for command execution and some commands must be disabled for execution at the priority level at which user tasks are run. That is, such processors must have what is called hardware support for the OS. By analogy with the fact that, for example, signal processors must have hardware support for floating point operations.
UPD9:
In the book "The architecture of the micro-processor 80286" you need to look at chapter 5. It describes what kind of processor has the 80286 hardware to support the OS.
