How to establish quality control of medical services: the experience of the DOC + team
In the past , we talked about how electronic medical records (EHRs) simplify the work of doctors and the lives of patients. Thanks to the EHR, the patient can count on the joint help of all the specialists to whom he has applied, and the doctor is able to quickly get a complete picture of the patient's condition.
EHR has another advantage: they help to improve the quality of medical care. This will be discussed today. We describe how the quality control of medical services in DOC + is implemented and what role electronic cards play in this process.
“ Just ask : about hospitals, about health, about children”
In “classic” hospitals, the quality of medical services is monitored by means of bypassing territories and wards and collecting medical consultations, as well as checking medical records (including insurance companies check them). For example, in the United States , 74% of doctors participate in the assessment of polyclinic work processes, and in Sweden 93%.
We are a mobile clinic, so our employees will find out the opinion of clients not by going through the wards, but by telephone. Our call center now makes two calls.
If a person has been prescribed treatment, we call him 40 minutes after the call and ask to rate the service on a five-point scale. Also, operators offer to use the services of partner clinics, if the doctor recommended a visit to a specialist or an additional examination: MRI, X-ray, etc.
The second call is made after 3 days: we call the client back and ask about his state of health. Our electronic system automatically sorts the dialing procedure using a special formula to simplify the work of the operator - first in the list are those who need to call first.
To get more feedback on the service, we communicate with patients in chat rooms, conduct surveys and carry out test calls when DOC + employees call the doctor as a client. However, all these are very labor-intensive processes, which are not devoid of subjectivism. They do not allow quality control to be carried out systematically and in full.
The only mass source of information about the medical services provided are electronic cards. In them we can see all the patient’s complaints and the results of the examination and evaluate the doctor’s prescriptions on their basis.
At the start of our project, we organized a so-called e-card exchange for checking the EHR, in which the peer-to-peer community of doctors-employees evaluated assignments and put corresponding points.
To balance the system, two doctors checked each card. If the points marked by them did not differ much from each other, the card was considered to have been successfully verified. Otherwise, the map was sent for additional testing to senior experts, - these experts we had deputy chief physician for therapy and pediatrics - who passed the final verdict on controversial points.
We continued to test and develop the system of checking medical records and improve the work of the exchange. To speed up the process of evaluating an EHR and improve its quality (and in the future, reduce the number of checks per card), we organized a special department whose task is to check cards:
However, in contrast to the previous model, now the medical commission (VC) was able to develop standards. In solving the problem that caused the controversy, a precedent was created, by which the doctors were guided further.
Over time, this approach has allowed us to accumulate experience and reduce the number of situations where the assessments of the two experts differed significantly, to 5% (and below). At this point, we moved on to the “one card - one expert” test model in order to reduce the load on the VC. However, ~ 5% of EHRs are still being checked by two experts. This allows us to make sure that the members of the commission give marks objectively and control the stability of the whole system.
The commission receives unidentified information about the patient to check: a history of the disease, indicators during the examination, a diagnosis, data on which drugs were prescribed by the doctor (whose name is also hidden), which tests he recommended to take and which narrow specialists to contact.
Experts check the EHR for accuracy and correctness of filling: the completeness of the patient's examination and history taking, the compliance of the diagnosis with the marked symptoms, etc. are assessed. The test is conducted according to 20 criteria according to the instructions given to the experts. Each criterion is evaluated on a three-point scale: 0 - “does not meet the standards”; 0.5 - "partially corresponds"; 1 - "meets the standards."
We use a three-point system, because there are still controversial situations. For such situations the option “partially corresponds” is left. However, over time, we plan to fully formalize all processes and move to a binary assessment system for each criterion - the EHR will either meet the requirements or not.
In this case there will not be any intermediate options, and the selection will become more stringent: either the doctor was right or not. All the latest recommendations of the Ministry of Health on the quality assessment offer exactly the binary system.
Such an assessment system allows us to improve the quality of services with each completed and verified EHR. All points that the expert has set are stored in the database. This information allows you to identify system deficiencies, analyze problems with the diagnosis and prescription of treatment. From the statistics, we learn how correctly doctors prescribe drugs so that you can immediately make adjustments to their work.
We see problems that most doctors face, as well as areas that cause difficulties for an individual doctor. This allows us to take action quickly: to hold seminars and lectures on the necessary topics or to ask some specialists to “pull up” others.
All this work helps to form treatment standards based on best practices and evidence-based medicine and “arm” all doctors with them.
Each specialist sees in the electronic system his current rating, compiled based on the results of the evaluation of all his EHRs for the current month, and a detailed transcript of all the card verification criteria with comments from a member of the medical commission. This allows you to remove most of the points at issue - the doctor reads the expert’s comment and understands his mistake.
Also, the doctor has the opportunity to come to the VC meeting and discuss all points of interest. In the near future we will start holding such meetings every month. These meetings will be integrated into the general training schedule of our doctors.
The examination is entirely under the authority of the medical board, which looks at the cards and analyzes them independently. However, we are working on automating this process to help experts with evaluating the EHR.
How it works now: a special card selection algorithm has already been created, which forms a stack for checking from 20% of the EHR and takes into account various factors: how many cards of this doctor have already been tested whether this card is “specific”. For example, we often check the EHR, for which sick leave lists were drawn up, due to the requirements of the FSS.
In the near future, the algorithm will begin to take into account the complexity of the nosology (the less often the disease occurs, the higher the probability of verification), as well as the current doctor's rating (the more experienced the doctor, the lower the probability of selecting his patients' cards).
The second algorithm of the system selects the cards from the formed queue and distributes them among the experts.
How it will work in the future: we try to make the card selection algorithms more intelligent. The idea is to use a machine learning technician to train the algorithm to predict the score that the commission will put.
The training sample for the system is formed on the basis of accumulated data on maps and points. On this data, 20 models will be trained - one for each criterion - which will begin to predict the probability of not receiving the highest score for a given criterion.
The scores obtained by these algorithms are combined into one total, showing the overall probability that the card will not receive the highest score. Thus, the set of EHRs to be checked can be ordered according to the probability estimate. This will make it possible to identify "good" and "bad" EHRs and assign higher priority to cards requiring verification by the medical board.
For processing unstructured data, an NLP preprocessor that was developed earlier for the needs of another internal project is used. The preprocessor is able to extract from the complaints of patients and history, which the doctor writes in free form, various symptoms and their characteristics. Using this data will help build better models (how our natural language processing system is structured, we will explain in one of the following materials).
This form of prioritization will reduce the burden on doctors and will allow to evaluate a larger number of cards. In addition, it will increase the speed of testing, since we will detect “erroneous” EHR right at the moment of filling (and not after three days when the expert checks them), and the doctor, while being with the patient, will receive hints from the system.
We at DOC + create a service for the mass market, so we must show consistently high quality in the face of growing audiences - this is a key aspect of our strategy. At the very beginning of the journey, we provided medical aid only by the forces of our doctors. This approach allowed us to control the quality of services provided, in order to expand their range, having an understanding of how the market works.
We provided more than 100 thousand medical services and recruited 350 doctors of twenty different specialties. For example, in December 2016, ENTs and neurologists began to work with DOC +, except for general practitioners and pediatricians. It also became possible to call a nurse for procedures and import laboratory test results from partner laboratories, and inspection data from partner clinics.
At the same time, our semi-automated quality control system for the work of doctors allowed DOC + to become the owner of a unique expertise for the entire market, which opened up opportunities for the development of new products. It allows us to maintain a consistently high quality of telemedicine care for 10 medical specialties in DOC + Online. And invisible integrations make it possible to control the level of medical services that affiliate clinics provide to our patients.
We are convinced that such process automation systems operating at the interface of IT and medicine will help reduce the likelihood of medical errors. And the knowledge base that we form will open opportunities for the provision of personalized assistance to customers.
Additional reading: materials from our blog “ Just ask ”:
EHR has another advantage: they help to improve the quality of medical care. This will be discussed today. We describe how the quality control of medical services in DOC + is implemented and what role electronic cards play in this process.
“ Just ask : about hospitals, about health, about children”
The first stage: the organization of the quality control system
In “classic” hospitals, the quality of medical services is monitored by means of bypassing territories and wards and collecting medical consultations, as well as checking medical records (including insurance companies check them). For example, in the United States , 74% of doctors participate in the assessment of polyclinic work processes, and in Sweden 93%.
We are a mobile clinic, so our employees will find out the opinion of clients not by going through the wards, but by telephone. Our call center now makes two calls.
If a person has been prescribed treatment, we call him 40 minutes after the call and ask to rate the service on a five-point scale. Also, operators offer to use the services of partner clinics, if the doctor recommended a visit to a specialist or an additional examination: MRI, X-ray, etc.
The second call is made after 3 days: we call the client back and ask about his state of health. Our electronic system automatically sorts the dialing procedure using a special formula to simplify the work of the operator - first in the list are those who need to call first.
To get more feedback on the service, we communicate with patients in chat rooms, conduct surveys and carry out test calls when DOC + employees call the doctor as a client. However, all these are very labor-intensive processes, which are not devoid of subjectivism. They do not allow quality control to be carried out systematically and in full.
The only mass source of information about the medical services provided are electronic cards. In them we can see all the patient’s complaints and the results of the examination and evaluate the doctor’s prescriptions on their basis.
At the start of our project, we organized a so-called e-card exchange for checking the EHR, in which the peer-to-peer community of doctors-employees evaluated assignments and put corresponding points.
To balance the system, two doctors checked each card. If the points marked by them did not differ much from each other, the card was considered to have been successfully verified. Otherwise, the map was sent for additional testing to senior experts, - these experts we had deputy chief physician for therapy and pediatrics - who passed the final verdict on controversial points.
The second stage: the creation of a special commission
We continued to test and develop the system of checking medical records and improve the work of the exchange. To speed up the process of evaluating an EHR and improve its quality (and in the future, reduce the number of checks per card), we organized a special department whose task is to check cards:
“We decided to replace the company of“ medical recruits ”with a squad of ten“ special forces ”, - Eldar Garifullin, deputy operational director for analytics and medical projects, draws an analogy, created a specialized medical commission that is responsible for the development of internal standards: how to treat this or other nosology ".At first, two specialists of the commission also checked one map, because disagreements persisted even among the best and most experienced doctors. Every time such a case arose, an arbitrator came in the person of the deputy head doctor and decided the points at issue.
However, in contrast to the previous model, now the medical commission (VC) was able to develop standards. In solving the problem that caused the controversy, a precedent was created, by which the doctors were guided further.
Over time, this approach has allowed us to accumulate experience and reduce the number of situations where the assessments of the two experts differed significantly, to 5% (and below). At this point, we moved on to the “one card - one expert” test model in order to reduce the load on the VC. However, ~ 5% of EHRs are still being checked by two experts. This allows us to make sure that the members of the commission give marks objectively and control the stability of the whole system.
How is the assessment done?
The commission receives unidentified information about the patient to check: a history of the disease, indicators during the examination, a diagnosis, data on which drugs were prescribed by the doctor (whose name is also hidden), which tests he recommended to take and which narrow specialists to contact.
Experts check the EHR for accuracy and correctness of filling: the completeness of the patient's examination and history taking, the compliance of the diagnosis with the marked symptoms, etc. are assessed. The test is conducted according to 20 criteria according to the instructions given to the experts. Each criterion is evaluated on a three-point scale: 0 - “does not meet the standards”; 0.5 - "partially corresponds"; 1 - "meets the standards."
We use a three-point system, because there are still controversial situations. For such situations the option “partially corresponds” is left. However, over time, we plan to fully formalize all processes and move to a binary assessment system for each criterion - the EHR will either meet the requirements or not.
In this case there will not be any intermediate options, and the selection will become more stringent: either the doctor was right or not. All the latest recommendations of the Ministry of Health on the quality assessment offer exactly the binary system.
Why do you need it
Such an assessment system allows us to improve the quality of services with each completed and verified EHR. All points that the expert has set are stored in the database. This information allows you to identify system deficiencies, analyze problems with the diagnosis and prescription of treatment. From the statistics, we learn how correctly doctors prescribe drugs so that you can immediately make adjustments to their work.
We see problems that most doctors face, as well as areas that cause difficulties for an individual doctor. This allows us to take action quickly: to hold seminars and lectures on the necessary topics or to ask some specialists to “pull up” others.
All this work helps to form treatment standards based on best practices and evidence-based medicine and “arm” all doctors with them.
Each specialist sees in the electronic system his current rating, compiled based on the results of the evaluation of all his EHRs for the current month, and a detailed transcript of all the card verification criteria with comments from a member of the medical commission. This allows you to remove most of the points at issue - the doctor reads the expert’s comment and understands his mistake.
Also, the doctor has the opportunity to come to the VC meeting and discuss all points of interest. In the near future we will start holding such meetings every month. These meetings will be integrated into the general training schedule of our doctors.
Sample and distribution of EHR
The examination is entirely under the authority of the medical board, which looks at the cards and analyzes them independently. However, we are working on automating this process to help experts with evaluating the EHR.
How it works now: a special card selection algorithm has already been created, which forms a stack for checking from 20% of the EHR and takes into account various factors: how many cards of this doctor have already been tested whether this card is “specific”. For example, we often check the EHR, for which sick leave lists were drawn up, due to the requirements of the FSS.
In the near future, the algorithm will begin to take into account the complexity of the nosology (the less often the disease occurs, the higher the probability of verification), as well as the current doctor's rating (the more experienced the doctor, the lower the probability of selecting his patients' cards).
The second algorithm of the system selects the cards from the formed queue and distributes them among the experts.
How it will work in the future: we try to make the card selection algorithms more intelligent. The idea is to use a machine learning technician to train the algorithm to predict the score that the commission will put.
The training sample for the system is formed on the basis of accumulated data on maps and points. On this data, 20 models will be trained - one for each criterion - which will begin to predict the probability of not receiving the highest score for a given criterion.
The scores obtained by these algorithms are combined into one total, showing the overall probability that the card will not receive the highest score. Thus, the set of EHRs to be checked can be ordered according to the probability estimate. This will make it possible to identify "good" and "bad" EHRs and assign higher priority to cards requiring verification by the medical board.
“Such a preventive algorithm will allow an automatic preliminary assessment of 100% of the cards at the time of filling them and send for manual verification only 20% of the most“ problematic ”,” says Ilya Larchenko, director of innovation at DOC +.Now work in this direction is the department of processing and analyzing data DOC +. Its task is to find the signs that affect the assessment for each of the criteria using machine learning and build a mathematical model that evaluates the map according to these signs.
For processing unstructured data, an NLP preprocessor that was developed earlier for the needs of another internal project is used. The preprocessor is able to extract from the complaints of patients and history, which the doctor writes in free form, various symptoms and their characteristics. Using this data will help build better models (how our natural language processing system is structured, we will explain in one of the following materials).
This form of prioritization will reduce the burden on doctors and will allow to evaluate a larger number of cards. In addition, it will increase the speed of testing, since we will detect “erroneous” EHR right at the moment of filling (and not after three days when the expert checks them), and the doctor, while being with the patient, will receive hints from the system.
We at DOC + create a service for the mass market, so we must show consistently high quality in the face of growing audiences - this is a key aspect of our strategy. At the very beginning of the journey, we provided medical aid only by the forces of our doctors. This approach allowed us to control the quality of services provided, in order to expand their range, having an understanding of how the market works.
We provided more than 100 thousand medical services and recruited 350 doctors of twenty different specialties. For example, in December 2016, ENTs and neurologists began to work with DOC +, except for general practitioners and pediatricians. It also became possible to call a nurse for procedures and import laboratory test results from partner laboratories, and inspection data from partner clinics.
At the same time, our semi-automated quality control system for the work of doctors allowed DOC + to become the owner of a unique expertise for the entire market, which opened up opportunities for the development of new products. It allows us to maintain a consistently high quality of telemedicine care for 10 medical specialties in DOC + Online. And invisible integrations make it possible to control the level of medical services that affiliate clinics provide to our patients.
We are convinced that such process automation systems operating at the interface of IT and medicine will help reduce the likelihood of medical errors. And the knowledge base that we form will open opportunities for the provision of personalized assistance to customers.
Additional reading: materials from our blog “ Just ask ”:
- How and when to make out a hospital
- How to check the health of parents
- What happens to the body in winter
Source: https://habr.com/ru/post/409673/