We are developing a team project in Python using Django. Code storage and versioning support via gitlab. When pushing a new code, the upload to the server is in progress. Database structure changes due to migrate in Django. How to deal with the data in the database? For example, changes are underway: the column has been deleted, new records have been filled, records (lines) have been deleted. At some point, an Exeption crashed. The structure can be rolled back through all the same migrations, and how to roll back data changes ?? (to return the server to its previous operating state) The dimensions of the database are tens of terabytes, for this reason "make a backup copy before the changes and, if it has flown off, substitute a copy" is not an option. What are the ideas or proven approaches, standard services?

    2 answers 2

    Otherwise, approach the design of database changes. Spread over time change code and database. First, change the code so that it works simultaneously on the current and on the new database schemas. Test this code and roll it out into production. If any errors occur at this stage, simply roll back the code. When you make sure that the new code is working and there are no errors, roll out the new database scheme.

      Django has DataMigration . In them (as well as in usual migrations) there is a function of rollback of changes. In the Jang documentation, unfortunately, it is only mentioned:

      You can not have to go back. If this callable is omitted, migrating backwards will raise an exception.

      That is, there is some second argument that can roll back your changes back. Let's add an example from the documentation:

      def combine_names(apps, schema_editor): # какая-Ρ‚ΠΎ функция мигрирования Π²ΠΏΠ΅Ρ€Ρ‘Π΄ def combine_names_backwards(apps, schema_editor): # какая-Ρ‚ΠΎ функция мигрирования Π½Π°Π·Π°Π΄ class Migration(migrations.Migration): initial = True dependencies = [ ('yourappname', '0001_initial'), ] operations = [ migrations.RunPython(combine_names, combine_names_backwards), ]

      A bit of help - by default nothing is added there, respectively, you get a migration that can be done only in one direction. If in the opposite direction you do not need to do anything, then you should use migrations.RunPython.noop

      So, we figured it out, now let's consider various situations when it may be useful to us:

      1. Suppose your function migrates data in such a way that it can be returned back by some algorithm. Then you write backwards datamigration and rejoice

      2. Data is deleted.

        Well, there's nothing to be done, the data is lost. There is an option not to delete this column, but to leave it, while changing it to nullable (so that the objects created afterwards do not affect it, the classic blank=True, null=True ).

      3. Delete, but store the deleted data in the json field. I see you have mysql, so this solution is postgres-only. Not sure if there is something like this in mysql

      4. Try to merge the deleted fields into the fixture file, then delete these columns.

        Warning This method has not been tested in personal practice, but in theory it should work

        It should be noted that the standard dumpdata command does not know this, but it seems that this addition is able. Accordingly, when you roll back download

      • Not exactly what you need. This solution involves self-writing / appending migrations files. I would like to automate the system as much as possible + use the files generated by the system using "python manage.py makemigrations" - StSan
      • @StSan is the standard files generated by manage.py makemigrations --empty yourappname , but they need to be filed. I sometimes have not enough for work and standard jung migrations, I almost have to write from scratch, this is normal here. The Dzhangov people themselves say that they still have a flimsy system, they will still be refined, so I don’t want to, but with migrations this is the only way out - FeroxTL
      • @StSan added another option, but not sure about it - FeroxTL
      • RunPython works inside transactions, if errors occur in it, it will roll back the database to the beginning of the transaction. But Mysql does not support DDL transactions. Requests to change a column, create a table, etc. have to do outside transactions. - Arnial