<> Types of data migration
With the development of business , Storage will also need to be migrated frequently . The following scenarios are frequently encountered in our development process
* business , The team is expanding rapidly , It is necessary to split microservices at an appropriate time , Need independent database , Migrate data from the source database to the new database
* The number of records in a single table is relatively large , Sub database and sub table are needed . The data of the old table needs to be migrated to the new sub table .
* Wrong storage selection , For example, the mutual migration of relational databases , PG,
MySQL,Oracle Mutual transfer of .NoSQL Of Mongo,Cassandra,Hbase Mutual transfer of .
* Relocation of computer room , Self built to cloud room migration
All of these scenarios require data migration , Although the details of the scheme are different , But there will be something in common .
<> Scheme of data migration
Data migration is simply about moving data from one place to another .
Because our data is not static , So we can't just write one job Just move . There are some migration standards that need to be ensured
Record cannot be lost after data migration , The data of a single record cannot be missing fields .
Data is constantly being written , Cannot be used to block writes , Data is not allowed to be written , The availability of business writing needs to be guaranteed .
The migration process can be interrupted , Roll back
This is very demanding , It's a strategy to ensure data integrity . Problems were found at all stages of migrating data , Can be rolled back to the original library , Ensure the normal operation of business .
<> Migration plan
In order to meet the above requirements , Generally, double writing strategy is adopted . That is to write two copies , Old writing , Write new things, too .
* Convergence reading and writing
More access to reading and writing , The more places need to switch in the future , The easier it is to make mistakes , So try to converge all the read and write entries to one place
* Double writing
Write incremental data to both storage systems at the same time . Make sure the new write code is OK . Double writing is subject to the old one , Old write success means successful operation , Write a new failure. You need to log the failure , Why analysis failed , Make corrections and compensations
* Migration of old stock data
The old stock data migration is through traversal id, Write to new storage . There are many specific plans . You can use the synchronization tool , such as binlog +flink To deal with it . Less data on the direct traversal line .
* data verification
Data consistency checking is the top priority , Ensure the number of records for both sides of the data , Data integrity of single record . If the amount of data is not large , Generally, it is full calibration . A lot of data , It can be checked by sampling .
* Switch new read
After data verification , You can switch to a new read , In case there is a problem , Can switch to the old read . Troubleshooting , Do it again .
* Stop double writing
It runs safely and smoothly in the new storage N Days later , You can stop the old ones , The entire migration process is complete .
<> matters needing attention
* For back end services , Storage is the cornerstone , It's the top priority . Stability requirements are the highest . Make sure that the data is migrated smoothly , No business perception .
At the same time, storage is stateful , The migration is difficult , Developers need to be forward-looking , Try to be careful in the selection , Choose the right database , Avoid database migration . When potential problems are found in database selection , It needs to be a decision , Early migration . Don't think the probability of problems is small , It's delayed . Otherwise, once there is a problem , It's a major failure , The damage is incalculable .