<> Types of data migration

With the development of business , Storage will also need to be migrated frequently . The following scenarios are frequently encountered in our development process

* business , The team is expanding rapidly , It is necessary to split microservices at an appropriate time , Need independent database , Migrate data from the source database to the new database
* The number of records in a single table is relatively large , Sub database and sub table are needed . The data of the old table needs to be migrated to the new sub table .
* Wrong storage selection , For example, the mutual migration of relational databases , PG,
MySQL,Oracle Mutual transfer of .NoSQL Of Mongo,Cassandra,Hbase Mutual transfer of .
* Relocation of computer room , Self built to cloud room migration
All of these scenarios require data migration , Although the details of the scheme are different , But there will be something in common .

<> Scheme of data migration

Data migration is simply about moving data from one place to another .

Because our data is not static , So we can't just write one job Just move . There are some migration standards that need to be ensured

<> standard

Data consistency
Record cannot be lost after data migration , The data of a single record cannot be missing fields .

Non-stop
Data is constantly being written , Cannot be used to block writes , Data is not allowed to be written , The availability of business writing needs to be guaranteed .

The migration process can be interrupted , Roll back
This is very demanding , It's a strategy to ensure data integrity . Problems were found at all stages of migrating data , Can be rolled back to the original library , Ensure the normal operation of business .

<> Migration plan

In order to meet the above requirements , Generally, double writing strategy is adopted . That is to write two copies , Old writing , Write new things, too .

* Convergence reading and writing
More access to reading and writing , The more places need to switch in the future , The easier it is to make mistakes , So try to converge all the read and write entries to one place
* Double writing

Write incremental data to both storage systems at the same time . Make sure the new write code is OK . Double writing is subject to the old one , Old write success means successful operation , Write a new failure. You need to log the failure , Why analysis failed , Make corrections and compensations
* Migration of old stock data
The old stock data migration is through traversal id, Write to new storage . There are many specific plans . You can use the synchronization tool , such as binlog +flink To deal with it . Less data on the direct traversal line .
* data verification
Data consistency checking is the top priority , Ensure the number of records for both sides of the data , Data integrity of single record . If the amount of data is not large , Generally, it is full calibration . A lot of data , It can be checked by sampling .
* Switch new read
After data verification , You can switch to a new read , In case there is a problem , Can switch to the old read . Troubleshooting , Do it again .
* Stop double writing
It runs safely and smoothly in the new storage N Days later , You can stop the old ones , The entire migration process is complete .
<> matters needing attention

* For back end services , Storage is the cornerstone , It's the top priority . Stability requirements are the highest . Make sure that the data is migrated smoothly , No business perception .
*
At the same time, storage is stateful , The migration is difficult , Developers need to be forward-looking , Try to be careful in the selection , Choose the right database , Avoid database migration . When potential problems are found in database selection , It needs to be a decision , Early migration . Don't think the probability of problems is small , It's delayed . Otherwise, once there is a problem , It's a major failure , The damage is incalculable .

Technology
©2019-2020 Toolsou All rights reserved,
Hundreds of millions of locusts rarely collide Locusts want to be self driving Heroes Share has surpassed Ningde Era !LG Chemical confirmation to spin off battery business unit TypeScript Data types in is enough Python Garbage collection and memory leak msf Generate Trojan horse attack android mobile phone Element-UI Implementation of secondary packaging TreeSelect Tree drop-down selection component element-ui+vue-treeselect Verification of drop down box Spring Boot Lesson 16 :SpringBoot Implementation of multithreading with injection class A guess number of small games , use JavaScript realization Unity3D Input Key system