This is a very practical problem in distributed system , It's not very detailed , Summarize .
1, Cleft brain and pseudodeath
1.1 Cleft brain
Official definition ： When different parts of a cluster think they are active at the same time , We can call this phenomenon cleft brain syndrome . In a popular way , Like when you cluster
There are two nodes in it , They all know it's here cluster We need to elect one master. So when there's no problem with the communication between them , There will be a consensus , Choose one of them as
master. But if there's something wrong with their communication , Then both nodes will feel that there is no master, So everyone elected themselves master. therefore cluster
There will be two in it master. give an example ：
UserA and UserB Register your own information in RouterA and RouterB in .RouterA and RouterB Use data synchronization （2PC）, To synchronize information . So when UserA Want to UserB When sending a message , Need now RouterA Found in UserA reach UserB Message routing path for , Then it is delivered to the corresponding path for routing .
When a brain crack occurs , Quite RouterA and RouterB Direct contact lost ,RouterA Think it's the only one in the whole system Router,RouterB That's what I think . So it's equivalent to RouterA Not in UserB Information about ,RouterB Not in UserA The information of , here UserA Resend message to UserB When ,RouterA Will think UserB It's offline , The information is then persisted offline , Is the route of the whole network in disorder .
about Zookeeper There is a very important question , Is to judge a node's death according to what kind of situation down It's gone .
In the distributed system, these are judged by the monitor , But it is difficult for the monitor to determine the state of other nodes , The only reliable way is to have a heartbeat ,Zookeeper The heartbeat is also used to determine whether the client is still alive , But using heartbeat mechanism to judge the survival state of nodes also brings about the problem of pseudo death .
1.2 Feign death
ZooKeeper Each node attempts to register a symbol master Temporary nodes for , Others who did not register successfully became slaver, And through watch Mechanism monitoring master Temporary nodes created ,Zookeeper Determined by internal heartbeat mechanism master Status of , once master There was an accident Zookeeper We'll be able to get to know and inform others soon slaver, other slaver Respond later . This completes a switch .
This mode is also a common one , Most of them are realized in this way , But there's a big problem , If you don't notice, it will lead to brain crack in the system for a short time , Because the heartbeat timeout might be master Hang up , But it could be master,zookeeper There's a problem with the network between , It can also lead to . This is a case of suspended animation ,master Not dead , But with ZooKeeper Network problems between Zookeeper Think it's hung up and inform other nodes to switch , such slaver One of them has become master, But the original master Not dead , At this time client Also obtained master Switched messages , But there will still be some delays ,zookeeper Need communication need a notification , At this point, the whole system is in chaos, and there may be a part of it client Notified to connect to new master It's up , yes , we have client Still connected to the old master If there are two at the same time client Need to master The same data of is updated and just the two client Now they are connected to the old and the new master upper , There will be serious problems .
Feign death ： Due to heartbeat timeout （ Caused by network ） think master Dead , But in fact master Still alive .
Cleft brain ： Because feigning death will launch a new one master election , Elect a new one master, But the old master The Internet is connected again , That led to two master
, Some clients connect to the old master Some clients link to the new master.
2,Zookeeper Solutions for
To solve Split-Brain Problems of , Generally 3 Ways :
* Quorums（ˈkwôrəm quorum ） ： such as 3 Cluster of nodes ,Quorums = 2,
That is to say, clusters can tolerate 1 Node failures , At this time, we can elect 1 individual lead, Clusters are also available . such as 4 Cluster of nodes , its Quorums =
3,Quorums To exceed 3, Is the tolerance equivalent to cluster or 1, If 2 Node failures , So the whole cluster is still invalid
* Redundant communications： Redundant communication mode , Multiple communication modes are adopted in the cluster , To prevent the failure of a communication mode that makes the nodes in the cluster unable to communicate .
* Fencing, How to share resources ： For example, if you can see the shared resources, they are represented in the cluster , The only way to get a lock on a shared resource is Leader, Can't see shared resources , Not in a cluster .
ZooKeeper By default Quorums This way , That is, only more than half of the nodes in the cluster can vote Leader. In this way, we can ensure that leader Uniqueness of , Or choose the only one leader, Or the election fails . stay ZooKeeper in Quorums Yes 2 Roles ：
The minimum number of nodes in the cluster is used for election Leader Ensure cluster availability ： Notify the client that the data has been saved safely before the minimum number of nodes in the cluster has saved the data . Once these nodes have saved the data , The client will be notified that it has been saved safely , You can continue with other tasks . And the rest of the nodes in the cluster will eventually save the data .
Suppose a leader Feign death , The rest followers A new one was elected leader. At this time , old leader Resurrect and still think of yourself as leader, At this time it's going to followers Making a write request will also be rejected . Because whenever new leader When generated , Will generate a epoch, this epoch It's incremental ,followers If new leader existence , Know it epoch, Will refuse epoch Less than current leader
epoch All requests for . Do you have any follower Don't know what's new leader There is , be on the cards , But certainly not most , Otherwise new leader Unable to generate .Zookeeper The writing of quorum mechanism , therefore , Writing without most support is invalid , used leader Even if they think they are leader, It still doesn't work .
To summarize, it's , adopt Quorums Mechanism to prevent brain crack and pseudodeath , When leader After hanging up , You can re elect a new one leader Nodes make the whole cluster agree ; When there's a fake death , adopt epoch Size to reject the old leader Requests initiated , I've talked about it before , This time , The old one who restored communication leader Node will enter recovery mode , And new leader Node data synchronization ,perfect.