I wanted to spray last week eBPF It's over , Because the weekend time is too tight , I'm going to put it off for a week , But I made a decision flag, I started with a circle of friends :

ebpf It's like cowhide moss , It's all over the world linux Every corner of the kernel , Each call point looks casual , There is no plan , It's not hard to make people feel as if they need such a call point …


But in fact, if you really try to add one somewhere ebpf When calling a point , I think it's very similar to the process of removing cowhide moss , Modify multiple files scattered in various directories , It has to be recompiled , Probability of failure , And do it again , It's hard to clean it all at once , When you finally succeed , There will be one “ nothing more than this ” I'm sorry …


I used to ebpf It's like spreading cancer cells , The metaphor has no effect on fear , So I switched to psoriasis . There is no such thing ebpf There is not a single call point , unnecessary ebpf It's all over the place , These points continue to increase , Up to now 5.11 kernel ,ebpf It's over thirty , It's still growing without planning …


eBPF It's an innovation , But it's clear that the craze has gone too far ,ebpf Adding call points is too arbitrary , Too business oriented , The cohesiveness of the kernel is damaged , It's not as good as it was netfilter Five of them hook Points and qdisc This well-designed mechanism , There is another problem ,netfilter Five of them hook If it's deployed ebpf spot , In fact, it can solve most of the performance problems , But not until now , It feels like the community has overreacted , Really thoroughly netfilter As a symbol of the old times , While killing the horse , No more wheels

People are here like worms. You fight for me , What you eat is conscience , It's all about ideas .

It's finally the weekend , I couldn't speak at last , I don't even know what to say .

yesterday , I sacrificed my blood , This self sacrifice was a virtue in the ancient Roman Republic , The consul will give himself to God , In exchange for the victory of the war .

To spray eBPF, In the usual work and study , I've accumulated a lot of material ,eBPF Our territory is divided into two parts :

* Network protocol stack function
* trace track
On the Internet , I use the metaphor of psoriasis to illustrate eBPF How bad the picture keeps growing . And in trace Tracking , I want to talk about performance and function .

I would like to use these materials to support some of my own strange views , yesterday , I'm going to use my actual work as my last material , As a result, it just verified my cognitive error , It just shows eBPF As trace How useful tracking tools are !

My story is like this .

In the context of high traffic , Especially if your code uses something similar Bonding,tun/tap,GRE,IPIP Virtual network card , Investigation skb Where was it discovered drop It has always been a very troublesome thing . Even if you already know a particular quintuple , It's not going to be easy .

Grab a bag ? Packet capture is always the first step , But it's just the first step , It can only tell you skb Received or not received , If not received , Further confirmation is needed skb Where did you lose it . Yes, of course , If we can't locate it in the end , Generally, operators will be left with the black hole .


similar wireguard such , All of its complex operations , Including encryption , The distribution logic is included wireguard Design of virtual network card xmit Function , track wg_xmit In addition to asking you to wireguard The code is very familiar , We also need all kinds of wonderful crafts .

Take the following scenario as an example :

I want a real hammer skb From 1 reach 11 Which step was taken by drop Of , How to do it ?


It's not exactly stap Are you in my territory ? oh , I can't say stap, We have a lot of money here bpftrace When there is a choice , besides stap It's not right , If I insist on using it stap, There will be a group of people who suggest that I use it bpftrace, And said stap How bad it is , obsolete , unfriendly .

That's good , Then use it bpftrace, The following script can skb Make the perfect journey trace:
#!/usr/local/bin/bpftrace ⁣ #include <linux/skbuff.h> #include <linux/udp.h> #
include <net/sock.h> ⁣ k:encrypt_packet { $skb = (struct sk_buff *)arg0; //
this skb Of mark need iptables To label a particular Quintet with , however encrypt_packet Here's what you can use mark The last place in the world . //
stay encrypt_packet After successful return ,skb Almost all of the affiliations will be deleted reset, include skb->mark. //
So here we have to use another marker , To ensure that encrypt_packet This feature can be used to track to specific objects later skb. //
because bpftrace You can only read but not write , Here I choose to use it directly skb Your address ! if ($skb->mark == 1234) { printf("encrypt got
%p\n", $skb); @addr = $skb; } } ⁣ //k:send4 k:udp_tunnel_xmit_skb { $daddr =
arg4; $saddr = arg3; $skb = (struct sk_buff *)arg2; //
Except here match Outside the address , Do you want it too match What about the other fields ? after all slub In skb It can be reused . // If mark 1234 Of skb Before that, I was killed drop
& free It's over , It was rebuilt alloc I'll still be here in the future , That's wrong ! // However, due to the controllable flow , And I'm a function, a function trace, The above probability is very low . Craftsmen don't seek perfection ! if (
$skb== @addr) { printf("---- skb:%p daddr:%08x saddr:%08x \n", @addr, $daddr,
$saddr); } } ⁣ k:iptunnel_xmit //k:dev_queue_xmit //k:dev_hard_start_xmit
//k:dev_queue_xmit_nit { $skb = (struct sk_buff *)arg2; // Take the content of the outer protocol header from the bare package . $udph
= (struct udphdr *)($skb->head + $skb->transport_header); $sport = $udph->source
; $dport = $udph->dest; if ($skb == @addr) { $port = (($sport & 0xff00) >>8) | (
($sport & 0xff) << 8); $port2 = (($dport & 0xff00) >>8) | (($dport & 0xff) << 8)
; printf("sport:%d dport:%d\n", $port, $port2); // trace end , Reset global variables . @addr = (
struct sk_buff *)0; } }

ah , I think it's a very smooth script ,skb Before entering wg_xmit Put it in front of you mark, stay wg_xmit In the process of cleaning skb Of mark Save its address before , The address is then tracked skb. But sadly ,skb It was sent out smoothly , I got nothing , But sadly , The inner layer message is at the opposite end wireguard Of wg I didn't catch it on the network card .

Go to the opposite end and do it backwards OK? Thinking is one thing , Landing is another thing .

How can we continue at the opposite end trace this skb What about it ?

If there is no way trace this skb, How can you tell if this message is sent by an intermediate network device drop Was it the other way around wireguard Receiving process drop What happened ? Because the sender can get the inner and outer specific quintuples , At the receiving end, the outer quintuple is used to remove the noise match Of course, the outer protocol header is a correct idea , The problem is that if the quintuple of the outer tunnel is reused by large traffic , How are you going to be there skb Matching inner quintuple before decryption , It's too much traffic , Just like a lot of packet capture can not be carried out due to the large traffic , The information you want will be almost drowned in an instant !

What I want to know is ,bpftrace How to do it . If we can't solve this problem quickly and conveniently , I have a good reason to use the things of the old times .

The problem is bpftrace I'm not allowed to modify it skb ah ! Now? , I decided to throw it away bpftrace, use stap To do the right thing .


All I need to do is , Mark a specific packet , The tag must be recognizable at the opposite end . I decided to use the innocuous IP head TTL field , use stap It's very easy to do it . By the way , I'll do it for you skb hit mark It also works stap Do it , So I deleted it iptables rule :
#!/usr/local/bin/stap -g %{ #include <linux/skbuff.h> #include <linux/ip.h> #
include <linux/tcp.h> #include <net/addrconf.h> struct sk_buff *tmp = NULL; %}
functiongetinfo(pskb:long) %{ struct sk_buff *skb = (struct sk_buff *)
STAP_ARG_pskb; struct iphdr *hdr = ip_hdr(skb); const struct skb_shared_info *
shinfo= skb_shinfo(skb); struct dst_entry *dst = skb_dst(skb); if (dst && !
strcmp("wg0", dst->dev->name)) { struct tcphdr *tp = (struct tcphdr *)((char *)
hdr+ sizeof(struct iphdr)); if (ntohs(tp->source) == 443 && ntohs(tp->dest) ==
3663) { STAP_PRINTF("sport:%d dport:%d addr:%p\n", ntohs(tp->source), ntohs(tp->
dest), skb); skb->mark = 1234; tmp = skb; } } %} probe kernel.function(
"ip_forward") { getinfo($skb) } function setttl(pskb:long) %{ struct sk_buff *
skb= (struct sk_buff *)STAP_ARG_pskb; struct iphdr *hdr; if (skb == NULL) return
; hdr = ip_hdr(skb); if (skb == tmp) { hdr->ttl = 120; tmp = NULL; } %} //
This is in confirmation skb It passed smoothly wg server after , In order to wg client The method used by the end trace . //
Since there are no good fields for marking packets , Just pick it up ttl, As a special value , Not necessarily 120,180 better ... probe kernel.function(
"ip_local_out") { setttl($skb) }⁣

Such a script , Directly on the opposite end match Outer quintuple sum TTL It's worth it , We just need to match the outer quintuple at the same time , matching TTL Value greater than 70 Of skb that will do .bpftrace You can only read but not write , To make it possible , I can only use it stap.


That's my attitude , It's not that I don't accept it bpftrace, It's not that I don't accept new things , I just want to say that you can't beat old things to death when you accept new things ! It can't be because eBPF The fashion of choice bpftrace And put stap Put it in the garbage can .



I'm preparing to make complaints about it. , But I'm getting ready to spray bpftrace When you can't do something , I don't have a blood sacrifice !


I can be there wireguard The data receiving end of is used as the data sending end bpftrace come trace This particular skb, After all, I just want to know where it is drop It's over , There is no need to write at all . However, for a pretense of style , It's a must stap To do it and give up bpftrace.


Even though I thought I was stap Skilled workers , But I use almost all of them -g Of guru pattern , It's not because I'm confident , It's because I don't know and I can't remember stap The grammar of . I can almost only C And compilation , I can hardly remember the grammar of any other language , include Bash included …

I'm using it stap come trace Before the function of kernel or module , I always look at its parameter resolution :
stap -L 'module("wireguard").function("wg_allowedips_lookup_src")'⁣
Unfortunately , No parameters can be used , The reason is unknown . So when I want to use its parameters , I can only access the register naked , Like the following :
... probe module("wireguard").function("wg_allowedips_lookup_src") { // because stap
-L Unable to resolve parameter , It can only be used x86_64 The calling rule of the register directly fetches the register if (cmpskb(register("rsi"))) { a = 1; } } probe
module("wireguard").function("wg_allowedips_lookup_src").return { if (a == 1) {
printf("peer returned::%p\n", register("rax")); a = 0; } }⁣ ...
Since it's all used stap It's over , Why not make things easier ? So I began to write with confidence . change skb Memory , Manual modification skb Of data, In the hope of bypass A lot of unnecessary processes are lost .



After I've been playing like this for about a few hours , It's probably the feeling of driving the plane to repair the engine , I'm a little tired , At a precise point in time ,crash or soft
lockup, Completed the blood sacrifice perfectly !

This is an online assignment , The pot is obviously mine .


That doesn't mean I'm not good at it , Before this hard landing , I've been flying for hours after all , But it shows a problem ,bpftrace It's better than that stap good , At least it's safe . And that's exactly what I want to refute , But I proved it .

In terms of stability ,eBPF It's enough that we don't allow it :

* eBPF You are not allowed to write any potentially risky code .
* eBPF You are not allowed to write any complex code .
stap You can have any one in the room while(true) And lock up the system ,bpftrace But not in the middle .



When dealing with similar problems , In fact, I am biased , The reason why I don't like to use tools, especially new tools, is that I'm lazy , I don't like to face and control a lot of irrelevant things , For example, I know there's one dropwatch But they didn't use it , Because it's so complicated , But also to understand the complexity of the command line , It's better than that , It's not as good as me stap
probe kfree_skb then dump_stack What about it .

At first, I was right stap It's also inconsistent , Because it's complex enough , I'd rather write it naked ftrace function , Let's say you manually insert the head of a function 5 Replace bytes with call
stub_handler such . Now, even if I'm right stap I'm familiar with it , I still insist on writing only guru Script for pattern , I'm still too lazy to study stap The grammar of .

Using tools to improve efficiency is for people who are familiar with the tools , For those unfamiliar with the tool , Like me , The time I spend learning how to use this tool will delay me from dealing with real problems .

same , Sharpening your ax will not delay your job of cutting wood , If a worker wants to do a good job, he must sharpen his tools first , I don't agree that this is universal , This kind of script is aimed at people who frequently solve similar problems , What they need is to come up with a paradigm , A hammer can drive nails , If you need to drive nails frequently , Of course you need to buy a hammer , But if you only need to drive a nail once , It may be more convenient to pick up a brick on the side than to buy a hammer .

If every time you play is a new pattern , No tools, of course .

in addition “ on the way ”
My values also contribute to my view of tools . I always feel like I'm on the road , So I hate to be burdened with anything , I'm not going back to carry it , Burden , Remember what you can get everywhere . Hangzhou moves to Shanghai , Since Shanghai can buy Quilts , Why should I send back the quilt , Wouldn't it be better to just throw it away ?

In the stone age , People already have tools that are rich in categories , But for a man who went out of the house in the matriarchal era , The only tool they can take away and voluntarily choose is the bow and arrow , Catapult is a similar long-range attack tool , Maybe even a knife , An axe or something , They are all cumbersome , Grasp the essence of the problem itself , That's enough .

Wet leather shoes in Wenzhou, Zhejiang Province , If it rains, you won't get fat !

Technology
©2019-2020 Toolsou All rights reserved,
Hikvision - Embedded software written test questions C Language application 0 The length of array in memory and structure is 0 In depth analysis data structure --- The preorder of binary tree , Middle order , Subsequent traversal How to do it ipad Transfer of medium and super large files to computer elementui Shuttle box el-transfer Display list content text too long 2019 The 10th Blue Bridge Cup C/C++ A Summary after the National Games ( Beijing Tourism summary )unity Shooting games , Implementation of first person camera python of numpy Module detailed explanation and application case Study notes 【STM32】 Digital steering gear Horizontal and vertical linkage pan tilt Vue Used in Element Open for the first time el-dialog Solution for not getting element