在系統領域中，最具挑戰性的問題通常是組件之間的邊界定位。其中，virtio-net前后端的定界尤為困難。當網絡報文從內核發送到virtio-net后端，或者從virtio-net后端發送到內核時，這一路徑難以進行觀測。一些復雜的網絡抖動問題很可能是由于網卡隊列不正常工作引起的。為了解決這類問題，我們基于eBPF技術擴展了網卡隊列的可觀測能力，使得virtio網卡前后端的定界問題不再困擾。

virtio-net 前后端驅動簡介

virtio-net （后面稱為 virtio 網卡）通常由兩個組件組成：virtio driver（也稱為virtio前端）和virtio device（也稱為virtio后端）。virtio前端運行在客戶機的內核中，而virtio后端可以由宿主機的內核承擔。virtio網卡通常支持多隊列，包括發送隊列和接收隊列。每個隊列通過三個 ring 來實現，即avail ring、used ring和desc ring。現在我們將重點介紹 virtio 網卡前端的報文發送和接收流程，以更好地理解整個工作流程。

virtio 網卡前端發送報文

virto網卡前端發送報文主要流程包括：

a.start_xmit：virtio網卡驅動的報文發送入口函數會首先清理已發送的報文，即通過調用free_old_xmit_skbs函數來釋放描述符中的報文，直到avail->idx等于used->idx為止；

b.xmit_skb：主要是為報文添加vnet_hdr頭部信息，并將skb以scatter-gather形式顯示，以記錄報文數據的地址和長度信息；

c.virtqueue_add_outbuf：進行DMA映射，將scatter-gather記錄的報文數據地址和長度信息添加到desc環中，并增加avail->idx的值；

d.virtqueue_notify：當發送隊列存在數據，則通知后端。

virtio 網卡前端接收報文

virito網阿卡前端接收報文主要流程包括：

a.網卡硬中斷：硬中斷會將napi加入到CPU的處理隊列，并啟用中斷抑制，以及觸發軟中斷；

b.net_rx_action：網絡軟中斷入口函數；

c.virtnet_poll：這個函數是virtio網卡的NAPI poll的回調函數。如果當前隊列是發送隊列，它將清理發送隊列，也就是執行virtnet_poll_cleantx函數。如果當前隊列是接收隊列，它將進行報文的接收；

d.virtnet_receive：根據used->idx的值，從描述符環中讀取報文數據，并更新last_used_idx。內核會為報文數據分配skb，并進入GRO流程，進行報文的合并；e.try_fill_recv：要給desc環添加空的內存區域，并增加avail->idx的值，以確保接收隊列始終有可用的內存；

f.virtqueue_napi_complete：當接收的報文數量少于預定的budget（一般為64）時，表示沒有更多的數據可以接收。這時，調用virtqueue_napi_complete來表示單次napi處理完畢。同時，通過virtqueue_enable_cb_prepare來關閉中斷抑制。

網卡隊列可觀測

經過前面的分析，我們了解到virtio網卡隊列中的幾個重要參數，即avail->idx、used->idx和last_used_idx。使用這些參數，我們可以清晰地了解網卡隊列當前包含的報文數量，并進一步得到以下可觀測指標：

a.發送隊列報文數：表示尚未被virtio網卡后端發送的報文數量。計算方法是avail->idx - used->idx；

b.接收隊列報文數：表示尚未被virtio網卡前端接收的報文數量。計算方法是used->idx - last_used_idx；

c.網卡隊列的last_used_idx：表示virtio網卡后端處理報文的進度；

d.隊列飽和度：表示當前網卡隊列使用量，計算方法是隊列報文數/隊列長度。

工作原理

我們將可觀測的代碼集成在了rtrace的工具里，rtrace是龍蜥社區推出的系統工具集SysAK的一個網絡診斷分析工具，關于rtrace的具體原理，我們將在下回分析，eBPF 具體代碼請參考代碼：

https://gitee.com/anolis/sysak/blob/opensource_branch_sync/source/tools/detect/net/rtrace/src/bpf/virtio.bpf.c

virtio 網卡隊列指標采集的主要流程如下：

a.rtrace掛載eBPF采集程序到內核dev_id_show和dev_port_show函數；

b.rtrace周期性讀取/sys/class/net/[interface]/dev_id和/sys/class/net/[interface]/dev_port兩個文件，其中dev_id文件用來表示采集發送隊列信息，dev_port文件用來表示采集接收隊列信息；

c.當讀取文件時，會觸發內核執行dev_id_show和dev_port_show兩個函數。由于已經掛載了eBPF采集程序，內核會先執行eBPF采集程序；

d.eBPF采集程序通過解析dev_id_show和dev_port_show入參struct net_device獲取網卡隊列vring，然后從vring中解析出avail idx、used idx、隊列長度和last_used_idx；

e.將數據發送給rtrace做進一步處理。

故障檢測

下面是rtrace采集的網卡隊列信息輸出。

我們可以看到0926的1號發送隊列的飽和度和last_used_idx分別是0.05%/3593，0928的1號發送隊列的飽和度和last_used_idx分別是0.07%/3593，可以看到發送隊列的飽和度在增加，但是last_used_idx在多個采集周期內保持不變。因此，可以確定1號發送隊列出現了故障。

隨后我們修復了1號發送隊列故障，可以看見在0906的1號發送隊列飽和度和last_used_idx分別是0.00%/3599，隊列里面不再有駐留的報文，恢復了正常。

0924
SendQueue0.05%/35930.00%/8520.00%/45060.00%/16000.00%/4570.00%/5090.00%/31400.00%/13520.00%/3860.00%/4100.00%/17140.00%/17580.00%/16190.00%/4460.00%/35770.00%/24430.00%/460.00%/940.00%/2120.00%/2310.00%/1460.00%/1480.00%/2260.00%/640.00%/1090.00%/840.00%/780.00%/560.00%/870.00%/880.00%/850.00%/52
RecvQueue0.00%/28050.00%/132970.00%/4750.00%/3670.00%/123780.00%/1300.00%/2220.00%/111200.00%/3550.00%/30160.00%/1330.00%/1800.00%/129800.00%/103630.00%/28250.00%/6500.00%/1510.00%/5050.00%/51800.00%/2000.00%/266700.00%/1690.00%/10420.00%/98200.00%/95860.00%/33740.00%/2290.00%/14020.00%/87960.00%/1170.00%/3010.00%/275
0925
SendQueue0.05%/35930.00%/8520.00%/45060.00%/16000.00%/4570.00%/5090.00%/31400.00%/13520.00%/3860.00%/4100.00%/17140.00%/17580.00%/16190.00%/4460.00%/35770.00%/24440.00%/460.00%/940.00%/2120.00%/2310.00%/1460.00%/1480.00%/2260.00%/640.00%/1090.00%/840.00%/780.00%/560.00%/870.00%/890.00%/850.00%/52
RecvQueue0.00%/28050.00%/132970.00%/4750.00%/3670.00%/123780.00%/1300.00%/2220.00%/111200.00%/3550.00%/30160.00%/1330.00%/1800.00%/129800.00%/103630.00%/28250.00%/6500.00%/1510.00%/5050.00%/51800.00%/2000.00%/266700.00%/1690.00%/10420.00%/98200.00%/95860.00%/33740.00%/2290.00%/14020.00%/87960.00%/1170.00%/3030.00%/275
0926
SendQueue0.05%/35930.00%/8520.00%/45060.00%/16000.00%/4570.00%/5090.00%/31400.00%/13520.00%/3860.00%/4100.00%/17140.00%/17580.00%/16190.00%/4460.00%/35770.00%/24440.00%/460.00%/940.00%/2120.00%/2310.00%/1460.00%/1480.00%/2260.00%/640.00%/1090.00%/840.00%/780.00%/560.00%/870.00%/910.00%/850.00%/52
RecvQueue0.00%/28050.00%/132970.00%/4750.00%/3670.00%/123780.00%/1300.00%/2220.00%/111200.00%/3550.00%/30160.00%/1330.00%/1800.00%/129800.00%/103630.00%/28250.00%/6500.00%/1510.00%/5050.00%/51800.00%/2000.00%/266700.00%/1690.00%/10420.00%/98200.00%/95860.00%/33740.00%/2290.00%/14020.00%/87960.00%/1170.00%/3050.00%/275
0927
SendQueue0.07%/35930.00%/8520.00%/45060.00%/16000.00%/4570.00%/5090.00%/31400.00%/13520.00%/3860.00%/4100.00%/17140.00%/17580.00%/16190.00%/4460.00%/35770.00%/24440.00%/460.00%/940.00%/2120.00%/2310.00%/1460.00%/1480.00%/2260.00%/640.00%/1090.00%/840.00%/780.00%/560.00%/870.00%/930.00%/850.00%/52
RecvQueue0.00%/28050.00%/132980.00%/4750.00%/3670.00%/123780.00%/1300.00%/2220.00%/111200.00%/3550.00%/30160.00%/1330.00%/1800.00%/129800.00%/103630.00%/28250.00%/6500.00%/1510.00%/5050.00%/51800.00%/2000.00%/266700.00%/1690.00%/10420.00%/98200.00%/95860.00%/33740.00%/2290.00%/14020.00%/87960.00%/1170.00%/3070.00%/275
0928
SendQueue0.07%/35930.00%/8520.00%/45060.00%/16000.00%/4570.00%/5090.00%/31400.00%/13520.00%/3860.00%/4140.00%/17140.00%/17580.00%/16190.00%/4460.00%/35770.00%/24450.00%/460.00%/940.00%/2120.00%/2310.00%/1460.00%/1490.00%/2260.00%/640.00%/1090.00%/840.00%/780.00%/560.00%/870.00%/960.00%/870.00%/52
RecvQueue0.00%/28050.00%/132980.00%/4750.00%/3670.00%/123780.00%/1300.00%/2220.00%/111200.00%/3550.00%/30160.00%/1330.00%/1800.00%/129800.00%/103630.00%/28250.00%/6500.00%/1510.00%/5050.00%/51800.00%/2050.00%/266700.00%/1690.00%/10420.00%/98200.00%/95860.00%/33740.00%/2290.00%/14020.00%/87970.00%/1180.00%/3090.00%/275
0929
SendQueue0.07%/35930.00%/8520.00%/45060.00%/16000.00%/4570.00%/5090.00%/31400.00%/13520.00%/3860.00%/4140.00%/17140.00%/17580.00%/16190.00%/4460.00%/35770.00%/24450.00%/460.00%/940.00%/2120.00%/2310.00%/1460.00%/1490.00%/2260.00%/640.00%/1090.00%/840.00%/780.00%/560.00%/870.00%/980.00%/870.00%/52
RecvQueue0.00%/28050.00%/132980.00%/4750.00%/3670.00%/123780.00%/1300.00%/2220.00%/111200.00%/3550.00%/30160.00%/1330.00%/1800.00%/129800.00%/103630.00%/28250.00%/6500.00%/1510.00%/5050.00%/51800.00%/2050.00%/266700.00%/1690.00%/10420.00%/98200.00%/95860.00%/33740.00%/2290.00%/14020.00%/87970.00%/1180.00%/3110.00%/275
0930
SendQueue0.07%/35930.00%/8520.00%/45060.00%/16000.00%/4570.00%/5090.00%/31400.00%/13520.00%/3860.00%/4140.00%/17140.00%/17580.00%/16190.00%/4460.00%/35770.00%/24450.00%/460.00%/940.00%/2120.00%/2310.00%/1460.00%/1490.00%/2260.00%/640.00%/1090.00%/840.00%/780.00%/560.00%/870.00%/1000.00%/870.00%/52
RecvQueue0.00%/28050.00%/132980.00%/4750.00%/3670.00%/123780.00%/1300.00%/2220.00%/111200.00%/3550.00%/30160.00%/1330.00%/1800.00%/129800.00%/103630.00%/28250.00%/6500.00%/1510.00%/5050.00%/51800.00%/2050.00%/266700.00%/1690.00%/10420.00%/98200.00%/95860.00%/33740.00%/2290.00%/14020.00%/87970.00%/1180.00%/3130.00%/275
//...省略
0906
SendQueue0.00%/35990.00%/8560.00%/45110.00%/16020.00%/4650.00%/5100.00%/31400.00%/13520.00%/3860.00%/4200.00%/17160.00%/17660.00%/16190.00%/4480.00%/35780.00%/24510.00%/460.00%/940.00%/2120.00%/2310.00%/1480.00%/1490.00%/2260.00%/640.00%/1090.00%/850.00%/870.00%/560.00%/870.00%/1010.00%/1030.00%/52
RecvQueue0.00%/28070.00%/132990.00%/4770.00%/3690.00%/123780.00%/1400.00%/2230.00%/111200.00%/3550.00%/30320.00%/1420.00%/1800.00%/129800.00%/103630.00%/28250.00%/6520.00%/1510.00%/5050.00%/51800.00%/2050.00%/266700.00%/1700.00%/10570.00%/98200.00%/95860.00%/33740.00%/2300.00%/14140.00%/88000.00%/1180.00%/3270.00%/275

總結

在virtio網卡中，前端和后端之間通過共享的網卡隊列進行通信。為了更好地理解和觀測網卡隊列的狀態和性能指標，通過觀測avail idx、used idx、last_used_idx等指標，我們可以對virtio網卡的性能進行評估和優化。同時，這些指標也為我們提供了對網卡隊列狀態的深入理解，有助于進行故障排查和性能調優。