《零入门kubernetes网络实战》视频专栏地址

https://www.ixigua.com/7193641905282875942

本篇文章视频地址(稍后上传)

---

本篇文章主要是分享一下tap虚拟网络设备。

创建tap网络设备的方式跟tun完全一样，只需要将类型改为tap即可。

这里不再占用篇幅介绍了。

1、本篇文章的核心点

• tap虚拟网络设备一般用在什么场景下
• tap虚拟网络设备使用什么文件描述符
• 简单验证/dev/net/tun文件描述符是否支持并发读写操作？即验证在同一个宿主机上实现多个kvm类型的虚拟机网络通信的关键点

2、tap虚拟网络设备一般用在什么场景下

• openvpn
• kvm

3、tap虚拟网络设备使用什么文件描述符

tap跟tun一样，都是将此文件

/dev/net/tun

作为文件描述符的。
通过此文件跟虚拟网络设备tap、tun进行数据交互。

在centos7.5系统下，其他其他并为测试。

4、简单验证/dev/net/tun文件描述符是否支持并发读写操作？即验证在同一个宿主机上实现多个kvm类型的虚拟机网络通信的关键点

在kvm里使用tap设备进行的网络构建，那么，

本小节主要是想测试一下，文件描述符/dev/net/tun是否支持并发的读写操作。

如果支持并发读写操作的话，那么，下面的网络构建是成立的，

即在同一个宿主机上可以创建多个kvm类型的虚拟机。

在上图中：

• 在一个宿主机内部，通过物理网卡eth0跟外部网络进行数据通信
• 在宿主机内部，创建一个虚拟网桥br0
• 可以创建多个tap虚拟网络设备，
  • 这些tap设备挂载到虚拟网桥上br0
  • 并且这些tap设备都跟宿主机上的/dev/net/tun文件进行绑定，所有tap设备通过此文件进行读写
• 创建好的kvm类型的虚拟机里可以将数据包写入到宿主机上的/dev/net/tun文件里，
• 数据一旦到达/dev/net/tunW文件里，就相当于到达了虚拟网络设备tap里了，即到达了虚拟网桥br0

这样的话，kvm类型的虚拟机就可以跟外界进行通信了。

因此，主要是测试/dev/net/tun是否支持并发读写，其他相关操作不再测试了，如将tap设备挂载到虚拟网桥上。

4.1、测试/dev/net/tun支持并发读写的方案

我们创建两个测试用例，两个测试用例里都绑定了同一个宿主机上的/dev/net/tun文件，

运行这两个测试用例，观察是否出现问题，没有问题说明支持并发读写操作：

• 测试用例1，如何实现？
  • 在前面的文章中，我们使用了tun设备来实现点对点的VPN，在此基础上，我们对其代码进行改造，将tun设备改成tap设备来实现。
• 测试用例2，如何实现？
  • 将前面文章<<tun设备编程ICMP协议>>里的代码直接拿过来使用即可。

4.2、测试环境

两台虚拟机

• centos7.5 10.211.55.122
• centos7.5 10.211.55.123

4.3、测试用例1：基于tap虚拟网络设备实现点对点的VPN

4.3.1、代码结构

并没有严格意义上的客户端，服务器端

请求由哪个服务发起的，被称为客户端，

被请求服务，称为服务器端。

4.3.2、客户端代码

4.3.2.1、main.go文件

package main

import (
	"bytes"
	"fmt"
	"github.com/vishvananda/netlink"
	"log"
	"net"
	"os"
	"strings"
	"syscall"
	"time"
	"unsafe"
)

const (
	tapName    = "tap99"
	tapDevice  = "/dev/net/tun"
	ifnameSize = 16
	localAddr  = "10.211.55.122:8285"
	remoteAddr = "10.211.55.123:8285"
	tapIP      = "10.244.2.2"
)

type ifreqFlags struct {
	IfrnName  [ifnameSize]byte
	IfruFlags uint16
}

func ioctl(fd int, request, argp uintptr) error {
	_, _, errno := syscall.Syscall(syscall.SYS_IOCTL, uintptr(fd), request, argp)
	if errno != 0 {
		fmt.Errorf("ioctl failed with '%s'\n", errno)
		return fmt.Errorf("ioctl failed with '%s'", errno)
	}
	return nil
}

func fromZeroTerm(s []byte) string {
	return string(bytes.TrimRight(s, "\000"))
}

func OpenTap(name string) (*os.File, string, error) {
	tap, err := os.OpenFile(tapDevice, os.O_RDWR|syscall.O_NONBLOCK, 0)
	if err != nil {
		fmt.Printf("OpenTap Failed! err:%v", err.Error())
		return nil, "", err
	}
	var ifr ifreqFlags
	copy(ifr.IfrnName[:len(ifr.IfrnName)-1], []byte(name+"\000"))
	ifr.IfruFlags = syscall.IFF_TAP | syscall.IFF_NO_PI

	err = ioctl(int(tap.Fd()), syscall.TUNSETIFF, uintptr(unsafe.Pointer(&ifr)))
	if err != nil {
		fmt.Printf("OpenTap Failed! err:%v\n", err.Error())
		return nil, "", err
	}

	ifName := fromZeroTerm(ifr.IfrnName[:ifnameSize])
	return tap, ifName, nil
}

func main() {
	fmt.Printf("======>Now----Client----Tap---VPN---UDP<======\n")
	tapFile, err := createTap()
	if err != nil {
		fmt.Printf("ICMP Listen Packet Failed! err:%v\n", err.Error())
		return
	}
	defer tapFile.Close()

	udpConn, err := createUDP()
	if err != nil {
		fmt.Printf("UDP conn Failed! err:%v\n", err.Error())
		return
	}
	defer udpConn.Close()

	go tapToUDP(udpConn, tapFile)

	go udpToTap(udpConn, tapFile)

	time.Sleep(time.Hour)
}

func createUDP() (*net.UDPConn, error) {
	localAddr, err := net.ResolveUDPAddr("udp", localAddr)
	if err != nil {
		log.Fatalln("failed to get udp socket:", err)
		return nil, err
	}
	conn, err := net.ListenUDP("udp", localAddr)
	if err != nil {
		log.Fatalln("failed to listen on udp socket:", err)
		return nil, err
	}

	return conn, nil

}

func tapToUDP(udpConn *net.UDPConn, tapFile *os.File) {
	packet := make([]byte, 1024*64)
	size := 0
	var err error
	for {
		if size, err = tapFile.Read(packet); err != nil {
			return
		}

		te := MACType(packet[:size])
		printMACHeader(packet[:size])

		if strings.EqualFold(fmt.Sprintf("%x", te), "0800") {
			b := packet[14:size]
			printIPv4Header(b)
			if b[9] == 1 {
				icmpPacket := b[20:]
				printICMPHeader(icmpPacket)
			}

			if b[9] == 6 {
				tcpPacket := b[20:]
				printTCPHeader(tcpPacket)
			}

			if b[9] == 17 {
				udpPacket := b[20:]
				printUDPHeader(udpPacket)
			}
		}

		if strings.EqualFold(fmt.Sprintf("%x", te), "0806") {
			b := packet[14:size]
			printARPHeader(b)
		}

		rAddr, err := net.ResolveUDPAddr("udp", remoteAddr)
		if err != nil {
			log.Fatalln("failed to get udp socket:", err)
			return
		}
		if size, err = udpConn.WriteTo(packet[:size], rAddr); err != nil {
			fmt.Println(err.Error())
			return
		}
		fmt.Printf("tapToUDP--->Write Msg To UDP Conn OK! size:%d\n", size)
	}
}

func udpToTap(udpConn *net.UDPConn, tapFile *os.File) {
	var packet = make([]byte, 1024*64)
	var size int
	var err error
	var addr net.Addr

	for {
		if size, addr, err = udpConn.ReadFrom(packet); err != nil {
			continue
		}

		size, err = tapFile.Write(packet[:size])
		if err != nil {
			continue
		}
		fmt.Printf("udpToTap--->Write Msg To /dev/net/tun OK! size:%d\tsrcIP:%v\n", size, addr)
	}
}

func createTap() (*os.File, error) {
	err := addTap()
	if err != nil {
		return nil, err
	}

	err = configTap()
	if err != nil {
		return nil, err
	}

	tapFile, _, err := OpenTap(tapName)
	if err != nil {
		return nil, err
	}

	return tapFile, nil
}

func addTap() error {
	la := netlink.LinkAttrs{
		Name:  tapName,
		Index: 8,
		MTU:   1500,
	}
	tap := netlink.Tuntap{
		LinkAttrs: la,
		Mode:      netlink.TUNTAP_MODE_TAP,
	}

	l, err := netlink.LinkByName(tapName)
	if err == nil {
		netlink.LinkSetDown(l)

		netlink.LinkDel(l)
	}

	err = netlink.LinkAdd(&tap)
	if err != nil {
		return err
	}
	return nil
}

func configTap() error {

	l, err := netlink.LinkByName(tapName)
	if err != nil {
		return err
	}

	ip, err := netlink.ParseIPNet(fmt.Sprintf("%s/%d", tapIP, 24))
	if err != nil {
		return err
	}

	addr := &netlink.Addr{IPNet: ip, Label: ""}
	if err = netlink.AddrAdd(l, addr); err != nil {
		return err
	}

	err = netlink.LinkSetUp(l)
	if err != nil {
		return err
	}

	return nil
}

具体下一小节在介绍。

目前，先把/dev/net/tun文件的并发性介绍完。

4.3.2.2、etherheader.go文件

package main

import (
	"fmt"
	"net"
)

func printMACHeader(packet []byte) {
	fmt.Println()
	fmt.Println()
	fmt.Println()
	fmt.Println("----->MAC Header<----")
	printSrcMACEther(packet)
	printDstMACEther(packet)
	printTypeEther(packet)
	fmt.Println("----->MAC Header<--end--")
}

func printDstMACEther(packet []byte) {
	fmt.Printf("Ether Header--->DstMac:%x:%x:%x:%x:%x:%x\n", packet[0], packet[1], packet[2], packet[3], packet[4], packet[5])
}

func printSrcMACEther(packet []byte) {
	fmt.Printf("Ether Header--->SrcMac:%x:%x:%x:%x:%x:%x\n", packet[6], packet[7], packet[8], packet[9], packet[10], packet[11])
}
func printTypeEther(packet []byte) {
	fmt.Printf("Ether Header--->Type:%04x\n", uint16(packet[12])<<8|uint16(packet[13]))
}

func MACDestination(macFrame []byte) net.HardwareAddr {
	return net.HardwareAddr(macFrame[:6])
}

func MACSource(macFrame []byte) net.HardwareAddr {
	return net.HardwareAddr(macFrame[6:12])
}

func MACType(macFrame []byte) []byte {
	return macFrame[12:14]
}

func MACTypeARP(macFrame []byte) []byte {
	return macFrame[2:4]
}

4.3.2.3、arpheader.go文件

package main

import (
	"fmt"
	"net"
)

func printARPHeader(packet []byte) {
	fmt.Println()
	fmt.Println()
	fmt.Println()
	fmt.Println("----->ARP Header<----")
	printARPTYPE(packet)
	printARPProTYPE(packet)
	printARPHardwareLen(packet)
	printARPPLen(packet)
	printARPOp(packet)
	printARPSrcHardwareMAC(packet)
	printARPSrcIP(packet)
	printARPDstHardwareMAC(packet)
	printARPDstIP(packet)
	fmt.Println("----->ARP Header<---END---")

}
func printARPTYPE(packet []byte) {
	fmt.Printf("ARP Header--->Type:%d\n", uint16(packet[0])<<8|uint16(packet[1]))
}

func printARPProTYPE(packet []byte) {
	fmt.Printf("ARP Header--->ProtocolType:%04x\n", uint16(packet[2])<<8|uint16(packet[3]))
}

func printARPHardwareLen(packet []byte) {
	fmt.Printf("ARP Header--->HardwareLen:%d\n", packet[4])
}

func printARPPLen(packet []byte) {
	fmt.Printf("ARP Header--->ProtocolLen:%d\n", packet[5])
}

func printARPOp(packet []byte) {
	fmt.Printf("ARP Header--->op:%d\n", uint16(packet[6])<<8|uint16(packet[7]))
}

func printARPSrcHardwareMAC(packet []byte) {
	fmt.Printf("ARP Header--->SrcHardwarMAC:%x:%x:%x:%x:%x:%x\n", packet[8], packet[9], packet[10], packet[11], packet[12], packet[13])
}

func printARPSrcIP(packet []byte) {
	fmt.Printf("ARP Header--->SrcIP:%d.%d.%d.%d\n", packet[14], packet[15], packet[16], packet[17])
}

func printARPDstHardwareMAC(packet []byte) {
	fmt.Printf("ARP Header--->DstHardwareMAC:%x:%x:%x:%x:%x:%x\n", packet[18], packet[19], packet[20], packet[21], packet[22], packet[23])
}

func printARPDstIP(packet []byte) {
	fmt.Printf("ARP Header--->DstIP:%d.%d.%d.%d\n", packet[24], packet[25], packet[26], packet[27])
}

func GetIPv4SrcARP(packet []byte) net.IP {
	return net.IPv4(packet[14], packet[15], packet[16], packet[17])
}

func GetIPv4DstARP(packet []byte) net.IP {
	return net.IPv4(packet[24], packet[25], packet[26], packet[27])
}

4.3.2.4、icmpheader.go文件

package main

import "fmt"

func printICMPHeader(packet []byte) {
	fmt.Println()
	fmt.Println()
	fmt.Println()
	fmt.Println("----->ICMP Header<----")
	printICMPType(packet)
	printICMPCode(packet)
	printICMPCheckSum(packet)
	printICMPIdentification(packet)
	printICMPSeqNum(packet)
	printICMPTimestamp(packet)
	printICMPData(packet)
	fmt.Println("----->ICMP Header<----End----")
}

func printICMPType(packet []byte) {
	fmt.Printf("ICMP Header--->Type:%d\n", packet[0])
}

func printICMPCode(packet []byte) {
	fmt.Printf("ICMP Header--->Code:%d\n", packet[1])
}

func printICMPCheckSum(packet []byte) {
	fmt.Printf("ICMP Header--->Checksum:%04x\n", uint16(packet[2])<<8|uint16(packet[3]))
}

func printICMPIdentification(packet []byte) {
	fmt.Printf("TICMP Header--->Identification(process):%d\n", uint16(packet[4])<<8|uint16(packet[5]))
}

func printICMPSeqNum(packet []byte) {
	fmt.Printf("ICMP Header--->SeqNum:%d\n", uint16(packet[6])<<8|uint16(packet[7]))
}

func printICMPTimestamp(packet []byte) {
	fmt.Printf("ICMP Header --->timestamp:%02x %02x %02x %02x %02x %02x %02x %02x\n", packet[8], packet[9], packet[10], packet[11], packet[12], packet[13], packet[14], packet[15])
}

func printICMPData(packet []byte) {
	fmt.Printf("ICMP Header--->data:%v\n", string(packet[20:]))
}

4.3.2.5、tcpheader.go文件

package main

import (
	"fmt"
)

func printTCPHeader(packet []byte) {
	fmt.Println()
	fmt.Println()
	fmt.Println()
	fmt.Printf("----->TCP Header<----len:%d\n", len(packet[:]))
	printTCPsrcPort(packet)
	printTCPdstPort(packet)
	printTCPSequenceNumber(packet)
	printTCPACKNumber(packet)
	printTCPHeaderLen(packet)
	printTCPFlagCWR(packet)
	printTCPFlagECE(packet)
	printTCPFlagUrgent(packet)
	printTCPFlagACK(packet)
	printTCPFlagPSH(packet)
	printTCPFlagRST(packet)
	printTCPFlagSYN(packet)
	printTCPFlagFIN(packet)
	printTCPWindowSize(packet)
	printTCPCheckSum(packet)
	printTCPUrgentPointer(packet)
	printTCPData(packet)
	fmt.Println("----->TCP Header<---END---")
}

func printTCPsrcPort(packet []byte) {
	fmt.Printf("TCP Header--->SrcPort:%d\n", int64(uint16(packet[0])<<8|uint16(packet[1])))
}

func printTCPdstPort(packet []byte) {
	fmt.Printf("TCP Header--->DstPort:%d\n", int64(uint16(packet[2])<<8|uint16(packet[3])))
}

func printTCPSequenceNumber(packet []byte) {
	fmt.Printf("TCP Header--->SequenceNumber:%d\n", uint32(packet[4])<<24|uint32(packet[5])<<16|uint32(packet[6])<<8|uint32(packet[7]))
}

func printTCPACKNumber(packet []byte) {
	fmt.Printf("TCP Header--->ACKNum:%d\n", uint32(packet[8])<<24|uint32(packet[9])<<16|uint32(packet[10])<<8|uint32(packet[11]))
}

func printTCPHeaderLen(packet []byte) {
	fmt.Printf("TCP Header--->HeaderLen:%d\n", packet[12]>>4*4)
}

func printTCPFlagCWR(packet []byte) {
	fmt.Printf("TCP Header--->FlagCWR:%d\n", packet[13]&0x80>>7)
}
func printTCPFlagECE(packet []byte) {
	fmt.Printf("TCP Header--->FlagEcho:%d\n", packet[13]&0x40>>6)
}

func printTCPFlagUrgent(packet []byte) {
	fmt.Printf("TCP Header--->FlagUrgent:%d\n", packet[13]&0x20>>5)
}

func printTCPFlagACK(packet []byte) {
	fmt.Printf("TCP Header--->FlagACK:%d\n", packet[13]>>4&0b0001)
	fmt.Printf("TCP Header--->FlagACK:%d\n", packet[13]>>4&0b1)
	//fmt.Printf("TCP Header--->FlagACK:%d\tpacket:%v\n", packet[13]>>4, packet[13])
	fmt.Printf("TCP Header--->FlagACK:%d\n", packet[13]&0x10>>4)
}

func printTCPFlagPSH(packet []byte) {
	fmt.Printf("TCP Header--->FlagPSH:%d\n", packet[13]&0x08>>3)
}

func printTCPFlagRST(packet []byte) {
	fmt.Printf("TCP Header--->FlagRST:%d\n", packet[13]&0x04>>2)
}

func printTCPFlagSYN(packet []byte) {
	fmt.Printf("TCP Header--->FlagSYN:%d\n", packet[13]&0x02>>1)
}

func printTCPFlagFIN(packet []byte) {
	fmt.Printf("TCP Header--->FlagFIN:%d\n", packet[13]&0x01)
}

func printTCPWindowSize(packet []byte) {
	fmt.Printf("TCP Header--->WindowSize:%d\n", uint16(packet[14])<<8|uint16(packet[15]))
}

func printTCPCheckSum(packet []byte) {
	fmt.Printf("TCP Header--->Checksum:%04x\n", uint16(packet[16])<<8|uint16(packet[17]))
}

func printTCPUrgentPointer(packet []byte) {
	fmt.Printf("TCP Header--->UrgentPointer:%d\n", uint16(packet[18])<<8|uint16(packet[19]))
}

func printTCPData(packet []byte) {
	headerLen := packet[12] >> 4 * 4
	p := packet[headerLen:]
	dataLen := len(p)
	if dataLen > 1 {
		fmt.Printf("TCP Header--->Data--->:%v\theaderLen:%v\tdataLen:%d\n", string(p[:dataLen-1]), headerLen, dataLen)
	}
}

4.3.2.6、udpheader.go文件

package main

import "fmt"

func printUDPHeader(packet []byte) {
	fmt.Println()
	fmt.Println()
	fmt.Println()
	fmt.Println("----->UDP Header<----")
	printUDPSrcPort(packet)
	printUDPDstPort(packet)
	printUDPAllLen(packet)
	printUDPChecksum(packet)
	printUDPData(packet)
	fmt.Println("----->UDP Header<---End--")
}

func printUDPSrcPort(packet []byte) {
	fmt.Printf("UDP Header--->SrcPort:%d\n", uint16(packet[0])<<8|uint16(packet[1]))
}

func printUDPDstPort(packet []byte) {
	fmt.Printf("UDP Header--->DstPort:%d\n", uint16(packet[2])<<8|uint16(packet[3]))
}

func printUDPAllLen(packet []byte) {
	fmt.Printf("UDP Header--->AllLen:%d\n", uint16(packet[4])<<8|uint16(packet[5]))
}

func printUDPChecksum(packet []byte) {
	fmt.Printf("UDP Header--->Checksum:%d\n", uint16(packet[6])<<8|uint16(packet[7]))
}

func printUDPData(packet []byte) {
	fmt.Printf("UDP Header--->data:%v\n", string(packet[8:]))
}

4.3.2.7、ipv4header.go文件

package main

import (
	"fmt"
	"net"
)

//  打印ip报文头的详情
func printIPv4Header(packet []byte) {
	fmt.Println()
	fmt.Println()
	fmt.Println()
	fmt.Println("----->IP Header<----")
	printVersionIPv4(packet)
	printHeaderLenIPv4(packet)
	printServiceTypeIPv4(packet)
	printAllLenIPv4(packet)
	printIdentificationIPv4(packet)
	printFlagsIPv4(packet)
	printFragmentOffsetIPv4(packet)
	printTTLIPv4(packet)
	printProtocolIPv4(packet)
	printChecksumIPv4(packet)
	printSrcIPv4(packet)
	printDstIPv4(packet)
	fmt.Println("----->IP Header<---End---")
}

func printVersionIPv4(packet []byte) {
	header := packet[0]
	fmt.Printf("IPv4 Header--->Version:%d\n", header>>4)
}

func printHeaderLenIPv4(packet []byte) {
	header := packet[0]
	fmt.Printf("IPv4 Header--->HeaderLen:%d byte\n", header&0x0f*4)
}

func printServiceTypeIPv4(packet []byte) {
	st := packet[1]
	fmt.Printf("IPv4 Header--->ServiceType:%v\n", st)
}

func printAllLenIPv4(packet []byte) {
	fmt.Printf("IPv4 Header--->AllLen:%d\n", uint16(packet[2])<<8|uint16(packet[3]))
}

func printIdentificationIPv4(packet []byte) {
	id := uint16(packet[4])<<8 | uint16(packet[5])
	fmt.Printf("IPv4 Header--->Identification:%x\t%d\n", id, id)
}

func printFlagsIPv4(packet []byte) {
	// 向右移动5位，相当于去掉后5位。数值降低
	fmt.Printf("IPv4 Header--->Flags:%03b\n", packet[6]>>5)
}
func printFragmentOffsetIPv4(packet []byte) {
	// 向左移动3位，相当于将前3位去掉。数值可能增加
	fmt.Printf("IPv4 Header--->FragmentOffset:%013b\n", uint16(packet[6])<<3|uint16(packet[7]))
}

func printTTLIPv4(packet []byte) {
	fmt.Printf("IPv4 Header--->TTL:%d\n", packet[8])
}

func printProtocolIPv4(packet []byte) {
	fmt.Printf("IPv4 Header--->ProtocolType:%d\n", packet[9])
}

func printChecksumIPv4(packet []byte) {
	fmt.Printf("IPv4 Header--->Checksum:%d\n", uint16(packet[10])<<8|uint16(packet[11]))
}

func printSrcIPv4(packet []byte) net.IP {
	fmt.Printf("IPv4 Header--->SrcIP:%d.%d.%d.%d\n", packet[12], packet[13], packet[14], packet[15])
	return net.IPv4(packet[12], packet[13], packet[14], packet[15])
}

func printDstIPv4(packet []byte) net.IP {
	fmt.Printf("IPv4 Header--->DstIP:%d.%d.%d.%d\n", packet[16], packet[17], packet[18], packet[19])
	return net.IPv4(packet[16], packet[17], packet[18], packet[19])
}

4.3.3、服务端代码

4.3.3.1、server.go代码

package main

import (
	"bytes"
	"fmt"
	"github.com/vishvananda/netlink"
	"log"
	"net"
	"os"
	"syscall"
	"time"
	"unsafe"
)

const (
	tapName    = "tap99"
	tapDevice  = "/dev/net/tun"
	ifnameSize = 16
	localAddr  = "10.211.55.123:8285"
	remoteAddr = "10.211.55.122:8285"
	tapIP      = "10.244.3.3"
)

type ifreqFlags struct {
	IfrnName  [ifnameSize]byte
	IfruFlags uint16
}

func ioctl(fd int, request, argp uintptr) error {
	_, _, errno := syscall.Syscall(syscall.SYS_IOCTL, uintptr(fd), request, argp)
	if errno != 0 {
		fmt.Errorf("ioctl failed with '%s'\n", errno)
		return fmt.Errorf("ioctl failed with '%s'", errno)
	}
	return nil
}

func fromZeroTerm(s []byte) string {
	return string(bytes.TrimRight(s, "\000"))
}

func OpenTap(name string) (*os.File, string, error) {
	tap, err := os.OpenFile(tapDevice, os.O_RDWR|syscall.O_NONBLOCK, 0)
	if err != nil {
		fmt.Printf("OpenTap Failed! err:%v", err.Error())
		return nil, "", err
	}
	var ifr ifreqFlags
	copy(ifr.IfrnName[:len(ifr.IfrnName)-1], []byte(name+"\000"))
	ifr.IfruFlags = syscall.IFF_TAP | syscall.IFF_NO_PI

	err = ioctl(int(tap.Fd()), syscall.TUNSETIFF, uintptr(unsafe.Pointer(&ifr)))
	if err != nil {
		fmt.Printf("OpenTap Failed! err:%v\n", err.Error())
		return nil, "", err
	}

	ifName := fromZeroTerm(ifr.IfrnName[:ifnameSize])
	return tap, ifName, nil
}

func main() {
	fmt.Printf("======>Now----Server----Tap---VPN---UDP<======\n")
	tapFile, err := createTap()
	if err != nil {
		fmt.Printf("ICMP Listen Packet Failed! err:%v\n", err.Error())
		return
	}
	defer tapFile.Close()

	udpConn, err := createUDP()
	if err != nil {
		fmt.Printf("UDP conn Failed! err:%v\n", err.Error())
		return
	}
	defer udpConn.Close()

	go tapToUDP(udpConn, tapFile)

	go udpToTap(udpConn, tapFile)

	time.Sleep(time.Hour)
}

func createUDP() (*net.UDPConn, error) {
	localAddr, err := net.ResolveUDPAddr("udp", localAddr)
	if err != nil {
		log.Fatalln("failed to get udp socket:", err)
		return nil, err
	}
	conn, err := net.ListenUDP("udp", localAddr)
	if err != nil {
		log.Fatalln("failed to listen on udp socket:", err)
		return nil, err
	}

	return conn, nil

}

func tapToUDP(udpConn *net.UDPConn, tapFile *os.File) {
	packet := make([]byte, 1024*64)
	size := 0
	var err error
	for {
		if size, err = tapFile.Read(packet); err != nil {
			return
		}

		srcMAC := MACSource(packet[:size])
		dstMAC := MACDestination(packet[:size])
		fmt.Printf("Server==>TapToUDP----srcMAC:%v\tdstMAC:%v\n", srcMAC.String(), dstMAC.String())

		rAddr, err := net.ResolveUDPAddr("udp", remoteAddr)
		if err != nil {
			log.Fatalln("failed to get udp socket:", err)
			return
		}
		if size, err = udpConn.WriteTo(packet[:size], rAddr); err != nil {
			fmt.Println(err.Error())
			return
		}
		fmt.Printf("tapToUDP--->Write Msg To UDP Conn OK! size:%d\n", size)
	}
}

func udpToTap(udpConn *net.UDPConn, tapFile *os.File) {
	var packet = make([]byte, 1024*64)
	var size int
	var err error
	var addr net.Addr

	for {
		if size, addr, err = udpConn.ReadFrom(packet); err != nil {
			continue
		}
		srcMAC := MACSource(packet[:size])
		dstMAC := MACDestination(packet[:size])
		fmt.Printf("Server==>UDPToTap----srcMAC:%v\tdstMAC:%v\n", srcMAC.String(), dstMAC.String())
		size, err = tapFile.Write(packet[:size])
		if err != nil {
			continue
		}
		fmt.Printf("udpToTap--->Write Msg To /dev/net/tun OK! size:%d\tsrcIP:%v\n", size, addr)
	}
}

func createTap() (*os.File, error) {
	err := addTap()
	if err != nil {
		return nil, err
	}

	err = configTap()
	if err != nil {
		return nil, err
	}

	tapFile, _, err := OpenTap(tapName)
	if err != nil {
		return nil, err
	}

	return tapFile, nil
}

func addTap() error {
	la := netlink.LinkAttrs{
		Name:  tapName,
		Index: 8,
		MTU:   1500,
	}
	tap := netlink.Tuntap{
		LinkAttrs: la,
		Mode:      netlink.TUNTAP_MODE_TAP,
	}

	l, err := netlink.LinkByName(tapName)
	if err == nil {
		netlink.LinkSetDown(l)

		netlink.LinkDel(l)
	}

	err = netlink.LinkAdd(&tap)
	if err != nil {
		return err
	}
	return nil
}

func configTap() error {

	l, err := netlink.LinkByName(tapName)
	if err != nil {
		return err
	}

	ip, err := netlink.ParseIPNet(fmt.Sprintf("%s/%d", tapIP, 24))
	if err != nil {
		return err
	}

	addr := &netlink.Addr{IPNet: ip, Label: ""}
	if err = netlink.AddrAdd(l, addr); err != nil {
		return err
	}

	err = netlink.LinkSetUp(l)
	if err != nil {
		return err
	}

	return nil
}

func MACDestination(macFrame []byte) net.HardwareAddr {
	return net.HardwareAddr(macFrame[:6])
}

func MACSource(macFrame []byte) net.HardwareAddr {
	return net.HardwareAddr(macFrame[6:12])
}

4.3.4、go.mod参考

module tun-test/icmp/v2

go 1.17

require github.com/vishvananda/netlink v1.1.1-0.20210330154013-f5de75959ad5

require (
	github.com/vishvananda/netns v0.0.0-20200728191858-db3c7e526aae // indirect
	golang.org/x/sys v0.0.0-20220728004956-3c1f35247d10 // indirect
)

记得修改module名称，修改自己实际的。

同样，客户端，服务器端代码同样修改一下即可。

4.3.5、本地编译，上传到10.211.55.122以及10.211.55.123节点上去

Makefile

build:
	CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o tap-client ./client/*.go
	CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o tap-server ./server/main.go

scp:
	scp tap-client root@10.211.55.122:/root
	scp tap-server root@10.211.55.122:/root

all:
	make build && make scp

在本地执行

make all

4.4、测试用例2：使用tun设备实现ICMP协议

4.4.1、main.go文件内容，如下

package main

import (
	"bytes"
	"encoding/binary"
	"fmt"
	"github.com/vishvananda/netlink"
	"golang.org/x/net/icmp"
	"golang.org/x/net/ipv4"
	"net"
	"os"
	"syscall"
	"time"
	"unsafe"
)

const (
	tunName    = "tun19"
	tunDevice  = "/dev/net/tun"
	ifnameSize = 16
	eth0       = "10.211.55.122"
	tunIP      = "10.244.2.9"
)

type ifreqFlags struct {
	IfrnName  [ifnameSize]byte
	IfruFlags uint16
}

func ioctl(fd int, request, argp uintptr) error {
	_, _, errno := syscall.Syscall(syscall.SYS_IOCTL, uintptr(fd), request, argp)
	if errno != 0 {
		fmt.Errorf("ioctl failed with '%s'\n", errno)
		return fmt.Errorf("ioctl failed with '%s'", errno)
	}
	return nil
}

func fromZeroTerm(s []byte) string {
	return string(bytes.TrimRight(s, "\000"))
}

func OpenTun(name string) (*os.File, string, error) {
	tun, err := os.OpenFile(tunDevice, os.O_RDWR|syscall.O_NONBLOCK, 0)
	if err != nil {
		fmt.Printf("OpenTun Failed! err:%v", err.Error())
		return nil, "", err
	}
	var ifr ifreqFlags
	copy(ifr.IfrnName[:len(ifr.IfrnName)-1], []byte(name+"\000"))
	ifr.IfruFlags = syscall.IFF_TUN | syscall.IFF_NO_PI

	err = ioctl(int(tun.Fd()), syscall.TUNSETIFF, uintptr(unsafe.Pointer(&ifr)))
	if err != nil {
		fmt.Printf("OpenTun Failed! err:%v\n", err.Error())
		return nil, "", err
	}

	ifName := fromZeroTerm(ifr.IfrnName[:ifnameSize])
	return tun, ifName, nil
}

func checkSum(data []byte) uint16 {
	var (
		sum    uint32
		length int = len(data)
		index  int
	)
	for length > 1 {
		sum += uint32(data[index])<<8 + uint32(data[index+1])
		index += 2
		length -= 2
	}
	if length > 0 {
		sum += uint32(data[index])
	}
	sum += sum >> 16

	return uint16(^sum)
}

func main() {
	fmt.Printf("======>Now----Test----Tun<======\n")
	tunFile, err := createTun()
	if err != nil {
		fmt.Printf("ICMP Listen Packet Failed! err:%v\n", err.Error())
		return
	}
	defer tunFile.Close()

	icmpConn, _ := icmp.ListenPacket("ip4:icmp", eth0)

	defer icmpConn.Close()

	go tunToIcmp(icmpConn, tunFile)

	go icmpToTun(icmpConn, tunFile)

	time.Sleep(time.Hour)
}

func tunToIcmp(icmpconn *icmp.PacketConn, tunFile *os.File) {
	var srcIP string
	packet := make([]byte, 1024*64)
	size := 0
	var err error
	for {
		if size, err = tunFile.Read(packet); err != nil {
			return
		}
		fmt.Printf("Msg Length: %d\n", binary.BigEndian.Uint16(packet[2:4]))
		fmt.Printf("Msg Protocol: %d (1=ICMP, 6=TCP, 17=UDP)\tsize:%d\n", packet[9], size)

		b := packet[:size]
		srcIP = GetSrcIP(b)
		dstIP := GetDstIP(b)
		fmt.Printf("Msg srcIP: %s\tdstIP:%v\n", srcIP, dstIP)

		var raddr = net.IPAddr{IP: net.ParseIP(dstIP)}

		b = b[20:size]

		if size, err = icmpconn.WriteTo(b, &raddr); err != nil {
			fmt.Println(err.Error())
			return
		}
		fmt.Printf("Write Msg To Icmp Conn OK! size:%d\n", size)
	}
}

func icmpToTun(icmpconn *icmp.PacketConn, tunFile *os.File) {
	var sb = make([]byte, 1024*64)
	var addr net.Addr
	var size int
	var err error

	for {
		if size, addr, err = icmpconn.ReadFrom(sb); err != nil {
			continue
		}

		ipHeader := createIPv4Header(net.ParseIP(addr.String()), net.ParseIP(tunIP), os.Getpid())
		iphb, err := ipHeader.Marshal()
		if err != nil {
			continue
		}
		fmt.Printf("Reply MSG Length: %d\n", binary.BigEndian.Uint16(iphb[2:4]))
		fmt.Printf("Reply MSG Protocol: %d (1=ICMP, 6=TCP, 17=UDP)\n", iphb[9])
		dstIP := GetDstIP(iphb)
		fmt.Printf("Reply src IP: %s\tdstIP:%v\n", addr, dstIP)

		var rep = make([]byte, 84)
		rep = append(iphb, sb[:size]...)

		size, err = tunFile.Write(rep)
		if err != nil {
			continue
		}
		fmt.Printf("Write Msg To /dev/net/tun OK! size:%d\ttime:%v\n", size, time.Now())
	}
}

func createIPv4Header(src, dst net.IP, id int) *ipv4.Header {

	iph := &ipv4.Header{
		Version:  ipv4.Version,
		Len:      ipv4.HeaderLen,
		TOS:      0x00,
		TotalLen: ipv4.HeaderLen + 64,
		ID:       id,
		Flags:    ipv4.DontFragment,
		FragOff:  0,
		TTL:      64,
		Protocol: 1,
		Checksum: 0,
		Src:      src,
		Dst:      dst,
	}

	h, _ := iph.Marshal()

	iph.Checksum = int(checkSum(h))

	return iph
}

func IsIPv4(packet []byte) bool {
	flag := packet[0] >> 4
	return flag == 4
}

func GetIPv4Src(packet []byte) net.IP {
	return net.IPv4(packet[12], packet[13], packet[14], packet[15])
}

func GetIPv4Dst(packet []byte) net.IP {
	return net.IPv4(packet[16], packet[17], packet[18], packet[19])
}

func GetSrcIP(packet []byte) string {
	key := ""
	if IsIPv4(packet) && len(packet) >= 20 {
		key = GetIPv4Src(packet).To4().String()
	}

	return key
}

func GetDstIP(packet []byte) string {
	key := ""
	if IsIPv4(packet) && len(packet) >= 20 {
		key = GetIPv4Dst(packet).To4().String()
	}

	return key
}

func createTun() (*os.File, error) {
	err := addTun()
	if err != nil {
		return nil, err
	}

	err = configTun()
	if err != nil {
		return nil, err
	}

	tunFile, _, err := OpenTun(tunName)
	if err != nil {
		return nil, err
	}

	return tunFile, nil
}

func addTun() error {
	la := netlink.LinkAttrs{
		Name:  tunName,
		Index: 8,
		MTU:   1500,
	}
	tun := netlink.Tuntap{
		LinkAttrs: la,
		Mode:      netlink.TUNTAP_MODE_TUN,
	}

	l, err := netlink.LinkByName(tunName)
	if err == nil {
		netlink.LinkSetDown(l)

		netlink.LinkDel(l)
	}

	err = netlink.LinkAdd(&tun)
	if err != nil {
		return err
	}
	return nil
}

func configTun() error {

	l, err := netlink.LinkByName(tunName)
	if err != nil {
		return err
	}

	ip, err := netlink.ParseIPNet(fmt.Sprintf("%s/%d", tunIP, 24))
	if err != nil {
		return err
	}

	addr := &netlink.Addr{IPNet: ip, Label: ""}
	if err = netlink.AddrAdd(l, addr); err != nil {
		return err
	}

	err = netlink.LinkSetUp(l)
	if err != nil {
		return err
	}

	return nil
}

4.4.2、编译、上传到10.211.55.122上

Makefile

build:
	CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o tun-driver main.go

scp:
	scp tun-driver root@10.211.55.122:/root

all:
	make build && make scp

本地执行

make all

即可

4.5、注意事项

修改代码时，要注意下

• 第一个测试用例中，创建是tap类型的虚拟网络设备tap99
• 第二个测试用例中，创建的tun类型的虚拟网络设备tun19

我们测试的关键点是，不同的虚拟网络设备绑定在同一个文件描述符/dev/net/tun上
只要这两个设备能够同时读写操作，即可。

实话实话，主要是不想再修改第二个测试用例的代码，将其改成tap类型的。
在tun、tap上已经用了很多时间了，太累了。

4.6、启动服务

4.6.1、登录到10.211.55.123节点上

4.6.1.1、启动第一个测试用例的服务器端

./tap-server

4.6.1.2、设置路由表，针对的是数据包去往10.211.55.122

route -n

route add -net 10.244.2.0/24 dev tap99

route -n

4.6.2、登录到10.211.55.122节点上

4.6.2.1、启动tap-client服务

4.6.2.1.1、启动tap-client服务

 ./tap-client 

4.6.2.1.2、设置通往10.211.55.123节点的路由

route -n

route add -net 10.244.3.0/24 dev tap99

route -n

4.6.2.2、启动第2个测试用例

./tun-driver 

设置路由反向检测属性

sysctl net.ipv4.conf.all.rp_filter

sysctl -w net.ipv4.conf.all.rp_filter=2

sysctl net.ipv4.conf.all.rp_filter

4.7、通过ping命令来测试/dev/net/tun是否支持并发读写

登录到
10.211.55.122 节点上去

4.7.1、在10.211.55.122 节点上，发起ping命令去ping10.211.55.123节点上的tap99设备

ping 10.244.3.3 -I tap99

抓包命令如下：

tcpdump -nn -i tap99

4.7.2、在10.211.55.122 节点上，发起ping命令去ping10.211.55.123节点上的对外物理网卡eth0

ping 10.211.55.123 -I tun19

抓包命令如下：

tcpdump -nn -i tun19

5、总结

在10.211.55.122节点上，
通过ping命令同时使用tun19虚拟设备以及tap99虚拟设备发起ping请求

整个请求中，数据包有去有回，无丢包现象，

而tap19,tun99同时绑定了/dev/net/tun文件描述符，

说明/dev/net/tun文件描述符支持并发读写操作。

这样的话，本次测试的目的之一就到达了。

---

<<零入门kubernetes网络实战>>技术专栏之文章目录

---