大布景
销售营业运转在 Kubernetes 天然情况中，他们时常会碰着许多头疼的互联网难题，该如何对 Pod 外部的收集流量展开Tourbe摸查难题呢？因为销售营业快照一般来说都是简化之后的，一般来说不会加拆 tcpdump 指示。一般来说的做法是登入 Pod 所处 Node 结点，接着进入罐子 netns，最初接纳结点上 tcpdump 辅助东西展开Tourbe，一般来说更要把互联网包文档阅读到临近地域条记本电脑，皮德盖 Wireshark 展开阐发。 但整座操做过程较为冗杂，出格是三四个办事项目，在差别的 Node 结点上。上面我就来如是说一种较为加速便利快速展开互联网Tourbe的配套办法。
TL;DR现阶段只不外有许多种Tourbe的体例，各有好坏。但有两个共通点，那就是都不提议在消费天然情况中接纳。假使必然要接纳，对小我所保举接纳上面如是说的第 3 种方案，不外现阶段他们的 GKE 软件财产暂还不全力撑持。
接纳附加罐子：该方案次要是在原有的 Pod 中加进两个附加的罐子，借助于互联网空间共享资本的长处，接纳 tcpdump 展开Tourbe。长处是必要批改本来的 Pod，那容貌 Pod 必要从头启动。ksniff: 是两个 kubectl 应用法式，ksniff 的实现是上载两个动态校对的 tcpdump 到 Pod 中，接着将 tcpdump 的输入链接到临近地域的 Wireshark 展开增容。kubectl-debug: Kubernetes 供给了两个临时性罐子（ EphemeralContainers ）供他们加进到要增容的 Pod 中， EphemeralContainers 那个长处必要 v1.23 之后才会变成预设迈入，现阶段他们的 GKE 暂不全力撑持该长处。所以，综合脾气况来看，若是他们要在测试天然情况做Tourbe，现阶段最适宜他们的是接纳 ksniff，他即借助于了 Wireshark 的强悍潜能，又削减了对 Pod 的影响。加拆 ksniff一般来说他们都是通过 krew 来加拆 kubectl 的应用法式。但他们必要先加拆 krew，那里我以 MacOS 控造系统为例，详述一下整座加拆操做过程。
加拆 krew ✘ p.zhang@pzhang-Mac ~ ( set -x; cd "$(mktemp -d)" && OS="$(uname | tr [:upper:] [:lower:])" && ARCH="$(uname -m | sed -e s/x86_64/amd64/ -e s/\(arm\)\(64\)\?.*/\1\2/ -e s/aarch64$/arm64/)" && KREW="krew-${OS}_${ARCH}" && curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" && tar zxvf "${KREW}.tar.gz" && ./"${KREW}" install krew ) +-zsh:104> mktemp -d +-zsh:104> cd /var/folders/sn/6hnm23ys3hg28xppsnk5r9tm0000gn/T/tmp.6Vi5Albr +-zsh:105> OS=+-zsh:105> uname +-zsh:105> OS=+-zsh:105> tr [:upper:] [:lower:] +-zsh:105> OS=darwin +-zsh:106> ARCH=+-zsh:106> uname -m +-zsh:106> ARCH=+-zsh:106> sed -e s/x86_64/amd64/ -e s/\(arm\)\(64\)\?.*/\1\2/ -e s/aarch64$/arm64/ +-zsh:106> ARCH=arm64 +-zsh:107> KREW=krew-darwin_arm64 +-zsh:108> curl -fsSLO https://github.com/kubernetes-sigs/krew/releases/latest/download/krew-darwin_arm64.tar.gz +-zsh:109> tar zxvf krew-darwin_arm64.tar.gz x ./LICENSE x ./krew-darwin_arm64 +-zsh:110> ./krew-darwin_arm64 install krew Adding "default" plugin index from https://github.com/kubernetes-sigs/krew-index.git. Updated the local copy of plugin index. Installing plugin: krew Installed plugin: krew \ | Use this plugin: | kubectl krew | Documentation: | https://krew.sigs.k8s.io/ | Caveats: | \ | | krew is now installed! To start using kubectl plugins, you need to add | | krews installation directory to your PATH: | | | | * macOS/Linux: | | - Add the following to your ~/.bashrc or ~/.zshrc: | | export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH" | | - Restart your shell. | | | | * Windows: Add %USERPROFILE%\.krew\bin to your PATH environment variable | | | | To list krew commands and to get help, run: | | $ kubectl krew | | For a full list of available plugins, run: | | $ kubectl krew search | | | | You can find documentation at | | https://krew.sigs.k8s.io/docs/user-guide/quickstart/. | / /根据 krew 申明加进 krew 加拆目次到 PATH
* macOS/Linux: - Add the following to your ~/.bashrc or ~/.zshrc: export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH" - Restart your shell.查抄能否加拆胜利
p.zhang@pzhang-Mac ~ kubectl krew list PLUGIN VERSION krew v0.4.3 加拆 sniff p.zhang@pzhang-Mac ~ kubectl krew install sniff Updated the local copy of plugin index. Installing plugin: sniff Installed plugin: sniff \ | Use this plugin: | kubectl sniff | Documentation: | https://github.com/eldadru/ksniff | Caveats: | \ | | This plugin needs the following programs: | | * wireshark (optional, used for live capture) | / / WARNING: You installed plugin "sniff" from the krew-index plugin repository. These plugins are not audited for security by the Krew maintainers. Run them at your own risk.查抄 sniff 能否加拆胜利
p.zhang@pzhang-Mac ~ kubectl krew list PLUGIN VERSION krew v0.4.3 sniff v1.6.2如许他们就做好了Tourbe前的筹办工做。
Tourbe前排错在 MacOS 控造系统现实的操做操做过程中，我发原有以下常见的错误。
p.zhang@pzhang-Mac ~ kubectl sniff deployment-xxx-9495f78b5-nnb4n -n testing INFO[0000] using tcpdump path at: /Users/p.zhang/.krew/store/sniff/v1.6.2/static-tcpdump INFO[0000] no container specified, taking first container we found in pod. INFO[0000] selected container: python-xxxx INFO[0000] sniffing method: upload static tcpdump INFO[0000] sniffing on pod: deployment-xxxxx-9495f78b5-nnb4n [namespace: testing, container: python-xxxx, filter: , interface: any] INFO[0000] uploading static tcpdump binary from: /Users/p.zhang/.krew/store/sniff/v1.6.2/static-tcpdump to: /tmp/static-tcpdump INFO[0000] uploading file: /Users/p.zhang/.krew/store/sniff/v1.6.2/static-tcpdump to /tmp/static-tcpdump on container: python-xxx INFO[0000] executing command: [/bin/sh -c test -f /tmp/static-tcpdump] on container: python-xxx, pod: deployment-xxx-9495f78b5-nnb4n, namespace: testing INFO[0001] command: [/bin/sh -c test -f /tmp/static-tcpdump] executing successfully exitCode: 1, stdErr : INFO[0001] file not found on: /tmp/static-tcpdump, starting to upload INFO[0002] verifying file uploaded successfully INFO[0002] executing command: [/bin/sh -c test -f /tmp/static-tcpdump] on container: python-xxxx, pod: deployment-xxxx-9495f78b5-nnb4n, namespace: testing INFO[0002] command: [/bin/sh -c test -f /tmp/static-tcpdump] executing successfully exitCode: 0, stdErr : INFO[0002] file found: INFO[0002] file uploaded successfully INFO[0002] tcpdump uploaded successfully INFO[0002] spawning wireshark! INFO[0002] starting sniffer cleanup INFO[0002] sniffer cleanup completed successfully Error: exec: "wireshark": executable file not found in $PATH那里呈现的 Error 信息，意思 PATH 没有 wireshark，那么他们必要先确认 wireshark 能否加拆胜利，若是加拆胜利。那么必要展开以下操做来处理上述错误：
p.zhang@pzhang-Mac ~ cd /usr/local/bin/ p.zhang@pzhang-Mac /usr/local/bin sudo vim wireshark Password: #!/usr/bin/env bash WIRESHARK="/Applications/Wireshark.app/Contents/MacOS/Wireshark" ${WIRESHARK} "$@" exit $? **##必要将以上内容填入 /usr/local/bin/wireshark 那个文档中。** p.zhang@pzhang-Mac /usr/local/bin sudo chmod +x wireshark p.zhang@pzhang-Mac /usr/local/bin cat wireshark #!/usr/bin/env bash WIRESHARK="/Applications/Wireshark.app/Contents/MacOS/Wireshark" ${WIRESHARK} "$@" exit $? 实时Tourbe一切筹办停当，上面他们起头实时Tourbe：
p.zhang@pzhang-Mac ~ kubectl sniff deployment-xxxx-9495f78b5-nnb4n -n testing INFO[0000] using tcpdump path at: /Users/p.zhang/.krew/store/sniff/v1.6.2/static-tcpdump INFO[0000] no container specified, taking first container we found in pod. INFO[0000] selected container: python-xxxx INFO[0000] sniffing method: upload static tcpdump INFO[0000] sniffing on pod: deployment-xxxx-9495f78b5-nnb4n [namespace: testing, container: python-xxxx, filter: , interface: any] INFO[0000] uploading static tcpdump binary from: /Users/p.zhang/.krew/store/sniff/v1.6.2/static-tcpdump to: /tmp/static-tcpdump INFO[0000] uploading file: /Users/p.zhang/.krew/store/sniff/v1.6.2/static-tcpdump to /tmp/static-tcpdump on container: python-xxxx INFO[0000] executing command: [/bin/sh -c test -f /tmp/static-tcpdump] on container: python-xxxx, pod: deployment-xxxxx-9495f78b5-nnb4n, namespace: testing INFO[0000] command: [/bin/sh -c test -f /tmp/static-tcpdump] executing successfully exitCode: 0, stdErr : INFO[0000] file found: INFO[0000] file was already found on remote pod INFO[0000] tcpdump uploaded successfully INFO[0000] spawning wireshark! INFO[0000] start sniffing on remote container INFO[0000] executing command: [/tmp/static-tcpdump -i any -U -w - ] on container: python-xxxx, pod: deployment-xxxx-9495f78b5-nnb4n, namespace: testing INFO[0024] starting sniffer cleanup INFO[0024] sniffer cleanup completed successfully那时会主动弹出 Wireshark 界面，接下来就能够展开阐发了。
保留Tourbe到文档当然他们还能够Tourbe的内容保留在 pcap 文档中，只必要参加 -o 参数：
p.zhang@pzhang-Mac ~ kubectl sniff deployment-xxxxx-9495f78b5-nnb4n -n testing -o test.pcap INFO[0000] using tcpdump path at: /Users/p.zhang/.krew/store/sniff/v1.6.2/static-tcpdump INFO[0000] no container specified, taking first container we found in pod. INFO[0000] selected container: python-xxxx INFO[0000] sniffing method: upload static tcpdump INFO[0000] sniffing on pod: deployment-xxxx-9495f78b5-nnb4n [namespace: testing, container: python-xxxxx, filter: , interface: any] INFO[0000] uploading static tcpdump binary from: /Users/p.zhang/.krew/store/sniff/v1.6.2/static-tcpdump to: /tmp/static-tcpdump INFO[0000] uploading file: /Users/p.zhang/.krew/store/sniff/v1.6.2/static-tcpdump to /tmp/static-tcpdump on container: python-xxxx INFO[0000] executing command: [/bin/sh -c test -f /tmp/static-tcpdump] on container: python-xxxxx, pod: deployment-xxx-9495f78b5-nnb4n, namespace: testing INFO[0000] command: [/bin/sh -c test -f /tmp/static-tcpdump] executing successfully exitCode: 0, stdErr : INFO[0000] file found: INFO[0000] file was already found on remote pod INFO[0000] tcpdump uploaded successfully INFO[0000] output file option specified, storing output in: test.pcap INFO[0000] start sniffing on remote container INFO[0000] executing command: [/tmp/static-tcpdump -i any -U -w - ] on container: python-xxxx, pod: deployment-dxxxr-9495f78b5-nnb4n, namespace: testing之后他们把保留下来的 test.pcap 文档载入到 Wireshark，就能够展开阐发啦。
Tourbe内容到尺度输入若是他们Tourbe内容间接打印到尺度输入 (stdout)，那么他们只必要接纳 -o - 并通过管道，用其他包阐发辅助东西，例如 tshark 来显示到尺度输入 (stdout)。
不外在macOS 上你必要先施行以下的指示，才能够找到 tshark 指示。
p.zhang@pzhang-Mac ~ ln -s /Applications/Wireshark.app/Contents/MacOS/tshark /usr/local/bin/tshark p.zhang@pzhang-Mac ~ tshark --version TShark (Wireshark) 3.6.5 (v3.6.5-0-g21f79ddbefbd) Copyright 1998-2022 Gerald Combs <gerald@wireshark.org> and contributors. License GPLv2+: GNU GPL version 2 or later <https://www.gnu.org/licenses/gpl-2.0.html> This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Compiled (64-bit) using Clang 13.0.0 (clang-1300.0.29.30), with libpcap, without POSIX capabilities, with GLib 2.68.4, with zlib 1.2.11, with Lua 5.2.4, with GnuTLS 3.6.15 and PKCS #11 support, with Gcrypt 1.8.7, with MIT Kerberos, with MaxMind DB resolver, with nghttp2 1.46.0, with brotli, with LZ4, with Zstandard, with Snappy, with libxml2 2.9.9, with libsmi 0.4.8. Running on macOS 12.3.1, build 21E258 (Darwin 21.4.0), with 16384 MB of physical memory, with GLib 2.68.4, with zlib 1.2.11, with libpcap 1.9.1, with c-ares 1.15.0, with GnuTLS 3.6.15, with Gcrypt 1.8.7, with nghttp2 1.46.0, with brotli 1.0.9, with LZ4 1.9.2, with Zstandard 1.4.2, with libsmi 0.4.8, with LC_TYPE=zh_CN.UTF-8, binary plugins supported (0 loaded).上面起头在尺度输入中实时显示Tourbe的内容
p.zhang@pzhang-Mac ~ kubectl sniff deployment-xxxx-9495f78b5-nnb4n -n testing -o - | tshark -r - INFO[0000] using tcpdump path at: /Users/p.zhang/.krew/store/sniff/v1.6.2/static-tcpdump INFO[0000] no container specified, taking first container we found in pod. INFO[0000] selected container: python-xxxx INFO[0000] sniffing method: upload static tcpdump INFO[0000] sniffing on pod: deployment-xxxx-9495f78b5-nnb4n [namespace: testing, container: python-xxxx, filter: , interface: any] INFO[0000] uploading static tcpdump binary from: /Users/p.zhang/.krew/store/sniff/v1.6.2/static-tcpdump to: /tmp/static-tcpdump INFO[0000] uploading file: /Users/p.zhang/.krew/store/sniff/v1.6.2/static-tcpdump to /tmp/static-tcpdump on container: python-xxxx INFO[0000] executing command: [/bin/sh -c test -f /tmp/static-tcpdump] on container: python-xxxxx, pod: deployment-xxxxx-9495f78b5-nnb4n, namespace: testing INFO[0001] command: [/bin/sh -c test -f /tmp/static-tcpdump] executing successfully exitCode: 0, stdErr : INFO[0001] file found: INFO[0001] file was already found on remote pod INFO[0001] tcpdump uploaded successfully INFO[0001] output file option specified, storing output in: - INFO[0001] start sniffing on remote container INFO[0001] executing command: [/tmp/static-tcpdump -i any -U -w - ] on container: python-xxxx, pod: deployment-xxxxx-9495f78b5-nnb4n, namespace: testing 1 0.000000 172.25.1.216 → 172.26.240.10 DNS 88 Standard query 0x73c9 A collector-005.newrelic.com 2 0.000613 172.26.240.10 → 172.25.1.216 DNS 271 Standard query response 0x73c9 A collector-005.newrelic.com CNAME tls10.newrelic.com.cdn.cloudflare.net A 162.247.243.148 A 162.247.243.149 3 0.001060 172.25.1.216 → 162.247.243.148 TCP 76 37732 → 443 [SYN] Seq=0 Win=42600 Len=0 MSS=1420 SACK_PERM=1 TSval=1031788613 TSecr=0 WS=128 4 0.004524 162.247.243.148 → 172.25.1.216 TCP 68 443 → 37732 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 SACK_PERM=1 WS=1024 5 0.004563 172.25.1.216 → 162.247.243.148 TCP 56 37732 → 443 [ACK] Seq=1 Ack=1 Win=42624 Len=0 6 0.008498 172.25.1.216 → 162.247.243.148 TLSv1 573 Client Hello 7 0.011971 162.247.243.148 → 172.25.1.216 TCP 56 443 → 37732 [ACK] Seq=1 Ack=518 Win=68608 Len=0 8 0.015108 162.247.243.148 → 172.25.1.216 TLSv1.2 3417 Server Hello, Certificate, Server Key Exchange, Server Hello Done 9 0.015163 172.25.1.216 → 162.247.243.148 TCP 56 37732 → 443 [ACK] Seq=518 Ack=3362 Win=41088 Len=0 10 0.016263 172.25.1.216 → 162.247.243.148 TLSv1.2 149 Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message 11 0.019668 162.247.243.148 → 172.25.1.216 TCP 56 443 → 37732 [ACK] Seq=3362 Ack=611 Win=68608 Len=0 12 0.019765 162.247.243.148 → 172.25.1.216 TLSv1.2 314 New Session Ticket, Change Cipher Spec, Encrypted Handshake Message 13 0.019777 172.25.1.216 → 162.247.243.148 TCP 56 37732 → 443 [ACK] Seq=611 Ack=3620 Win=42368 Len=0 14 0.020645 172.25.1.216 → 162.247.243.148 TLSv1.2 1430 Application Data 15 0.020726 172.25.1.216 → 162.247.243.148 TLSv1.2 989 Application Data 16 0.024135 162.247.243.148 → 172.25.1.216 TCP 56 443 → 37732 [ACK] Seq=3620 Ack=1985 Win=71680 Len=0 17 0.024376 162.247.243.148 → 172.25.1.216 TCP 56 443 → 37732 [ACK] Seq=3620 Ack=2918 Win=74752 Len=0 18 0.145180 172.25.1.216 → 172.26.240.10 DNS 88 Standard query 0x68f4 A collector-005.newrelic.com 19 0.145821 172.26.240.10 → 172.25.1.216 DNS 271 Standard query response 0x68f4 A collector-005.newrelic.com CNAME tls10.newrelic.com.cdn.cloudflare.net A 162.247.243.148 A 162.247.243.149 20 0.146091 172.25.1.216 → 162.247.243.148 TCP 76 37748 → 443 [SYN] Seq=0 Win=42600 Len=0 MSS=1420 SACK_PERM=1 TSval=1031788758 TSecr=0 WS=128 21 0.149906 162.247.243.148 → 172.25.1.216 TCP 68 443 → 37748 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 SACK_PERM=1 WS=1024 22 0.150021 172.25.1.216 → 162.247.243.148 TCP 56 37748 → 443 [ACK] Seq=1 Ack=1 Win=42624 Len=0 23 0.154189 172.25.1.216 → 162.247.243.148 TLSv1 573 Client Hello 24 0.157981 162.247.243.148 → 172.25.1.216 TCP 56 443 → 37748 [ACK] Seq=1 Ack=518 Win=68608 Len=0 25 0.161018 162.247.243.148 → 172.25.1.216 TLSv1.2 3417 Server Hello, Certificate, Server Key Exchange, Server Hello Done 26 0.161070 172.25.1.216 → 162.247.243.148 TCP 56 37748 → 443 [ACK] Seq=518 Ack=3362 Win=41088 Len=0 27 0.162097 172.25.1.216 → 162.247.243.148 TLSv1.2 149 Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message 28 0.165888 162.247.243.148 → 172.25.1.216 TCP 56 443 → 37748 [ACK] Seq=3362 Ack=611 Win=68608 Len=0 29 0.166033 162.247.243.148 → 172.25.1.216 TLSv1.2 314 New Session Ticket, Change Cipher Spec, Encrypted Handshake Message 30 0.166052 172.25.1.216 → 162.247.243.148 TCP 56 37748 → 443 [ACK] Seq=611 Ack=3620 Win=42368 Len=0 31 0.166709 172.25.1.216 → 162.247.243.148 TLSv1.2 1430 Application Data 32 0.166772 172.25.1.216 → 162.247.243.148 TLSv1.2 988 Application Data 33 0.170364 162.247.243.148 → 172.25.1.216 TCP 56 443 → 37748 [ACK] Seq=3620 Ack=1985 Win=71680 Len=0 34 0.170402 162.247.243.148 → 172.25.1.216 TCP 56 443 → 37748 [ACK] Seq=3620 Ack=2917 Win=74752 Len=0 总结以上即是今天禀享的内容，通过 ksniff 他们能够很轻松在 Kubernetes 中展开简单，高效的Tourbe操做。后续再分享如何接纳 kubectl-debug 来Tourbe。
参考内容https://github.com/eldadru/ksniff
https://github.com/eldadru/ksniff/issues/96#issuecomment-762454991
https://github.com/JamesTGrant/kubectl-debug
https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/
0