Packet Capture & Network Troubleshooting

A packet capture is an observation made at one place in one software/hardware path. It is not “what happened on the network.” The same exchange can look different before segmentation offload, after NAT, inside a tunnel, on the physical link, and after receive coalescing.

Reliable troubleshooting therefore starts with a falsifiable symptom and places observations at boundaries that distinguish competing causes. The goal is not to collect the largest pcap; it is to make the fault domain shrink after each safe test.

Write the symptom as a testable claim #

“The network is slow” cannot be disproved. A useful symptom record states:

• affected source and destination identities, including namespace, VRF, VLAN, and address family where relevant;
• application operation and transport tuple, not only a hostname;
• direction, start/end time in UTC, frequency, and last known good time;
• expected result and observed result;
• payload/packet size, concurrency, and load dependence;
• whether all clients, one path, one rack, one protocol, or one direction is affected;
• the exact probe and its side effects.

Example: “From pod namespace payments-7f9, TCP connections to service address 2001:db8:20::8 port 443 complete the three-way handshake, but the first response carrying more than 1380 bytes is retransmitted until timeout between 14:02:10Z and 14:04:00Z. Requests with a 512-byte response succeed.” That statement immediately separates name resolution and handshake failure from a size-dependent data-path fault.

Start with hypotheses that predict different evidence #

A hypothesis is useful only if it predicts an observation that alternatives do not. For the example above:

Do not change MTU, firewall rules, routes, and application timeouts together. That can restore service while destroying the evidence needed to identify cause and prevent recurrence.

Observation points define what a capture can prove #

At least six transformations can sit between an application and the medium:

1. socket and transport state create bytes, segments, retransmissions, and acknowledgements;
2. routing and policy select an egress and may reject traffic;
3. firewall, NAT, proxy, or service-mesh logic can modify addresses, ports, or connections;
4. tunnel encapsulation creates outer headers with a different path and MTU;
5. segmentation/checksum offload defers work from the kernel to the NIC;
6. the receive path can validate, strip, coalesce, decapsulate, or discard before a capture hook sees data.

Name every capture point precisely: client namespace eth0 egress before NAT, switch mirror of physical port 17 both directions, or server host bond0 receive after decapsulation. “Captured on the client” is too vague when the client contains a pod veth, host bridge, overlay device, bond, and physical NIC.

Compare boundaries, not screenshots #

For one test interval, define a stable flow key and compare the same protocol event at two points:

Absence is evidence only after ruling out an inadequate capture: wrong interface or direction, too-narrow capture filter, short snap length, buffer overflow, unsupported monitor/mirror behavior, sampling, or an observation hook after the discard.

Packet identity also changes. NAT rewrites tuples; a tunnel's outer packet is not keyed like its inner packet; segmentation offload turns one large kernel buffer into several wire packets; receive coalescing can combine several wire packets. Correlate invariant fields and sequence ranges rather than demanding byte-for-byte equality across every boundary.

Capture filters and display filters have different failure modes #

Wireshark and tcpdump-style workflows use two different filter stages:

• a capture filter decides which packets are admitted to the capture buffer and file;
• a display filter hides or shows records already captured.

A narrow capture filter reduces volume and sensitive data but irreversibly removes context. A display filter is safer during exploration because the original records remain. Begin with the smallest filter that still preserves the hypotheses: both directions, required control protocols, and related ICMP errors. Filtering only tcp port 443, for example, can omit ICMP Packet Too Big messages needed to explain a stalled TLS connection.

An explicit command records intent better than a GUI screenshot:

tcpdump -i eth0 -nn -s 0 -B 4096 -w incident.pcapng \
  'host 192.0.2.10 and (tcp port 443 or icmp or icmp6)'

The interface, numeric-name behavior, snap length, buffer size, output format support, filter, tool version, start/end time, and reported dropped-packet count belong with the artifact. Adapt the command to the platform; do not assume every tcpdump build writes pcapng or exposes the same buffer semantics.

Snap length, buffering, and time #

Snap length limits bytes retained from each captured packet. Header-only capture may answer routing and handshake questions while protecting payload, but it can hide encapsulation fields, TLS records, application messages, or padding required by a hypothesis. -s 0 requests complete packets in common tcpdump versions, but the actual capture path can still truncate.

A capture has two loss channels:

• the network or host drops a packet under investigation;
• the capture process drops a record because its buffer or storage path cannot keep up.

Record interface counters and capture-tool drop statistics before and after. If the capture itself drops records, a sequence gap is ambiguous until another point or lower-volume reproduction confirms network loss.

Timestamps are observations too. Precision is not accuracy: nanosecond fields do not guarantee synchronized clocks. For one-way delay across hosts, establish clock synchronization and error bounds. Without that, use causal sequence—SYN before SYN-ACK, request before response—and round-trip measurements from one clock. pcapng can retain per-interface metadata and timestamp resolution, so preserve the original file rather than exporting only decoded text.

Offload artifacts that imitate faults #

Modern hosts deliberately let NICs perform work after a host capture hook:

• checksum offload: an outbound capture can show an incomplete or “bad” TCP/UDP checksum that hardware fixes before transmission;
• TCP segmentation offload / generic segmentation offload: one large outbound buffer appears in the host capture but becomes several MTU-sized wire packets;
• generic/large receive offload: several received wire packets can appear as one larger host-side unit;
• VLAN acceleration: tags can be stripped into metadata on receive or inserted later on transmit, so the capture bytes do not always show the on-wire tag;
• Ethernet FCS handling: bad-FCS frames are commonly discarded in hardware and valid FCS bytes stripped before host capture.

Wireshark can recognize partial checksums on supported platforms, but a red checksum annotation is not a root cause by itself. Compare with a capture beyond the NIC, inspect offload state, and check whether only locally generated packets show the pattern. Disabling offload changes runtime behavior and performance; do it only as a bounded diagnostic change with a recorded before/after condition.

Reading a TCP failure as state transitions #

TCP sequence and acknowledgement numbers describe a state machine, not a collection of colored rows. For connection establishment:

1. client sends SYN with initial sequence number $x$;
2. server replies SYN-ACK with its initial sequence $y$ and acknowledgement $x+1$;
3. client acknowledges $y+1$.

Two-sided captures turn missing transitions into location evidence:

Retransmission is evidence that the sender did not receive an acceptable acknowledgement in time. It does not prove the original packet was lost: the data, the ACK, or processing time could be responsible. Duplicate ACKs likewise describe receiver observations; interpret them with sequence ranges and captures from both directions.

Size-dependent failure and path MTU #

An MTU fault often passes small probes and connection handshakes. The first large data packet then disappears because some link cannot carry it and the control message needed for Path MTU Discovery is filtered or not delivered. RFC 8201 explicitly describes IPv6 connections that complete the TCP handshake and then hang when data is transferred under this condition.

Test the mechanism, not folklore:

1. record interface and tunnel MTUs along the intended path;
2. vary packetization size while holding destination, path, and load stable;
3. capture the large packet before and after the suspected boundary;
4. include ICMPv4 fragmentation-needed and ICMPv6 Packet Too Big messages in the filter;
5. verify whether the transport reduces packet size or repeats the same failing range;
6. confirm the final configuration with real application traffic in both directions.

Changing TCP MSS can mask one manifestation without correcting non-TCP traffic, tunnel overhead, or blocked control messages. Treat it as a specific mechanism only when the complete path model justifies it.

Use the cheapest discriminating evidence first #

Packet capture is powerful but not always first. The evidence ladder is:

Telemetry gives breadth and packet evidence gives detail. A counter spike tells where to capture; a capture tells which conversation contributes to the spike. Neither should be forced to answer the other's question.

Prove the repair #

A repair is established only when the original falsifiable test changes as predicted and adjacent failure modes remain absent.

1. preserve the pre-change state and capture metadata;
2. apply one authorized change;
3. repeat the same source, destination, size, direction, and load;
4. compare success rate, latency distribution, retransmissions, drops, and relevant state transitions;
5. test the reverse direction and one unaffected control case;
6. explain why the change caused the observed difference;
7. remove any diagnostic-only capture, mirror, or offload change and confirm behavior again;
8. record the prevention mechanism—validation, monitoring, capacity, configuration ownership, or topology correction.

“Ping works now” does not prove a repaired application path if the original failure involved TCP state, a larger MTU, a different address family, or another namespace.

Artifact hygiene and privacy #

Packet captures can contain credentials, cookies, personal data, internal addresses, and plaintext application payload. Capture only what the hypothesis requires, restrict access, preserve the original hash and metadata, and create derived/redacted views for sharing. Do not replace the original with a lossy export: later analysis may need timing, interface, truncation, or encapsulation information absent from screenshots.

Common reasoning traps #

• “I captured on the host, so this is the wire.” Offload, filtering, tunnelling, and the capture hook can all intervene.
• “Wireshark says bad checksum, so the network corrupted it.” Locally transmitted offloaded packets commonly look incomplete before the NIC fixes them.
• “No packet in the pcap means it never arrived.” It may have hit another interface, failed the capture filter, been dropped before the hook, or been lost by the capture process.
• “Traceroute shows the path.” It shows responses to particular probes; load balancing, filtering, and return paths can differ from the application flow.
• “A retransmission proves forward loss.” The data or its acknowledgement may be missing, delayed, or rejected.
• “One successful small ping clears the network.” It tests one address family, size, protocol, path selection, and moment.
• “Changing several knobs is faster.” It can restore service while leaving cause unproven and recurrence likely.