Skip to content

Latest commit

 

History

3 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 

Repository files navigation

dpitool

A deep packet inspection tool for Linux. Version 0.6.0 — complete and in use: a sniffer with flow tracking, protocol classification, reference-data enrichment, TCP stream reassembly, application metadata extraction, file carving and multi-worker scale-out, built around a capture-backend seam that AF_XDP or DPDK can plug into if AF_PACKET ever runs out of headroom.

It links nothing but libc and pthreads. No libpcap, no nDPI, no libmaxminddb — the classifier, the MaxMind reader and the SHA-256 are all in this tree, which is why make on a bare toolchain is the whole build.

What it does

Capture

  • AF_PACKET (TPACKET_V3) mmap-ring capture on a named interface
  • L2–L4 decode: Ethernet, 802.1Q/QinQ VLAN, ARP, IPv4, IPv6, TCP, UDP, ICMP/ICMPv6, IPv6 extension-header chains, IPv4/IPv6 fragments
  • Live per-packet output, plus a periodic running-stats line
  • Rotating pcap output (opens in Wireshark/tcpdump)
  • Kernel drop accounting (a capture tool that hides drops is lying)
  • A capture_backend seam that AF_XDP / DPDK can plug into

Flows and classification

  • Per-worker 5-tuple flow table: direction-symmetric keys, bidirectional counters, TCP teardown tracking, timeout and capacity eviction
  • Protocol classification with no external library: 31 payload matchers (HTTP+Host, TLS+SNI, DNS/mDNS/LLMNR+qname, SSH, SMB2, DHCP+hostname, QUIC, WireGuard, MySQL, PostgreSQL, MQTT, STUN, SIP, TFTP+filename and more), backed by a well-known port table
  • Every verdict reports its evidencepayload (we saw the protocol speak) or port (we inferred it) — so a guess never reads as a fact
  • Rotating JSONL flow records alongside the pcap
  • No per-packet allocation: flows and DPI state come from memory pools

Enrichment

  • MAC vendor (IEEE OUI), service name (IANA), country/city/coordinates and AS number/organization attached to every flow record
  • A from-scratch MaxMind DB (.mmdb) reader — no libmaxminddb, no other dependency; the file is mmap'd read-only and shared across workers
  • update-db.sh: fetch, verify, convert, then promote atomically via a current -> symlink, keeping the previous version for --rollback
  • Every source is optional. A missing database means fewer columns, never a failed capture

Reassembly and extraction

  • Per-flow, per-direction TCP reassembly: out-of-order segments, retransmits, overlaps, sequence wraparound, and abandoned holes reported rather than hidden
  • Application metadata from the reassembled stream, as a stream of events:
    • HTTP — method, URL, Host, User-Agent, status, server, content type, response length, elapsed time; one event per transaction, keep-alive and chunked bodies stepped over exactly
    • TLS — SNI, ALPN, negotiated version, cipher, and (through TLS 1.2) the certificate subject, issuer and validity, parsed from DER
    • DNS — query name and type, response code, and decoded A/AAAA/CNAME/ PTR/MX answers, over both UDP and TCP
    • FTP — logins and file transfers with their status; passwords are never recorded
    • SMTP — HELO identity, envelope sender and recipients, acceptance
  • Bounded by construction: a capped pool of reassembly windows, released the moment a flow has nothing more to say, and flows nothing recognizes are dropped immediately rather than followed

Carving

  • Files recovered from cleartext streams into <PREFIX>.files/, with a "type":"file" record giving the path, size, SHA-256 and sniffed content type
  • Sources: HTTP response bodies and uploads (Content-Length and chunked), SMTP message bodies as .eml, and FTP data connections — correlated to their transfer by tracking the PASV/EPSV announcement on the control channel
  • SHA-256 implemented from scratch, so the tool still links nothing but libc
  • Off unless you ask for it (-C), hard-capped per object, in total and by count, and every name from the wire is reduced to one safe component

Scale-out

  • -w N capture threads, each with its own AF_PACKET ring joined to one PACKET_FANOUT group and its own flow table, DPI, reassembly and carving — nothing on the packet path is shared, so nothing on it is locked
  • Both directions of a conversation reach the same worker, which is what makes a per-worker flow table correct; verified on live traffic, not assumed
  • Workers pinned to CPUs by default when there is a core each
  • tune.sh for real: reports and fixes the NIC settings that distort a capture or drop packets before the tool sees them, with --show and --restore
  • --web [ADDR:]PORT serves a live status page — rates and their history, traffic mix, top protocols, per-worker split, and the flows and events as they happen. Off unless asked for, loopback-only unless told otherwise, and compiled into the binary with nothing to install

Application inference

  • What was this traffic for?, answered from the names the other stages already recover: the SNI or Host a flow carried, or a name seen resolving to the address it went to, then the AS organization, then the port
  • A DNS reverse map learned from the answers this capture parses, so a bare TLS connection to an address with no SNI is still attributed to the name that produced it — the single highest-value signal once everything is encrypted
  • About 400 name rules across 23 categories (streaming, messaging, gaming, updates, ads and telemetry, remote access…), plus AS-organization, User-Agent, port and protocol tables — all data, in src/appid/apptable.c
  • Every verdict carries its evidence, and a recognized application is kept distinct from a name merely reduced to its registrable domain. Inference is never presented as identification

Build

No external dependencies — just a C toolchain.

make

Test

The pipeline above the capture backend is verified offline, so it needs no interface and no privileges:

make test        # decode / dpi / protocol matchers / flow / jsonl / mmdb /
                 # enrich / reasm / extract (http tls dns ftp smtp) / sha256 /
                 # carve / appid / web ui / concurrency, plus three fuzz passes
make test-asan   # the same suite under AddressSanitizer + UBSan
make test-tsan   # the same suite under ThreadSanitizer

A clean run reports 388 checks, 0 failures.

The MaxMind reader is checked against a database the test builds itself with a hand-written encoder, so the reader is not merely agreeing with its own author's reading of the format. It was additionally confirmed against real GeoLite2 and DB-IP files. The TLS certificate parser is checked the same way, against a structurally valid DER certificate the test assembles.

The second fuzz pass matters more than the first. Random bytes almost never get past parser dispatch, so they never reach the code that walks a header block, a TLS extension list or a DER structure. That pass starts from traffic each parser accepts and then corrupts it, which is what puts pressure on the length arithmetic where an out-of-bounds read would actually live.

make test-asan is the run that matters: both fuzz passes drive malformed input through decode, DPI, flow tracking, reassembly and every extractor, which is where an out-of-bounds read surfaces.

The AF_PACKET backend is the one thing this cannot cover — it needs CAP_NET_RAW and a real interface. That gap is closed by a separate script, which generates known traffic on loopback (HTTP, DNS, and a TLS 1.2 handshake with a self-signed certificate if openssl is present), captures it, and checks that the flows, the enrichment and the extracted events all came back:

sudo ./scripts/livetest.sh

It cleans up after itself and prints a PASS/INCOMPLETE verdict along with the records it recovered.

Run

Packet capture needs privilege. Either run as root, or grant the binary the capabilities once:

make setcap        # sudo setcap cap_net_raw,cap_net_admin+eip ./dpitool
./dpitool -i eth0

Capture packets and flow records while watching it live:

./dpitool -i eth0 -o /tmp/capture
# -> /tmp/capture.pcap   (rotating raw capture)
# -> /tmp/capture.jsonl  (flow records and extracted events, one per line)

At high rates, drop the per-packet scroll and watch flows and application events instead:

./dpitool -i eth0 -q -F -E --stats-interval 2

Recover files from cleartext streams as well:

./dpitool -i eth0 -o /tmp/capture -C
# -> /tmp/capture.files/000001-http-logo.png

On a fast link, give it a thread per core:

./dpitool -i eth0 -o /tmp/capture -q -w 0     # 0 = one worker per CPU

Only ever capture on networks you're authorized to monitor.

Live status page

--web starts a small HTTP server alongside the capture. It is off by default; passing the flag is the only thing that starts it.

./dpitool -i eth0 -q --web 8080          # http://127.0.0.1:8080/
./dpitool -i eth0 -q --web 10.0.0.5:8080 # a specific address
./dpitool -i eth0 -q --web 0             # let the kernel pick the port

The page shows throughput, packet rate and kernel drops with the last few minutes of history for each; the applications the traffic was for and the activity they add up to; the local endpoints doing it and what each one was mostly up to; anything worth a look (cleartext logins, obsolete TLS, file sharing leaving the network); the traffic mix by transport and address family; the protocol breakdown; the per-worker packet split; the pipeline counters; and the flows and extracted events as they happen. It refreshes once a second and holds the last render while a refresh is in flight, so nothing flashes.

It is read-only and has no authentication. A bare port therefore binds 127.0.0.1 and nothing else; to watch a remote capture, forward the port over ssh rather than exposing it:

ssh -N -L 8080:127.0.0.1:8080 sensor-host

Giving --web an address that is not loopback binds it there and says so at startup, in as many words. Anyone who can reach that port can read hostnames, addresses, URIs and everything else the page displays.

Two routes exist, and both are GET:

Route What it is
/ (also /index.html) the page — one self-contained HTML file compiled into the binary
/api/stats everything the page draws, as JSON

/api/stats is a stable enough shape to script against:

curl -s localhost:8080/api/stats | jq '.totals.drops, .history.pps[-1]'

The server is one thread, handles one connection at a time, and reads the same counters the periodic stats line does — it never touches the packet path. With --web absent it costs nothing at all: no thread, no socket, no branch beyond a null check per flow.

What the traffic was for

Protocol classification answers what a flow speaks. On a network where everything is TLS on 443, that answer is correct and nearly useless: a video stream, a chat client and a software update are all "TLS". So there is a second stage that answers a different question — what was this connection for? — from the names the other stages already recover.

It works down a ladder, strongest evidence first:

Evidence What it means
name the flow carried the name itself — SNI, HTTP Host, or a DNS question
user-agent the client named itself, in cleartext HTTP
dns this capture saw a name resolve to the address the flow went to
as-org the organization that owns the address
port a well-known port, and nothing better
protocol only the protocol verdict, given a category

The dns rung is the one that matters most. dpitool already parses DNS answers, so it keeps a bounded map of address → the name that resolved to it. A later TLS or QUIC connection carrying no SNI at all still comes back as YouTube, because something on this network asked for googlevideo.com and got that address. The map holds 8192 entries, is shared by every worker, and costs one lock per DNS answer and one per retired flow.

Two honesty rules are built in:

  • Inference never masquerades as identification. Every verdict reports the rung it came from, in the JSONL and on the page.
  • A recognized application is distinct from a reduced name. If nothing in the tables matches, the name is cut back to its registrable domain and reported as that, flagged "appid_known":false. Netflix is a claim; example-corp.net is just where the packets went.

The tables — 399 domain suffixes across 23 categories, plus 52 AS-organization, 43 User-Agent, 71 port and 36 protocol rules — are plain data in src/appid/apptable.c. Adding an application is adding a row. They are a heuristic and they will age; that is why they are sorted by category and commented rather than generated.

Turn the whole stage off with --no-appid.

# what was this capture actually doing?
jq -r 'select(.type=="flow" and .appid) | [.appid_category, .appid] | @tsv' \
   /tmp/capture.jsonl | sort | uniq -c | sort -rn

# only the confident answers
jq -r 'select(.appid_known and .appid_evidence=="name") | .appid' \
   /tmp/capture.jsonl | sort -u

# which local host talked to what
jq -r 'select(.type=="flow" and .appid) | [.src_ip, .appid] | @tsv' \
   /tmp/capture.jsonl | sort | uniq -c | sort -rn | head

Scaling up

-w N runs N capture threads. Each gets its own AF_PACKET ring, joined to a single PACKET_FANOUT group so the kernel spreads packets across them, and its own flow table, DPI engine, reassembly windows and carve writer. Nothing on the packet path is shared between workers, so nothing on it needs a lock.

Two things are shared, and the asymmetry is the point:

why
<PREFIX>-wN.pcap, one per worker written per packet — a lock here would sit in the hot path
<PREFIX>.jsonl, single and locked written per flow and per event, rare enough that a mutex costs nothing, and one file is far more useful than N

Merge the pcaps when you need one file: mergecap -w all.pcap /tmp/capture-w*.pcap.

Both directions of a flow must reach the same worker, or each worker sees half a conversation and reassembly has nothing to work with. This holds because the kernel's fanout hash is deliberately direction-independent — the flow dissector canonicalizes the address and port pair before hashing. That is a claim about the kernel rather than about this code, so livetest.sh verifies it against live traffic instead of trusting it: it runs a two-worker capture and asserts that no flow was recorded with traffic in only one direction.

PACKET_FANOUT_FLAG_ROLLOVER is deliberately not used. It moves packets to a less busy worker under load, trading exactly the affinity everything above depends on for slightly less loss.

Before a serious capture, prepare the NIC:

sudo ./scripts/tune.sh --show eth0     # report only, changes nothing
sudo ./scripts/tune.sh eth0            # disable GRO/LRO, grow rings and buffers
sudo ./scripts/tune.sh --restore eth0  # put the offloads back

Disabling GRO/LRO is not an optimization — it is the difference between a capture that describes the network and one that describes the driver. With them on, the kernel hands you coalesced frames that never existed on the wire, and TCP reassembly faithfully reconstructs a stream nobody sent.

Carving

-C writes objects recovered from HTTP bodies, SMTP messages and FTP data connections into <PREFIX>.files/, and adds a "type":"file" record to the JSONL carrying the path, size, SHA-256 and sniffed content type.

It is off by default even when -o is given, which is deliberate: <prefix>.files/ could reasonably have been just another thing an output prefix expands into, alongside the pcap and the JSONL. Carving takes bytes chosen by whoever is on the network and writes them to the local disk under a name partly derived from what they sent, which is a side effect that should be asked for rather than inherited from an unrelated flag.

Everything about the implementation follows from that:

  • The name on the wire is a hint, never a path. It is reduced to a single safe component from a character whitelist, leading dots are dropped, and the result is always prefixed with a counter — so nothing can escape the directory, collide, or overwrite. A fuzz pass throws 5000 hostile names at it.
  • Objects are written under a temporary name and renamed only when complete, so a partial file is never mistaken for a whole one. Anything truncated or cut short keeps a .partial suffix and is flagged in its record.
  • Caps are hard: per object, in total, and by count. A capture cannot fill a disk.

<PREFIX>.files/ holds untrusted content by definition. Treat it the way you would any directory of unknown files.

Reference databases

Enrichment is driven by files fetched with:

./dpitool --update-db          # or: ./scripts/update-db.sh
File Source Adds
oui.tsv IEEE registry src_mac_vendor, dst_mac_vendor
services.tsv IANA port assignments service
GeoLite2-City.mmdb MaxMind country_*, city_*, lat_*, lon_*
GeoLite2-ASN.mmdb MaxMind asn_*, as_org_*

The IEEE and IANA files need nothing but network access. GeoLite2 requires a free MaxMind license key:

MAXMIND_LICENSE_KEY=... ./scripts/update-db.sh
./scripts/update-db.sh --no-geo      # skip them entirely

Each run installs into a new timestamped directory and flips the current symlink only once every source has downloaded and parsed into a plausible table, so an interrupted or rejected update leaves the working set untouched:

./scripts/update-db.sh --status      # what is installed
./scripts/update-db.sh --rollback    # go back to the previous version

Anything after -- is passed straight through, so the script's flags are reachable from the binary too:

./dpitool --update-db -- --status
./dpitool --update-db -- --no-geo

Databases live under ${XDG_DATA_HOME:-~/.local/share}/dpitool/db/current. Override with --db-dir PATH (or $DPITOOL_DB), and turn the whole layer off with --no-enrich.

A DB-IP "lite" .mmdb works in place of GeoLite2 if you'd rather not register: the schema keys are the same, so drop it in under the GeoLite2 filename.

Options

-i, --interface NAME   interface to capture on (required)
-o, --output PREFIX    write <PREFIX>.pcap and <PREFIX>.jsonl (rotating)
-u, --update-db        refresh reference databases and exit
                       (anything after `--` is passed to update-db.sh)
-q, --quiet            suppress per-packet lines (stats only)
-F, --flows            log each flow as it completes
-E, --events           log extracted events (http/tls/dns/ftp/smtp)
-C, --carve            write recovered files to <PREFIX>.files/
-w, --workers N        capture threads, 0 = one per CPU (default 1)
    --web [ADDR:]PORT  serve the live status page (default off; a bare
                       port binds 127.0.0.1, and there is no auth)
    --no-pin           do not pin workers to CPUs
    --no-promisc       do not enable promiscuous mode
    --no-dpi           disable protocol classification
    --no-flows         disable flow tracking (and JSONL output)
    --no-enrich        disable reference-database lookups
    --no-appid         disable application inference
    --no-extract       disable reassembly and metadata extraction
    --carve-max-file N per-object byte cap (default 64 MiB)
    --carve-max-total N total byte cap (default 4 GiB)
    --carve-max-files N object count cap (default 10000)
    --max-reasm N      flows reassembled at once (default 2048)
    --reasm-window N   per-direction reassembly window (default 8192)
    --db-dir PATH      reference database directory
    --frame-size N     ring frame size / capture cap (default 2048)
    --block-size N     ring block size bytes (default 1 MiB)
    --block-nr N       ring block count, per worker (default 64 -> 64 MiB)
    --rotate-bytes N   output rotation size, 0=off (default 1 GiB)
    --stats-interval S seconds between stats lines (default 5)
    --max-flows N      flow table capacity, per worker (default 1000000)
    --tcp-timeout S    idle TCP flow timeout (default 300)
    --udp-timeout S    idle UDP flow timeout (default 120)

Output records

The JSONL file carries two kinds of record, told apart by type. Flow records ("type":"flow") summarize a connection and are written when it is retired (clean TCP teardown, idle timeout, capacity pressure, or shutdown):

{"type":"flow","start":1700000000.0,"end":1700000005.0,"duration":5.0,
 "proto":6,"src_ip":"10.0.0.1","src_port":51000,
 "dst_ip":"93.184.216.34","dst_port":443,
 "pkts_fwd":10,"bytes_fwd":1500,"pkts_rev":8,"bytes_rev":9000,
 "app":"TLS","hostname":"www.example.org","evidence":"payload","confident":true,
 "appid":"Netflix","appid_category":"Streaming video","appid_evidence":"dns",
 "appid_name":"ichnaea.netflix.com","appid_known":true,
 "tcp_flags_fwd":26,"tcp_flags_rev":18,"closed":true,
 "service":"https","country_dst":"US","country_name_dst":"United States",
 "city_dst":"Los Angeles","lat_dst":34.0544,"lon_dst":-118.2441,
 "asn_dst":15133,"as_org_dst":"EDGECAST","src_mac":"00:1b:44:11:22:33",
 "src_mac_vendor":"SanDisk Corporation"}

src/dst are from the initiator's point of view, and fwd/rev follow that same direction. Everything from service onward comes from the reference databases and is present only when a database answered.

app and appid are deliberately different claims. app is the protocol the bytes spoke, decided by the DPI matchers; appid is what the connection was probably for, inferred from names — see above. appid_evidence says which rung of that ladder answered, appid_name is the name it rests on, and appid_known is false when the "application" is really just the destination's own domain. The whole group is absent when nothing matched, and when --no-appid is given.

Event records carry what actually happened on the connection. One flow can produce many, and each is written as it occurs rather than held until the flow ends:

{"type":"http","ts":1700000001.5,"proto":6,"src_ip":"10.0.0.1","src_port":51000,
 "dst_ip":"93.184.216.34","dst_port":80,"method":"GET","uri":"/index.html",
 "host":"www.example.org","user_agent":"curl/8.5.0","status":200,"reason":"OK",
 "content_type":"text/html","server":"nginx","resp_bytes":1256,"elapsed_ms":42}

{"type":"tls","ts":1700000002.1,...,"sni":"www.example.org","alpn":"h2",
 "version":"TLS 1.2","cipher":"ECDHE_RSA_AES128_GCM_SHA256",
 "cert_issuer":"R3","cert_subject":"www.example.org",
 "cert_not_after":"260401120000Z"}

{"type":"dns","ts":1700000000.9,...,"query":"www.example.org.","qtype":"A",
 "rcode":"NOERROR","answers":1,"addresses":"93.184.216.34"}

{"type":"file","ts":1700000001.6,...,"source":"http",
 "path":"/tmp/capture.files/000001-http-logo.png","name":"logo.png",
 "bytes":18244,"sha256":"9f86d081884c7d65...","kind":"png"}

Events get the same endpoint enrichment as flow records. A record built from a stream with a hole in it carries "inexact":true.

Since it's JSONL, it's valid and greppable while still being written:

# every URL fetched
jq -r 'select(.type=="http") | .host + .uri' /tmp/capture.jsonl

# every name looked up, and what it resolved to
jq -r 'select(.type=="dns" and .addresses) | [.query,.addresses] | @tsv' \
   /tmp/capture.jsonl | sort -u

# TLS destinations that are not where you expected
jq -r 'select(.type=="tls") | [.sni,.cert_subject] | @tsv' /tmp/capture.jsonl

# flows to another country
jq -r 'select(.type=="flow" and .country_dst!="US")
       | [.dst_ip,.country_dst,.as_org_dst] | @tsv' /tmp/capture.jsonl | sort -u

Notes

  • Loopback shows each packet twice. Capturing on lo taps both the transmit and receive path of the same host, so packets appear duplicated. This is a loopback artifact, not a bug; real interfaces don't double like this.
  • --frame-size caps captured length. Frames larger than the ring frame size are truncated (flagged [truncated]). Raise it for jumbo frames.
  • Tuning for accuracy. Before serious capture, run sudo ./scripts/tune.sh <iface> to disable receive offloads (GRO/LRO) that would otherwise coalesce segments into frames that never existed on the wire.
  • Classification is still single-packet; extraction is not. The app field on a flow record comes from per-packet matching, so a protocol that only identifies itself across a segment boundary can be missed there. The event records go through reassembly and do see it — if app says Unknown but an http event exists for the same connection, the event is the better answer.
  • Check evidence before trusting app. "evidence":"payload" means the protocol was recognized from the bytes on the wire. "evidence":"port" means only the port number matched, which is exactly what a service on a non-standard port defeats. The live view marks the weaker case with a trailing ? (e.g. MySQL?).
  • Coverage is deliberate, not exhaustive. The classifier recognizes around thirty protocols, chosen for analytic value and for having signatures distinctive enough to match without false positives. Unrecognized traffic is reported as Unknown rather than guessed at. Adding a protocol means adding a matcher in src/dpi/dpi_builtin.c and a case in the test matrix.
  • MAC vendors describe the local hop. On a routed path the Ethernet source address belongs to your gateway, not the peer, so src_mac_vendor identifies hosts on the capture segment and nothing beyond it. Locally administered addresses (the 0x02 bit) are left unattributed rather than matched against a registry they were never in.
  • Geolocation is an estimate. City-level accuracy varies by a lot, VPNs and anycast defeat it outright (the IPv6 resolvers above both geolocate to Montreal), and the coordinates are a database's opinion rather than a measurement. Country and ASN are considerably more dependable than city.
  • Reassembly is bounded, and says when it gave up. Each direction gets one window (8 KB by default) from a capped pool, so at most --max-reasm flows are reassembled at once — roughly 50 MB at the defaults. A flow that cannot get a window loses extraction, not capture. A hole that later data outruns is abandoned and the stream resynchronizes, with every record from that point marked "inexact":true. The window bounds how much is buffered at once, not how much can pass through it: a segment or an object larger than the window is delivered in pieces, so carved files are not limited by it. What the window does limit is a single message — a header block bigger than it cannot be parsed, and the flow is dropped rather than half-read.
  • TLS 1.3 hides the certificate. From 1.3 onwards everything after the ServerHello is encrypted, so sni is all there is, and the record says "cert":"encrypted" rather than leaving its absence ambiguous. Encrypted Client Hello removes the SNI too, and nothing here can recover it.
  • Credentials are not recorded. FTP passwords and SMTP AUTH exchanges cross the wire in the clear and this tool can see them. It records that an authentication happened and what it was for, never the secret — a stored capture should not become a credential dump. There is no flag to change this; the raw pcap is still the raw pcap, and that is a deliberate difference.
  • Application inference is inference. The name tables are a heuristic and the DNS map is a correlation, not a proof: a shared CDN address answers for many names, and the most recent one wins. That is why every verdict carries its evidence and why appid_known exists. Treat name evidence as strong, dns as good, as-org and port as hints — and the raw pcap as the record.
  • The status page is unauthenticated. --web serves capture metadata — hostnames, addresses, URIs — to anything that can reach the port. That is why a bare port number binds loopback and why binding anywhere else prints a warning. Put it behind an ssh tunnel rather than in front of a network.
  • Carving is bounded and honest about it. A .partial suffix and an "incomplete" or "truncated" flag mean the object on disk is a fragment: the stream ended early, or a cap was reached. Only objects a protocol framed are carved, so their boundaries are known rather than guessed — there is no magic-byte scavenging of unstructured streams.

Layout

src/
  main.c                 args, orchestration, signals, capture loop
  capture/  backend.h    the swap seam (run/stop/get_stats/close)
            af_packet.c  TPACKET_V3 + PACKET_FANOUT, one ring per worker
  decode/   decode.[ch]  bounds-checked L2-L4 dissection
  flow/     flow.h       flow key/record definitions
            flowtable.c  hash table, eviction, direction accounting
  dpi/      dpi.[ch]     engine seam + selection
            dpi_builtin.c  payload matchers + port table
  reasm/    tcp_reasm.[ch] per-direction TCP stream reassembly
  extract/  extract.[ch] engine, event model, parser dispatch
            http.c tls.c dns.c ftp.c smtp.c
            carve.[ch]   recovered objects on disk, safely named
  db/       mmdb.[ch]    MaxMind DB reader (mmap, no dependencies)
  enrich/   enrich.[ch]  OUI / services / geo / ASN lookups
  output/   pcap_writer  rotating classic-pcap writer
            jsonl_writer rotating JSONL flow records
            live         per-packet terminal line
  appid/    appid.[ch]   application inference: names, DNS map, evidence
            apptable.c   the name/ASN/User-Agent/port tables — data, edit freely
  web/      webui.[ch]   optional status server (--web): one thread, two routes
            page.html    the status page — edit this one
            page.c       generated from page.html by scripts/embed-page.sh
  util/     stats        per-worker counters, summed for the summary
            mempool      fixed-size object pools
            hash         FNV-1a + avalanche for flow keys
            sha256       content hashes for carved objects
tests/      selftest.c   offline verification + fuzz pass
scripts/    tune.sh  update-db.sh  livetest.sh  embed-page.sh

Status

Everything described above is built, tested and in use. There is no half-finished stage: capture, flow tracking, classification, enrichment, reassembly, extraction, carving, application inference, multi-worker scale-out and the status page are all complete, and the offline suite covers every one of them.

Two things were considered and deliberately not built:

  • An AF_XDP capture backend. The capture_backend seam exists and AF_XDP would drop into it, but AF_PACKET with a tuned NIC has not been the bottleneck at the rates this runs at. Adding a second backend means a second thing to keep correct for a gain nothing has yet asked for. The seam is the hedge; taking it is a later decision, made with a profile in hand.
  • A full-screen TUI. The summarized live view is --web instead: a status page keeps the terminal free for the per-packet and per-flow output that is already there, and it works over ssh without a terminal that agrees about escape sequences.

The parts most likely to need attention are not code. The application tables in src/appid/apptable.c are a heuristic about a moving target and will age; so will the reference databases behind --update-db. Both are data, and both are meant to be edited or refreshed without touching anything else.

About

A deep packet inspection tool for linux

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages