For a while now, my team and I at Qpoint.io have been grappling with the challenge of understanding what's actually happening inside the encrypted traffic leaving our production systems. Modern apps rely heavily on third-party APIs (think payment processors, data providers, etc.), but once TLS kicks in, figuring out exactly what data is being sent, identifying PII exposure, or debugging integration issues becomes incredibly difficult without resorting to complex and often brittle solutions.
Traditional approaches like forward proxies require terminating TLS (MITM), managing certificates, and often introduce performance bottlenecks or single points of failure. Network firewalls usually operate at L3/L4 and lack payload visibility. We felt there had to be a better way.
That's why we built qtap. It's a lightweight agent that uses eBPF to tap into network traffic at the kernel level. The key idea is to hook into common TLS libraries (like OpenSSL) before encryption and after decryption. This gives us deep visibility into the actual request/response payloads of HTTPS/TLS traffic without needing to terminate the connection or manage certs. Because it leverages eBPF, the performance impact is minimal compared to traditional methods.
With qtap, we can now see exactly which external services our apps are talking to, inspect the payloads for debugging or security auditing (e.g., spotting accidental PII leaks), monitor API performance/errors for third-party dependencies, and get a much clearer picture of our egress traffic patterns.
We've found this approach really powerful for improving reliability and security posture. We've packaged qtap as a Linux Binary, Docker container, and Helm chart for deployment.
This is still evolving, but we're excited about the potential of using eBPF for this kind of deep, yet non-intrusive, visibility.
We'd love to get the HN community's feedback:
Do you face similar challenges monitoring encrypted egress traffic?
What are your thoughts on using eBPF for this compared to other methods?
Any suggestions or potential use cases we haven't considered?
"This is not a vulnerability: eBPF currently requires root access to do this. Also, eBPF makes this easy but does not make it possible, as debuggers, interposers/shims, and other tools can also attach to pre-encryption points, and therefore banning eBPF (as some people want to do after seeing projects like this) would not actually improve security, but it would instead _reduce_ security as it would prevent eBPF-based security solutions from being used."
On an unrelated note, your work has inspired most of my career in Solaris/Illumos/Linux systems and honestly this project likely wouldn't have happened if it wasn't for all of your books/blogs/projects to help me along the way. Thank you!
We do this by scanning every version of Go that is released to find offsets in the standard library that won't change. Then when we detect a new Go process, we use an ELF scanner to find some function offsets and hook into those with uprobes. Using both of these, we have all the information we need to see Go pre-encryption content as well as attribute it to connections and processes.
ISTR, at some point in the far past, using LD_PRELOAD with my own shims to capture TLS traffic before encryption/after decryption. I might have it lying around somewhere here.
Are these offsets consistent across compilation targets, and they vary only by version of the Go binary? Or do you need to do this scan for every architecture?
The long answer is that the offsets are the byte alignment offsets for the go structs containing the pointers to the file descriptor and buffers. Fortunately we only have to calculate these for each version where the TLS structs within go actually change, so not even for every version. For instance, if a field is added, removed, or changes type then the location in memory where those pointers will be found changes. We can then calculate the actual offset at runtime where we know which architecture (amd64, arm64, etc) with a simple calculation. Within the eBPF probe, when the function is called, it uses pointer arithmetic to extract the location of the file descriptor and buffer directly.
> The key idea is to hook into common TLS libraries (like OpenSSL) before encryption and after decryption
This is the eBPF side: https://github.com/qpoint-io/qtap/blob/main/bpf/tap/openssl....
The Go side which indicates what we are scanning for is here: https://github.com/qpoint-io/qtap/blob/main/pkg/ebpf/tls/ope...
For more docs on the topic: - https://docs.ebpf.io/ is a must read - https://eunomia.dev/en/tutorials/30-sslsniff/ has a tutorial on cracking OpenSSL open and getting the content as well. The tutorials they have are fantastic in general
> Beyla supports a wide range of programming languages (Go, Java, .NET, NodeJS, Python, Ruby, Rust, etc.)
Although "gRPC and HTTP2 are not supported at the moment"
Our aim is to make Qtap extensible and via a plugin system. We have http1/2 streaming capabilities and a plugin engine to run these in what we call a stack. Our goal is to add more protocols, like gRPC in the near future.
We have a few example plugins that do things like report request/response's and push access information to standard out in a console or structure log format. Our Pro version has a few more plugins like the ability to report errors (eg. an AI agent is getting HTTP 429 errors). These can be pushed to a service or log aggregator.
To summarize, we do a lot of the same things that ecapture does. We'd like to be less of a tool and more of a "always running" that ops, opsec, and devs use to answer tough questions. We look forward to open sourcing more of plugins as they mature!
I work a bunch with vpn-like networking on Android phones and it would be cool to have a bit of info on how I might get something like working on phones. I guess its probably not your typical usecase.
Currently since the project is a VPN client, I already intercept all of the packets, I have a pcap writer and can write to files or a tcp sockets and connect wireshark to it - but it needs a bunch of complication to setup the keys so that I can see through encryption, so anything that would make that process easier would be great.
I've seen that type of behavior for apps that inject ads and add affiliate marketing links
The wireshark stuff is only for when I'm debugging
There's an alternative implementation where SSLKEYLOGFILE is more "dynamic" and permits being toggled on an off during runtime, but that doesn't currently exist.
We've also added caching so frequently used bins don't require multiple scans. Shared libraries as well. This has proven effective with optimized binaries, especially bins that are optimized, start, make a super quick network call, then exit, which was the bane of our existence for a little while.
Anyone have any experience with it?
Modifying response bodies in the forward proxy is less than ideal. The proxy must wait for the full response body to be received before making modifications.
Can eBPF be any better in this regard.
> Security enforcement: Allowing or denying traffic based on precise conditions
Very cool. What are your supported log sinks?
As far as log sinks, we have stdout right now. We have been working on Fluentbit and will eventually add a bunch more. If you have a request, drop them here!
We also have a services concept which support an "event store" and "object store", where the object store handles artifacts that may contain sensitive data and don't need to be indexed for search/aggregation (this is an S3 compliant store). The event store handles all of the events from connection audit logs (these cover the ip protocol level) to individual http request/response pairs. The event store is a custom API we use and need to write some proper documentation for, stay tuned!
Edit: Another user posted a link to https://github.com/gojue/ecapture which looks like it supports android and has some overlapping functionality.
How can we remove the impact that proxies have on connections, AND see the content without having to manage a custom certificate authority, AND not have to instrument all of our code.
Perhaps with `direction: all`
https://github.com/qpoint-io/qtap/issues/29#issuecomment-286...
That said, the eBPF verifier has robust security guarantees and runs on every load. So arbitrary mem access for example isn't possible. Qtap runs exclusively on your nodes, so you control what it captures and where that data goes. Our paid offering provides more functionality with a Control Plane solutions that provides dashboards, alerting, and live config updates. However, all sensitive information, like captured http bodies, are uploaded to a S3 compliant bucket that you control. This could be S3, Minio, or anything else that supports the S3 API. We never see this information.
It's intentionally designed for deployment within your infra and abides by the security policies you set within your org.
Is uploading clear text encrypted in flight data to another system even a good idea in most cases? In some cases that won't even be allowed because you'd end up storing regulated information in a way that auditors won't approve of (e.g. sometimes there is a requirement for field level encryption when data is in storage/at rest)
Qtap can be locked down with local firewalls or perimeter firewalls like other applications running within a network. The TLS inspection can also be disabled with a `--tls-probes=none` flag on startup.
Even without inspection enabled, Qtap provides rich context when it comes to connections to processes. For example, source/destination information, bandwidth usage, SNI information, container meta, even Kubernetes pod and namespace meta. All of this can paint a thorough picture of what's happening with zero instrumentation.
When it comes down to it, some orgs may not be able to use the TLS inspection or require specific methods of persisting data. If we can't support this today, our goal is to address these as they come up and hopefully help devs and ops folks working in these constrained environments get what they need while maintaining compliance.
Also worth noting this is very similar to the code path that got Crowdstrike in trouble when they crashed every device on the internet because of a bug in the parser of their rules engine.
1. uprobes can be expensive and add latency (they force a context switch and copy data), especially when the hooked functions are called a lot
2. EBPF is not widely available outside of Linux, requires elevated privileges (compared to a MITM proxy that requires no privileges and works with every OS)
3. Doesn't work with JVM, Rust, any runtime that doesn't use the hooked functions
To address your points:
1. In our testing, uprobes add a statistically insignificant amount of latency and in comparison to a MITM proxies it's nearly identical to native.
2. True, we're focused on Linux right now. I'm looking forward to exploring Microsoft's eBPF implementation and exploring how we can support the Windows ecosystem.
3. You're right that the technique we are using for OpenSSL will not work for other runtimes. That said, there are other techniques that we've implemented in our Pro offering for the JVM, Go, and NodeJS. Rust is in the works!
I'm always looking for a way to make sniffing traffic from inside a container easier, and if I could attach a debug sidecar with something like an eBPF based SSL pre-master key extractor (both on incoming and outgoing requests) it starts to feel a lot like having network JTAG.
There are some important flags when spinning it up in docker: `--privileged`, `--cap-add CAP_BPF`, `--cap-add CAP_SYS_ADMIN`, and `--pid=host`. These provide access to load eBPF programs, and monitor traffic.
Many deployments use Kubernetes daemonsets where Qtap runs in a container, but monitors all of the traffic on the node. The Qpoint paid offering comes with a Control Plane that produces context specific dashboards so seeing what's happening from a specific container, or pod namespace can provide a lot of insights into your deployments.
Supposedly `kubectl debug` does allow you to set a `sysadmin` profile and grant the debug sidecar `privileged` access. I think that would be a neat low cost way to get value out of your product quickly, right when I have an issue, which would maybe help build some organizational trust and goodwill to make the rest of the stack easier to buy.
It would also solve an issue for me that I would really like solving :P
You can customize config and/or integrate with existing observability pipelines, but initially you just need to turn it on for it to work. No app instrumentation required.