OpenSSL HollowByte Memory Leak Vulnerability
Eleven bytes are enough to force a weak OpenSSL server to use up to 131 KB of RAM. After that, it waits for a message that never comes. Following a sudden drop, the code frees the buffer. Yet, Linux setups using glibc may keep this memory inside the active task. Repeating this quick request slowly drives up resource use. As a result, this memory leak can cause a full server crash.
Discovering the HollowByte Flaw
Red Team experts at Okta first found this flaw and called it HollowByte. The flaw hides deep inside the start of the TLS connection handler. Oddly, the fix entered the June builds of OpenSSL without any formal notice. It lacks a CVE tag or any mention in the main bug fix list.
This bad issue starts well before login steps, key swaps, or secure setup tasks. Thus, a hacker needs no passwords or valid keys to launch a strike. They just open a basic link and send the start of a rigged TLS message.
How the TLS Handshake Fails
In general, TLS keeps web links and network tasks safe. Before sending real data, the client and server do a quick handshake to agree on rules and keys. Next, the client often sends a ClientHello message first with its main settings.
Every handshake message starts with a simple four-byte header. The first byte shows the message type. Then, the next three bytes contain the stated length of the main body. In weak OpenSSL builds, the code blindly trusted this size value. So, it swiftly grew the receive buffer before the data even arrived.
For a new ClientHello, the buffer size could easily hit 131 KB. After reading the short header, the active task kept waiting for the rest of the data. Meanwhile, the hacker sent nothing else at all. They either kept the link open or closed it right after forcing the memory grab.
Memory Fragmentation and Glbic Issues
This sneaky method closely mimics the classic Slowloris attack style. In those cases, a server holds links open while waiting for slow data. Yet, HollowByte hurts much more than just the total number of open links. The main damage grows from how the system memory tool handles the freed buffers.
On many Linux setups, the standard glibc tool handles system memory. After a free command, it does not always return small or medium chunks to the OS kernel right away. Instead, the tool keeps them inside the process to speed up future memory tasks.
Why Standard Defenses Fail
In fact, this setup works as a normal and smart fix. A real problem only shows up when requests force OpenSSL to build buffers of changing sizes. Soon, the freed memory blocks stop fitting new tasks. Thus, empty gaps appear between used zones, and the memory slowly breaks apart.
Formally, OpenSSL already freed the unused buffers. Yet, the server cannot use the left over pieces well. Also, it fails to return them to the main system. As a result, the RSS metric keeps growing. It never drops even after the attack stops. Users can only reclaim memory fully by restarting the process.
During an active NGINX test on a server with 1 GB of RAM, the system killed the process due to a lack of free space. At that point, about 547 MB stayed locked inside the broken heap. On a larger 16 GB server, the attack quickly grabbed about a quarter of all memory. It did this without ever breaking the active link limits.
Therefore, standard defense tools might fail to stop HollowByte. For instance, limiting links per IP or total client counts will not block the threat. A hacker does not need to hold many open links at once. Each done request can leave behind fresh, awkward memory pieces.
Fixing the Vulnerability
Right now, these published facts come solely from Okta’s private tests. They remain unverified by outside checks. Also, the team did not release real attack code. It also remains unclear how strongly this bug hurts other memory tools besides glibc.
OpenSSL quickly fixed this TLS message flaw in versions 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21. These secure builds launched on June 9. Conversely, older releases in those exact branches still hold the weak code.
After the new patch, the tool no longer grows the buffer to the client’s claimed size right away. Instead, memory grows slowly as real data arrives. Thus, a client can no longer force a server to grab a huge block with a mere promise of a long message.
The Danger of Silent Updates
Surprisingly, OpenSSL builders treated this change as a routine bug fix. It never received a CVE tag or an official warning notice. As a result, this choice makes updating firm systems much harder. Security scanners usually track installed apps using CVE numbers. Meanwhile, admins rely heavily on vendor bulletins. Without a clear ID, standard tools might fail to show that a server needs a patch.
Also, checking safety based solely on the OpenSSL version number is tricky. Some Linux builds do not replace the tool fully. Instead, they backport fixes into older branches. Thus, a package version might look weak even if the vital patch is already active.
Therefore, admins must check the package changelog or ask their vendor. Builders added the fix in merge requests 30792 for the main branch, 30793 for newer branches, and 30794 for version 3.0. For manual builds, users should upgrade to the latest secure OpenSSL releases right away. After setup, you must restart all services using the old tool to avoid running unsafe code.
Unfixed DTLS Risks
Crucially, this fix only covers standard TLS. The DTLS protocol variant still uses the claimed client length during buffer tasks. The patch maker stated that fixing DTLS required much deeper code changes. Therefore, that section remains unfixed for now. The project has not stated if they plan to change it later.
This case stands out among other OpenSSL memory flaws. In January, a similar TLS 1.3 certificate issue received a low-risk CVE tag despite needing complex rules. In June, a fast memory growth issue in the QUIC handler also gained a unique ID.
Conversely, HollowByte requires no extra features or complex setups. As detailed in the OpenSSL HollowByte DoS report, the request hits the weak zone during a normal TLS handshake start. Despite this easy, direct attack path, OpenSSL did not rate the bug even as low risk.
Luckily, the June 9 updates closed 18 known flaws at the same time. Therefore, users running the latest builds stay fully safe from HollowByte. Yet, the release notes fully skip this detail. The main risk remains for servers where teams only install updates after a formal CVE appears.
Support Our Threat Intelligence
If you find our technology report and cybersecurity news helpful, consider supporting our work.
Buy Me a Coffee
PayPal
