Hotfix 2 Status - Server and Stability Update - Fri 24th October
[p][/p][p]Today marks a major milestone in our ongoing efforts to resolve the stability and crashing issues that have affected Hell Let Loose since Update 18.[/p][p]After many days of investigation, testing, and community support, we’re pleased to share that the team has successfully identified the root cause of the problem and implemented a fix. This new build has been tested and is almost ready for an rollout as Hotfix 2.[/p][p]We know how disruptive the last week has been for many of you, and we want to thank everyone who has continued to provide reports, feedback, and patience as we worked to get the game back to a stable state. Your input has been instrumental in helping the team reach this point.[/p][p][/p]TLDR Summary
- [p]The team has identified the root cause of the crashes affecting Hell Let Loose since Update 18.[/p][/*]
- [p]The issue was linked to a physics material reference error caused by certain assets, which could result in client and server crashes.[/p][/*]
- [p]A new build containing a fix for both client and server crashes has been created, tested, and certified.[/p][/*]
- [p]This fix is now available for Steam players and server owners through a new ‘Experimental Branch’.[/p][/*]
- [p]The full Hotfix 2 release will roll out on Steam and other platforms early next week following platform approvals.[/p][/*]
- [p]Players and server hosts on the main branch or console platforms are advised to remove the following maps from rotation over the weekend: Carentan, Foy, Omaha, Utah, Remagen, Sainte-Mère-Église (SME), Stalingrad[/p][/*]
- [p]The team will continue to monitor for any remaining issues, while also preparing additional asset corrections for Update 19.[/p][/*]
- [p]Thank you to everyone who provided reports, feedback, and patience. Your support directly contributed to identifying and fixing the issue.[/p][/*]
Status of the Investigation
[p][/p]Latest Update
[p]After yesterday’s breakthrough in reproducing the crash, Kieran and his team worked late into the night to pinpoint the exact cause. It wasn’t an easy process, but by the end of the evening they had successfully identified the specific issue, allowing work on the fix to begin immediately.[/p][p]This morning, the first new build containing the fix was completed. It also includes an additional correction for another potential fault in the same chain of logic.[/p][p]Since then, our QA team has been rigorously testing the new build, which was successfully certified this afternoon.[/p][p]With this update, we have addressed both client crashes (often seen as mass disconnects) and server crashes (which previously caused players to lose connection).[/p][p]We would like to note that, due to the nature of the issue, there may have been other undefined behaviours linked to it that did not always result in a crash. These should now also be resolved. However, following this hotfix, we will continue monitoring to ensure there are no remaining causes of client disconnections.[/p][p][/p]Details on the Hotfix 2 for Update 18
[p]Thank you all for your patience. We can now confirm that a new build of Hell Let Loose has been created and includes a fix for the crashing issue. Below are the key details to be aware of:[/p][p][/p]1. New Steam Branch:
[p]A new Experimental Branch is now available on Steam. You can access it by navigating to:Steam → Hell Let Loose → Properties → Betas → Experimental Branch[/p][p][/p][p][img src="https://26630753.fs1.hubspotusercontent-eu1.net/hubfs/26630753/Hell%20Let%20Loose/Crash/image.jpg"][/img][/p][p]Head to the Properties section of the Hell Let Loose app on Steam. Do this by right clicking on Hell Let Loose, or hitting the settings cog within the Steam page.[/p][p] [/p][p][img src="https://26630753.fs1.hubspotusercontent-eu1.net/hubfs/26630753/Hell%20Let%20Loose/Crash/image%20(2).png"][/img][/p][p]Go to Betas, and then select ‘experimental_branch’[/p][p] [/p]
- [p]The “None” option is the default/main branch that has the current Live 18.0 Hotfix 1 build[/p][/*]
- [p]The “experimental_branch” right now has 18.0 Hotfix 2 build[/p][/*]
2. Other Platforms:
[p]Due to approval requirements, this version will not go live on other platforms today. It is currently scheduled for release early next week.[/p][p]In the meantime, for all other platforms (and for those on Steam who have already begun seeding on the main branch) we have shared a list of maps that have been identified as potential triggers for the crash:[/p]- [p]Carentan[/p][/*]
- [p]Foy[/p][/*]
- [p]Omaha[/p][/*]
- [p]Utah[/p][/*]
- [p]Remagen[/p][/*]
- [p]Sainte-Mère-Église (SME)[/p][/*]
- [p]Stalingrad[/p][/*]
From the Dev Team: Understanding the Issue
[p]Yesterday, Kieran, Studio Technical Director at Expression Games, shared a detailed breakdown of the work happening behind the scenes to uncover the root cause of the crashes, and why this issue only appeared in the live environment. Following the breakthrough we had last night, we were able to identify the cause of the crashing issue and deliver a fix.[/p][p]With this in mind, we wanted to continue our commitment to transparency by providing a deeper explanation of what went wrong and how it was resolved.[/p][p][/p]How The Crash Presents Itself
[p]The crash can manifest on any client or the game server. In both cases, one of the following functions would typically appear near the top of the call stack:[/p][p][img src="https://26630753.fs1.hubspotusercontent-eu1.net/hubfs/26630753/Hell%20Let%20Loose/Crash/Screenshot%202025-10-24%20181649.png"][/img][/p][p]The crash appeared to happen randomly and didn’t have a 100% reproduction rate, which made it extremely difficult to track down.[/p][p][/p]How The Core Issue Was Discovered
[p]When the issue first began to appear in community reports post-Update 18, the team reviewed crashes captured through Backtrace in the development environment. We later confirmed that the issue had existed since 28 July 2025, with only four crashes recorded during that time (between 28 July - 16 October): two server-side and two client-side. This immediately showed how hard it was to reproduce under normal development conditions.[/p][p]At first, it was unclear whether the cause was related to server type (GPortal versus community servers), differences in server tools like CRCON, whether it required a full server populations, or specific gameplay situations that you you seen in a public game environment.[/p][p]To narrow things down, we explored all these possibilities in parallel. We collaborated closely with the community, server owners, and War Correspondents to gather reports and eliminate potential causes.[/p][p]Because the issue occurred so rarely during testing, it became clear that the only way to properly diagnose it was to continue investigating while the bug remained live in the game. If we had rolled back Update 18, there was a real chance we might never reproduce it during internal testing, which would have prevented us from finding and fixing the issue.[/p][p]On Wednesday 22 October, the first successful reproduction of the crash occurred on an live official server, ruling out both the server provider and external tools (e.g. CRCON) as contributing factors. During this time, we identified that the Foy map had the highest rate of reproductions, thanks to reports from the server owners and War Correspondents.[/p][p]The next day (Thursday 23 October), QA focused on Foy specifically. At that point, there was a theory that the new artillery strike commander ability might be related. While that turned out to be true, although not in the way initially suspected, it was during these tests that we got lucky.[/p][p]A QA tester experienced a crash after entering a building. One by one, other testers entered the same building and also crashed. That gave us our 100% reproduction case.[/p][p][/p]What The Problem Was
[p]When performing physics tests in Hell Let Loose, the game often requests the physical material of the object being interacted with. This information determines what visual effects, particles, and sounds are used when bullets or projectiles hit that surface.[/p][p]With Update 18, some assets were re-imported to resolve texture issues, but this process unintentionally affected their Level of Detail (LOD) setup. As a result, certain objects were referencing one fewer material than they should have.[/p][p]When the physics system interacted with these assets, it sometimes attempted to read a material from an invalid memory address. Depending on what that memory address pointed to, what happens next is down to chance, with one of two outcomes occuring:[/p]- [p]The game continued running, unknowingly using corrupted data.[/p][/*]
- [p]The game crashed immediately in one of the previously listed functions.[/p][/*]
