We're amazed every day by the beautiful structures you've built since our first Alpha, the building system is one of the most highly engaged in Pax Dei so far. As such, we thought you might be interested in knowing a bit more about the tech behind the system - which is probably one of the most complex systems in the history of video games.
Currently, the hottest home valley zones have around 400,000 building pieces. With a total of 24 home valleys per world, there are many millions of pieces in each world. The typical build rate is 10,000 new pieces per week in a single home valley.
Our replication system
In a networked multiplayer game, the replication system is used to ensure that all the players have a consistent and up-to-date game state. In other words, It receives the input information of a player and distributes it to other players.
The building data size for a single home valley is currently around 30MB. That doesn’t sound like a lot, right? But let’s say we have 100 players - we are now talking about 3GB of data that needs to be managed by the server, the client (the player’s machine), and our replication system.
This amount is so massive that it needs very special handling in all stages of the game’s pipeline. If we use Unreal Engine replication for gameplay, we can’t use it for the buildings. The backbone of our building replication is the Repli backend server - aptly named ‘Repli’ - which sends build data from the world database to the clients and the servers.
When logging in, both the client and the server connect to Repli, which first sends the current full state of the zone. As we said, initial data is currently around 30MB in the largest zones. After the initial state replication, the client and the server listen to new build events from Repli. We use gRPC streaming as a communication channel to build events from Repli. We won’t go into detail about gRPC here but, simply put, it ensures all parties involved, the server, Repli, and the clients, communicate correctly and understand each other.
Adding a building piece: under the hood
Here is simply what happens after a player swings their construction hammer:
[olist]
[*] The client sends a build request to the Unreal server;
[*] The server does some permission checks to verify if the build request is allowed;
[*] If the server allows it, it then sends the build request to the backend;
[*] The backend does its own checks to allow or refuse the build request;
[*] Once allowed, the building piece is added to the main world database;
[*] The build event is sent from the database to our custom Repli backend server;
[*] Repli will send the build event to both the server and the client;
[*] The server and all the clients in the zone - here, the Home Valley - spawn the building piece;
[*] The server does the integrity check that the structure still holds;
[*] The client calculates integrity as well to display it in the UI.
[/olist]
Making the building visible to all
Massive data replication is one of many challenges. After the client has all 400,000 pieces of the home valley, the 30MB package we discussed earlier, we need to make them visible. Your computer must draw them, and here comes the second big challenge.
It would be impossible to use a system that draws 400,000 pieces one after the other. We need something tailored to suit our needs. The two key elements we use are instanced static meshes (ISM) and area managers.
ISM are used to draw several, even thousands, of pieces in one draw call, instead of drawing them one by one.
Area managers, the parallelepipeds boxes in the video below, control all props and building pieces in their areas. They handle the creation and destruction of building pieces and props, integrity calculations, and other things, including the ISM.
The area managers look at all the standard building pieces in its area - not the crafters nor the containers - and sorts them spatially. Then, they insert them in custom instanced static meshes. And now the magic can happen.
That being said, we are not done yet. In addition to buildings, we have containers, crafters, and decoration pieces that need to be drawn, and most of them have their own gameplay functionality. All these props are standard Unreal actors, but contrary to the buildings you can see from pretty far away, they spawn only around the character inside the streaming distance.
In the video below, you can see how the area managers are working and how the props become visible or not. Each box you can see is an area manager. As explained above, each area manager controls its own area for ISM and integrity, but they also control the visibility of the prop actors - crafters, containers, and other decoration pieces. Depending on the position of a character and the streaming distance of its client, the manager will activate or deactivate the visibility of all its prop actors.
In the video, red shows the area managers that deactivate the props’ visibility and cyan the active ones. Purple shows the ones pending ongoing activation, and magenta the ones pending ongoing deactivation.
[previewyoutube=ZQG9MZJavwo;full][/previewyoutube]
There is another type of prop that needs special treatment: lighting sources. When your character is close to them, the lights are running through the Unreal Lumen lighting - the dynamic global illumination system from Unreal Engine 5 - to improve immersion. However, the lights are switched to lightweight light cards at a certain distance to improve performance. Some valleys have more than 20,000 lights that you can observe simultaneously!
Of course, we also have some constraints on the server side. One single Unreal Engine server can handle only a maximum of 150-200 players, and if more players want to be in the same village, then we need to create some server instances. That’s another area where having some custom systems helps a lot. Our replication system allows the buildings to be visible across all the server instances and in real time! If one player builds in one server instance, then all the players in the other server instances see the building changing at the same time.
A few last words
Spawning all this building stuff around the player without visible lag is complicated and is done over several seconds, even minutes, sliced into many frames. Also, as you might have guessed from our step-by-step process above, the integrity calculation is a very complex process. Our servers also stream world content, so integrity calculation needs careful setup so that all supporting landscapes are there before they even start. Creating only physics states for pieces needs to be done without delay, and after that, querying overlaps from physics and actual integrity calculations need to be done iteratively.
Now you know why when logging in, you often can see the landscape first, then the buildings, and finally the props.
Players have been building much faster than we initially anticipated. The whole system has seen many optimization passes so far, and we still have many tricks up our sleeves. The current system should work fine until around 500 000 pieces. After that, we will start seeing new issues that must be solved. Good news, though! We already have plans to scale the system to 1 000 000 pieces per zone. But, that is a story for another time.
In the meantime, you can keep building and follow these tips from players who have mastered our systems and environment to create gorgeous structures!
Pax vobiscum,
The Mainframe Team
