Server Performance
The last two weeks I tried nailing down our performance problems on the server side.
I started by writing a custom tool which can start multiple clients on one pc in order to simulate a specific number of players. The tool proved quite useful in order to profile our performance and showed that with only 40+ players we already dropped from 5000 FPS (without players) to 40 FPS which is inacceptable when we want to have 100 players and hundereds of NPCs running simultaneously on one server.
I identified one problem with the way we handled network messages between the server and our backendclient. The message assembly of our animation states already resulted in a lot of garbage due to the constant allocation of new strings. Approximately 10-20 MB garbage piled up over multiple frames and this is only one type of packet which created garbage. I realized we had to finally rewrite the network stack to use binary data. After some planing I decided to scrap our custom solution in order to use the same api our clients use. The api is pretty fast and lightweight. Now the backend-client connects just as a regular client with special abilities. The downside of this ist hat the limit of players is now 99 instead of 100, because the backendclient takes one slot. The reason why we didn’t consider it earlier was that we planned to spawn instances for dungeons and other areas of the game. For each instance we would have needed a custom instance of a backend-client that is connected to that instance. So we would have ended up using an user slot for each backend-client which wouldn’t have been maintainable, but since we came up with a different solution to instances we can now use this method.
First results show that we nearly have no memory allocations that result in huge spikes. Next week I have to further optimize and rewrite a lot of the packet handling to use the new system and test performance once again.