Jump to content
Sign in to follow this  
dacracot

Server Configuration & Tuning

Recommended Posts

I'm curious about the specification of the server that you run.  I run Java application servers at work (Tomcat, Glassfish, and Solr) on RedHat Enterprise Linux and have considerable experience troubleshooting performance issues.  This may be apples vs oranges, but maybe worth a shot.  Maybe you could share the p.nerd.nu server config with me so I could see first if there is anything glaringly obvious.  Second, if the server isn't too beefy, I might try to replicate it in a VM and see what some of my monitoring tools can shed light on what causes the lag.

Yours, dacracot 

Share this post


Link to post
Share on other sites

https://wiki.nerd.nu/wiki/Server

6 physical cores, 12 virtual. All the map and config are on the SSD. 16GB/64GB allocated to PvE.

JVM flags: java.cli-extra=-Dlog4j.configurationFile=/home/minecraft/shared/log4j2.xml -XX:+UnlockExperimentalVMOptions -Xlog:gc:gc.log -XX:+UseG1GC -XX:G1NewSizePercent=20 -XX:G1ReservePercent=20 -XX:MaxGCPauseMillis=50 -XX:G1HeapRegionSize=32M

Garbage collection pauses about a tick every 20-30 seconds:

[716.024s][info][gc] GC(55) Pause Young (Normal) (G1 Evacuation Pause) 10809M->1170M(16384M) 31.630ms
[746.407s][info][gc] GC(56) Pause Young (Normal) (G1 Evacuation Pause) 10802M->1188M(16384M) 29.309ms
[772.753s][info][gc] GC(57) Pause Young (Normal) (GCLocker Initiated GC) 10820M->1201M(16384M) 29.421ms
[798.676s][info][gc] GC(58) Pause Young (Normal) (G1 Evacuation Pause) 10833M->1218M(16384M) 32.409ms


But that's with 24 players. It tightens a bit under heavier load but still on the order of tens of seconds between pauses.

It's running on Java 11 atm. I encountered stability issues on Java 13, with Shenandoah; 12 was fine but is no longer getting updates.

Spigot timings have not been helpful in pinpointing the issue.

My current theory is that chunks are getting hindered on the way out of the box. They're clearly loaded, since physics does work in invisible chunks, e.g. swimming. The chunks are just not reaching the client. Mob counts are sufficiently low. Timings are good enough. So I'm looking at extricating plugins that utilise ProtocolLib (LibsDisguises) so I can test without that plugin-based network code. It takes time to make those code changes, however. The other candidate I want to look at is taking the BungeeCord proxy out of the loop. That's where our ban management happens, so again there is work in extricating it.

I would not be at all surprised if it turns out to be an algorithmic complexity faux pas in some bit of network-queue processing code. Since it only really kicks in above a certain load level, and then really bites. Like a polynomial algorithm or exhausting some time allocation per queueing theory.

Open to suggestions on freely available tools I could use to track it down. But bear in mind: it has to be debugged on a box with similar load and hardware specs. And fiddling with JVM flags and even the Java version itself has made fuck all difference over the last two versions of Minecraft. We're running at decent enough TPS up to about 50 players. Also, the general consensus on https://reddit.com/r/admincraft is that Minecraft performance sucks. And finally bear in bind that the time I spend humouring armchair experts (not to denigrate your particular skills) takes away from the time and energy I have available to (start) meet my current deadlines.

I might also add that MobLimiter had a severe impact on TPS and has been reconfigured not to load extra chunks (synchronously on the main thread!). That's another avenue of investigation.

  • Upvote 1

Share this post


Link to post
Share on other sites

FWIW: Dedotating moar wam didn't help. Switching (back) to Aikar's JVM flags didn't help.

What has worked absolutely stunningly well is waiting for the PaperSpigot guys to add more chunk loading optimisations. The server's running a lot better today and I attribute that to PaperSpigot build 155. The previous build was 146. They optimised a few things in between. https://papermc.io/ci/job/Paper-1.15/changes

Build 156 has even more chunk-related optimisations, so we'll try upgrading again.

  • Upvote 1

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...