dacracot Posted March 29, 2020 Report Share Posted March 29, 2020 I'm curious about the specification of the server that you run. I run Java application servers at work (Tomcat, Glassfish, and Solr) on RedHat Enterprise Linux and have considerable experience troubleshooting performance issues. This may be apples vs oranges, but maybe worth a shot. Maybe you could share the p.nerd.nu server config with me so I could see first if there is anything glaringly obvious. Second, if the server isn't too beefy, I might try to replicate it in a VM and see what some of my monitoring tools can shed light on what causes the lag. Yours, dacracot 1 Quote Link to comment Share on other sites More sharing options...
totemo Posted March 29, 2020 Report Share Posted March 29, 2020 https://wiki.nerd.nu/wiki/Server 6 physical cores, 12 virtual. All the map and config are on the SSD. 16GB/64GB allocated to PvE. JVM flags: java.cli-extra=-Dlog4j.configurationFile=/home/minecraft/shared/log4j2.xml -XX:+UnlockExperimentalVMOptions -Xlog:gc:gc.log -XX:+UseG1GC -XX:G1NewSizePercent=20 -XX:G1ReservePercent=20 -XX:MaxGCPauseMillis=50 -XX:G1HeapRegionSize=32M Garbage collection pauses about a tick every 20-30 seconds: [716.024s][info][gc] GC(55) Pause Young (Normal) (G1 Evacuation Pause) 10809M->1170M(16384M) 31.630ms [746.407s][info][gc] GC(56) Pause Young (Normal) (G1 Evacuation Pause) 10802M->1188M(16384M) 29.309ms [772.753s][info][gc] GC(57) Pause Young (Normal) (GCLocker Initiated GC) 10820M->1201M(16384M) 29.421ms [798.676s][info][gc] GC(58) Pause Young (Normal) (G1 Evacuation Pause) 10833M->1218M(16384M) 32.409ms But that's with 24 players. It tightens a bit under heavier load but still on the order of tens of seconds between pauses. It's running on Java 11 atm. I encountered stability issues on Java 13, with Shenandoah; 12 was fine but is no longer getting updates. Spigot timings have not been helpful in pinpointing the issue. My current theory is that chunks are getting hindered on the way out of the box. They're clearly loaded, since physics does work in invisible chunks, e.g. swimming. The chunks are just not reaching the client. Mob counts are sufficiently low. Timings are good enough. So I'm looking at extricating plugins that utilise ProtocolLib (LibsDisguises) so I can test without that plugin-based network code. It takes time to make those code changes, however. The other candidate I want to look at is taking the BungeeCord proxy out of the loop. That's where our ban management happens, so again there is work in extricating it. I would not be at all surprised if it turns out to be an algorithmic complexity faux pas in some bit of network-queue processing code. Since it only really kicks in above a certain load level, and then really bites. Like a polynomial algorithm or exhausting some time allocation per queueing theory. Open to suggestions on freely available tools I could use to track it down. But bear in mind: it has to be debugged on a box with similar load and hardware specs. And fiddling with JVM flags and even the Java version itself has made fuck all difference over the last two versions of Minecraft. We're running at decent enough TPS up to about 50 players. Also, the general consensus on https://reddit.com/r/admincraft is that Minecraft performance sucks. And finally bear in bind that the time I spend humouring armchair experts (not to denigrate your particular skills) takes away from the time and energy I have available to (start) meet my current deadlines. I might also add that MobLimiter had a severe impact on TPS and has been reconfigured not to load extra chunks (synchronously on the main thread!). That's another avenue of investigation. 2 Quote Link to comment Share on other sites More sharing options...
totemo Posted April 2, 2020 Report Share Posted April 2, 2020 FWIW: Dedotating moar wam didn't help. Switching (back) to Aikar's JVM flags didn't help. What has worked absolutely stunningly well is waiting for the PaperSpigot guys to add more chunk loading optimisations. The server's running a lot better today and I attribute that to PaperSpigot build 155. The previous build was 146. They optimised a few things in between. https://papermc.io/ci/job/Paper-1.15/changes Build 156 has even more chunk-related optimisations, so we'll try upgrading again. 2 Quote Link to comment Share on other sites More sharing options...
zedadex Posted August 13, 2020 Report Share Posted August 13, 2020 On 4/2/2020 at 1:40 AM, totemo said: FWIW: Dedotating moar wam didn't help press moar buttons?! D: Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.