Life has slowed down a bit and we were finally able to have our first "official" padmin meeting of Rev 24.
Here are some of the things we talked about:
We want to thank totemo for getting the rev launched pretty much on time. He had a few tricks up his sleeve which made the change over take a bit less time on launch night and I believed it worked well enough to use those tricks again in the future.
We also want to thank the moderators that helped spruce up the ship before our launch into the new rev. The decorations and secrets are amazing.
Finally, thank you for being patient while we continue to deal with issues as fast as we can.
We took some time to go through the current suggestions on PvE and posted the responses on the forum here.
Keep those suggestions coming. We do like to hear from you.
It is still early rev, and the 3 of us have barely had time to build in our towns but we are excited to start hosting some events.
Our first big event will be an Admin Hunt. We haven't decided on a date and time yet but we do ask for your help. We are in need of a good arena. In the past players have liked when builds or towns from previous revs were used. So check out this post if you would like to suggest a build from a previous revision to use for admin murderizing.
But while we work out the details of the first admin hunt, don't forgt to go hunting for those spawn secrets. (No one has found cheezy's yet and he's happy he his it so well)
I did not have a chance to place a spawn secret before the launch, so I will be adding a small scavenger hunt this weekend.
We also had a discussion about PvP events. Many revs ago I remember player run PvP fights being hosted almost every weekend, but recently it seems that players wait and expect the admins to set up and host these events. I, for one, would love to see the return of more player run events. I enjoyed heading to the town early to check the place out before heading to the arena to be killed over and over again. So how do we bring back player run get togethers? Is it a matter of newer players not knowing they can host events? Or do players just not like traveling to other towns and would rather be teleported to an arena not on the map? Cheezy is planning something, but we do want to encourage everyone - If you have an idea for an event (PvP, spleef, thimble, anything) go for it! If you need help getting the word out, we will help.
Deaygo has been working on the new nerd.nu web site, with some help from other techs, and we now have the technology to give you regular blog updates on the state of the servers. Thanks, Deaygo!
If you've been playing regularly in the last couple of months, it will not have escaped your attention that the new server hardware is much snappier than the old box. However, we'll make a note of it here for the sake of posterity. PvE restarts are now so quick that there is only just enough time to get from the lobby spawn to the PvE lobby portal before the server is open again.
The new server specs are documented on the Server page of the nerd.nu wiki. The main highlights are double the amount of RAM, a faster CPU with 2 more physical cores and an SSD that has doubled in size.
Creative Rev 34 and Minecraft 1.13.2
Creative got stuck on Minecraft 1.12.2 for some time while we waited for FastAsyncWorldEdit to update. But by mid June, a development server was running on 1.13.2 with the current Rev 34 worlds and updated plugins, courtesy of totemo with some help from Challenger2. The final step was for the CAdmins to do some testing. By the time they were satisfied with the state of it, at the end of July, Dumbo52 had rejoined the tech team, and his first major act was to copy the dev server configuration over to the main host. He's currently doing minor tweaks on the configuration to fix issues that slipped through the net during the testing phase.
PvE Rev 24 and Minecraft 1.14.4
Chunk I/O and Lag
Minecraft 1.14 was a very buggy update with serious performance problems. We run PaperSpigot, (a.k.a. Paper) which is an optimised server derived from Spigot, which is the continuation of the Bukkit project. The PaperSpigot devs describe 1.14 thus:
MC 1.14.x was a rushed release from Mojang and has significant performance issues in how it handles chunk I/O. It is not recommended for large servers. Extensive backups and patience are recommended. [Source]
The PaperSpigot project has an official statement on 1.14 performance on their web site here.
That's certainly been our experience on PvE so far as we approach the end of the second week of Rev 24. We seem to be able to run at 20 TPS (Ticks Per Second) up to about 50 players, but beyond that we fall off some kind of performance cliff. Chunks taking a long time to load and long (7-8 seconds) saves saves are the most obvious symptoms of our current performance woes. Chunk I/O is also apparently blocking the main server thread at times, causing low TPS.
The problem is certainly not due to lack of RAM. We have allocated perhaps a little too much RAM to the server just to rule that out. It is possible that the problem is due to some programming error in Mojang's chunk I/O code which has the effect of serialising multithreaded I/O - effectively making it single-threaded. Or perhaps we have hit the limit of what the CPU can do in each I/O thread. We don't have much visibility into the internal state of the server in these situations, other than that timings report (image above). The situation requires more research and better tools.
It's possible that the chunk I/O issues may be exacerbated by Garbage Collector (GC) churn, whereby many short-term object allocations are overwhelming the GC's ability to free up memory in a timely manner, resulting in a costly full collection pause. So we're keen to try out a Java Virtual Machine (JVM) with the Shenandoah GC algorithm soon.
However, we're skeptical that Shenandoah will make a major difference to performance. We're mostly reliant on the PaperSpigot developers to improve chunk I/O. Mojang have stated that there will be no further 1.14 releases - 1.14.4 is the last - but have vowed to deliver performance improvements in 1.15. Unfortunately, that is a long way down the track.
WorldEdit has been failing on startup, taking down WorldGuard, LWC and various other plugins with it and necessitating world restoration from the most recent backup. This has happened about 3 times now. It doesn't make any sense. We've held off on reporting it until now in order to try the latest WorldEdit snapshot. We'll be reporting it to the WorldEdit project ASAP and in the meantime will add a small plugin to detect the failure and automatically stop the server before any damage can be done.
Horses, donkeys and apparently pigs were not dropping their inventories on death. That problem was fixed in Spigot build 2437 and made it's way onto PvE in PaperSpigot build 158.
Beds coming back from the end are unreliable. That turns out to be a bug in NerdSpawn. It's randomly selecting either your bed or world spawn as the spawn location. It's a dirty little plugin with a lot of undocumented, legacy code that's no longer needed and complicates maintenance. We'll fix that soon.
Portals coming back from the nether to the overworld don't quite get there, with a message about the player hitting the world border. This issue is reported in WorldBorder issue #131 and at least 3 duplicate reports so far. But it's not exactly a WorldBorder bug. The Spigot server is raising nonsensical teleport events when the player goes through a portal:
onPlayerPortal(): NETHER_PORTAL from (world_nether,296,68,321) to (world,2370,68,2568)
onPlayerTeleport(): NETHER_PORTAL from (world_nether,296,68,321) to (world,2370,68,2568)
onPlayerTeleport(): UNKNOWN from (world_nether,2370,68,2568) to (world_nether,2332,65,2466)
onPlayerTeleport(): UNKNOWN from (world,2332,65,2466) to (world,2332,65,2466)
PlayerTeleportEvents with the reason
UNKNOWN are the main problem. The first of these is completely nonsensical. It uses the overworld X, Y and Z coordinates but says the teleport happens in the nether. That's a problem for us because our nether world border is at +/-1500. Clearly there is a some kind of a bug in Spigot. We're currently liaising with those developers to get that fixed and we'll also talk to the WorldBorder plugin developer about whether WorldBorder can implement a work-around.
PaperSpigot build 160 fixes MC-156852, "phantom" blocks that the client thinks are deleted but the server does not. So that's good news for people mining ice, sandstone, netherrack and when using haste beacons.
The Case of the Missing Drops
There's a rumour going around in PvE chat that player death drops are despawning too fast - in under 5 minutes. It's the same rumour that was going around last rev, and probably the rev before that. It will probably be going around next rev.
I've personally looked at four such claims in recent memory. In the first case, on PvE Rev 23, the chunks were proven to be loaded by virtue of a player nearby. In the three cases that I have looked at in Rev 24 so far, every time, much more than 5 minutes had elapsed. The minimum time period between death and a modreq was 8 minutes. However, all three players stated that less than 5 minutes had elapsed, and in some cases that the chunks were definitely not loaded because nobody appeared on the live map.
I'm beginning to think we have an X File on our hands. Time dilation or "missing time" is a well known feature of alien abduction. Perhaps it is a feature of this alien planet we have landed on in Rev 24.
If you look on the live map and don't see anyone in the area, that does not mean that nobody is there. There could be invisible staff members or players who have hidden themselves from the live map with
/dynmap hide. It's very unlikely that drops are despawning too fast. There is nothing in the server configuration that removes drops. Whenever we test it, the drops stay on the ground for exactly 5 minutes (subject to tick rate scaling). Every time items do disappear, there is an alternate explanation that constitutes normal gameplay.