Tech Admin blog

Tweak of the Week #2 - August 18, 2019

Tweak of the Week is our weekly roundup of server configuration changes where we list everything we changed during a specific calendar week.


General Change Log

  • 2019-08-14: totemo fixed the /dynmap wrapper so admins can use all commands.
  • 2019-08-14: Deaygo backed up dev host in preparation for OS distro upgrade.

Web Site Change Log

  • Nothing much.

Creative Change Log

  • Nothing much.

PvE Change Log

  • 2019-08-12: totemo updated Minecraft Overviewer to 1.14 and enabled PvE 24 cartos to render once per day.
  • 2019-08-12: totemo updated CommandHelper with a potential fix for WorldEdit failures, per this bug report.
  • 2019-08-14: Deaygo updated NerdyDragon for PaperSpigot 1.14.4.
  • 2019-08-14: totemo fixed the visibility limits on the mapworld live map and updated the dynmap plugin to 3.0-beta5. We may need to fly around and do spot renders to fill in the map. The full render only seems to show chunks that have been loaded by a player at some time.
  • 2019-08-14: totemo updated EasyRider to allow arbitrary saddle disguises.
  • 2019-08-16: totemo adjusted NoCheatPlus to allow elytras and slow fall potions.
  • 2019-08-16: totemo diagnosed a bug with NerdyDragon mobs not dropping picked up items. (ModReq 1362)
  • 2019-08-17: totemo improved documentation and added new features to EasySigns, as version 2.3.0.
  • 2019-08-17: totemo did initial investigation and plugin configuration of PAdmin secret project [REDACTED].
  • 2019-08-17: totemo fixed HelpHelp for PaperSpigot 1.14.4 and added conveniences for tracking the raw help text.
  • 2019-08-17: totemo fixed /myregions (use sudo() in CH, not runas()) and also granted permission to list own regions with /region list.
  • 2019-08-18: totemo reported a WorldGuard bug whereby ravagers can trample crops. Looks like it's caused by an omission in Spigot.
  • 2019-08-18: totemo installed and configured Java 12 and Shenandoah GC on dev and production hosts, for PvE and BungeeCord. Shenandoah is a low-latency, parallel garbage collector that avoids long garbage collection pauses.
  • 2019-08-18: totemo investigated a Y375 ceiling on elytras. It's NoCheatPlus, it's not configurable and it's not obvious where it occurs in the 2500-line SurvivalFly check source code. We'll report a bug, but don't get your hopes up. Actually it's a false positive in the CreativeFly check and it's fixed. :)

Tweak of the Week #1 - August 11, 2019

Tweak of the Week is our weekly roundup of server configuration changes where we list everything we changed during a specific calendar week.


General Change Log

  • Nothing much.

Web Site Change Log

  • 2019-08-07: Deaygo added a Tech blog.
  • 2019-08-11: Deaygo added past mods to the staff page and related tools.

Creative Change Log

  • 2019-08-04: Dumbo52 installed LightCleaner, a plugin that will allow admins to fix lighting in certain areas of the map. We previously used FastAsyncWorldEdit's //fixlighting for this, but as of 1.13, this command breaks lighting (as do many other WorldEdit operations) instead of fixing it. We'll investigate the root cause behind this in the coming weeks to find a better solution. Until then, players can submit a modreq to fix lighting in their region.
  • 2019-08-07: Dumbo52 modified /i to handle old (numeric) item IDs.
  • 2019-08-08: Dumbo52 re-enabled Creative's overviewer map. Because of the time it takes to render, this will only be updated every 3 days.
  • 2019-08-11: Dumbo52 restored the wooden shovel as the WorldEdit wand item. Reverting to the wooden axe was a symptom of a larger issue caused by an undocumented change in how FastAsyncWorldEdit reads its configuration files.

PvE Change Log

  • 2019-08-09: totemo forked WorldBorder and added a work-around for SPIGOT-5252 that fixes portals on PvE. Code submitted to WorldBorder as PR #152.
  • 2019-08-09: totemo raised a bug on WorldEdit failing to load and taking down WorldGuard and other dependent plugins.
  • 2019-08-10: totemo fixed NerdSpawn's handling of beds when returning from the end.
  • 2019-08-11: Deaygo made PvE 24 dynmap tiles visible.
  • 2019-08-11: totemo raised a bug on CommandHelper, which may be the root cause of WorldEdit failures.

Talking Tech #1 - August 2019

First Post

Deaygo has been working on the new nerd.nu web site, with some help from other techs, and we now have the technology to give you regular blog updates on the state of the servers. Thanks, Deaygo!


New Hardware

If you've been playing regularly in the last couple of months, it will not have escaped your attention that the new server hardware is much snappier than the old box. However, we'll make a note of it here for the sake of posterity. PvE restarts are now so quick that there is only just enough time to get from the lobby spawn to the PvE lobby portal before the server is open again.

The new server specs are documented on the Server page of the nerd.nu wiki. The main highlights are double the amount of RAM, a faster CPU with 2 more physical cores and an SSD that has doubled in size.


Creative Rev 34 and Minecraft 1.13.2

Creative got stuck on Minecraft 1.12.2 for some time while we waited for FastAsyncWorldEdit to update. But by mid June, a development server was running on 1.13.2 with the current Rev 34 worlds and updated plugins, courtesy of totemo with some help from Challenger2. The final step was for the CAdmins to do some testing. By the time they were satisfied with the state of it, at the end of July, Dumbo52 had rejoined the tech team, and his first major act was to copy the dev server configuration over to the main host. He's currently doing minor tweaks on the configuration to fix issues that slipped through the net during the testing phase.


PvE Rev 24 and Minecraft 1.14.4

Chunk I/O and Lag

Minecraft 1.14 was a very buggy update with serious performance problems. We run PaperSpigot, (a.k.a. Paper) which is an optimised server derived from Spigot, which is the continuation of the Bukkit project. The PaperSpigot devs describe 1.14 thus:

MC 1.14.x was a rushed release from Mojang and has significant performance issues in how it handles chunk I/O. It is not recommended for large servers. Extensive backups and patience are recommended. [Source]

The PaperSpigot project has an official statement on 1.14 performance on their web site here.

That's certainly been our experience on PvE so far as we approach the end of the second week of Rev 24. We seem to be able to run at 20 TPS (Ticks Per Second) up to about 50 players, but beyond that we fall off some kind of performance cliff. Chunks taking a long time to load and long (7-8 seconds) saves saves are the most obvious symptoms of our current performance woes. Chunk I/O is also apparently blocking the main server thread at times, causing low TPS.

The problem is certainly not due to lack of RAM. We have allocated perhaps a little too much RAM to the server just to rule that out. It is possible that the problem is due to some programming error in Mojang's chunk I/O code which has the effect of serialising multithreaded I/O - effectively making it single-threaded. Or perhaps we have hit the limit of what the CPU can do in each I/O thread. We don't have much visibility into the internal state of the server in these situations, other than that timings report (image above). The situation requires more research and better tools.

It's possible that the chunk I/O issues may be exacerbated by Garbage Collector (GC) churn, whereby many short-term object allocations are overwhelming the GC's ability to free up memory in a timely manner, resulting in a costly full collection pause. So we're keen to try out a Java Virtual Machine (JVM) with the Shenandoah GC algorithm soon.

However, we're skeptical that Shenandoah will make a major difference to performance. We're mostly reliant on the PaperSpigot developers to improve chunk I/O. Mojang have stated that there will be no further 1.14 releases - 1.14.4 is the last - but have vowed to deliver performance improvements in 1.15. Unfortunately, that is a long way down the track.


Major Bugs

WorldEdit has been failing on startup, taking down WorldGuard, LWC and various other plugins with it and necessitating world restoration from the most recent backup. This has happened about 3 times now. It doesn't make any sense. We've held off on reporting it until now in order to try the latest WorldEdit snapshot. We'll be reporting it to the WorldEdit project ASAP and in the meantime will add a small plugin to detect the failure and automatically stop the server before any damage can be done.

Horses, donkeys and apparently pigs were not dropping their inventories on death. That problem was fixed in Spigot build 2437 and made it's way onto PvE in PaperSpigot build 158.

Beds coming back from the end are unreliable. That turns out to be a bug in NerdSpawn. It's randomly selecting either your bed or world spawn as the spawn location. It's a dirty little plugin with a lot of undocumented, legacy code that's no longer needed and complicates maintenance. We'll fix that soon.

Portals coming back from the nether to the overworld don't quite get there, with a message about the player hitting the world border. This issue is reported in WorldBorder issue #131 and at least 3 duplicate reports so far. But it's not exactly a WorldBorder bug. The Spigot server is raising nonsensical teleport events when the player goes through a portal:

onPlayerPortal(): NETHER_PORTAL from (world_nether,296,68,321) to (world,2370,68,2568)
onPlayerTeleport(): NETHER_PORTAL from (world_nether,296,68,321) to (world,2370,68,2568)
onPlayerTeleport(): UNKNOWN from (world_nether,2370,68,2568) to (world_nether,2332,65,2466)
onPlayerTeleport(): UNKNOWN from (world,2332,65,2466) to (world,2332,65,2466)

The two PlayerTeleportEvents with the reason UNKNOWN are the main problem. The first of these is completely nonsensical. It uses the overworld X, Y and Z coordinates but says the teleport happens in the nether. That's a problem for us because our nether world border is at +/-1500. Clearly there is a some kind of a bug in Spigot. We're currently liaising with those developers to get that fixed and we'll also talk to the WorldBorder plugin developer about whether WorldBorder can implement a work-around.


Annoyances

PaperSpigot build 160 fixes MC-156852, "phantom" blocks that the client thinks are deleted but the server does not. So that's good news for people mining ice, sandstone, netherrack and when using haste beacons.


The Case of the Missing Drops

There's a rumour going around in PvE chat that player death drops are despawning too fast - in under 5 minutes. It's the same rumour that was going around last rev, and probably the rev before that. It will probably be going around next rev.

I've personally looked at four such claims in recent memory. In the first case, on PvE Rev 23, the chunks were proven to be loaded by virtue of a player nearby. In the three cases that I have looked at in Rev 24 so far, every time, much more than 5 minutes had elapsed. The minimum time period between death and a modreq was 8 minutes. However, all three players stated that less than 5 minutes had elapsed, and in some cases that the chunks were definitely not loaded because nobody appeared on the live map.

I'm beginning to think we have an X File on our hands. Time dilation or "missing time" is a well known feature of alien abduction. Perhaps it is a feature of this alien planet we have landed on in Rev 24.

If you look on the live map and don't see anyone in the area, that does not mean that nobody is there. There could be invisible staff members or players who have hidden themselves from the live map with /dynmap hide. It's very unlikely that drops are despawning too fast. There is nothing in the server configuration that removes drops. Whenever we test it, the drops stay on the ground for exactly 5 minutes (subject to tick rate scaling). Every time items do disappear, there is an alternate explanation that constitutes normal gameplay.