Build of the Week

/warp F1 has been completed this week!

Builder: marting11

View post on imgur.com

Complete album

Check out this build in game on Creative 1.13.2 at /warp F1

Posted by Bardidley on Sept. 21, 2019 in Creative

Tweak of the Week #6 - September 15, 2019

Tweak of the Week is our weekly roundup of server configuration changes where we list everything we changed during a specific calendar week.

General Change Log

• 2019-09-10: totemo looked into some problems using scoreboards on the fundraiser development server and produced a work-around.
• 2019-09-11: totemo fixed another NPE in TPControl.
• 2019-09-13: totemo did some work on system performance monitoring tools.
• 2019-09-13: totemo transferred the 2019 fundraiser server from development to the production host, and along the way wrote an internal tech note on how we transfer and sanitise the databases.

Web Site Change Log

• 2019-09-10: Deaygo did more work on the donation page.
• 2019-09-13: Deaygo did more work on the donation page while his wife gave birth (!).
• 2019-09-14: Deaygo did more work on the donation page.
• 2019-09-15: Deaygo finished the donation page.

Creative Change Log

• Nothing much.

PvE Change Log

• 2019-09-11: totemo corrected an error in the LWC configuration. You can now lock a chest even without build rights in the region (as per last rev).

Posted by totemo on Sept. 15, 2019 in Tech Admin

Padmin Ponderings

Slime Spawner has been found!

The second of four custom spawners on the map has been located by Sir_Didymus. You can find it with /place Slime_Grinder. And, yes, it was hidden in a swamp. I honestly thought I didn't hide that one very well. Thanks for all the offers of bribes, the messages asking for hints, and the accusations of forgetting to place the spawner before the rev started. It is always more fun to be involved and you all kept me involved for the entire hunt!

This leaves two spawners left to find: the creeper and the witch. Remember you are looking for a 1 wide by 2 tall pillar of bedrock with a sign on it. If you find it click the sign to claim the spawner! If these are not found soon we think the scientists might be able to help!

Suggestion Box Responses

We have responded to the next batch of suggestions. You can find the responses here

Skull Planet is now Open!

Designed by kumquatmay, skull planet is a place where you can purchase decorative skulls you can not obtain through dopplegangers. There is a transport ship ready at spawn to take you there! You can purchase a Custom Skull Token, then click the sign of the decorative skull you want. And, don't worry you cannot die on Skull Planet. Check out the reddit post here

Events

Admin Hunt

Thank you for all the suggestions for Admin Hunt arenas. We are going through them and have picked out a few we hope to use this rev. We expect to have the first hunt the weekend after the fundraiser.

Weekly Events

We would like to start up holding mini events weekly. This would change weekly from Spleef to Thimble, to PvP and archery events but we want your help! If you would like a week to be hosted in your town, build whatever style of arena you would like in your town and let us know when it is ready for use! More information will be coming soon

Other Events

Just kidding! No information on this until we are ready to release it, but something is being planned.

Around the Map

Pez, the dragon Slayer

No one will ever know the full story. What we do know was that the Pez252 will likely remain at the top spot of the dragon leaderboard for the duration of this rev.
We also know that Pez was trouble shooting the dragon fight, the dragon died and the announcement alerted those online that Pez252 killed the dragon in 3 seconds! Buzzie71 created an amazing "recreation" of the feat immortalizing the event forever. You can find that video here. Buzzie did check with Hanstra before releasing the animation, and Hanstra is the actual current leader with a time of 6 minutes.

Killing Platforms

Solace has announced SPUNK, or Solace Platform Used for Nether Kills. It can be found North of the Northwest Nether Portal.

Pez252 has built a killing platform in the end, directly above the spawn area.

Feature Build

Nepenthes

Welcome to Nepenthes.

The town is located in the South at -1373, 1797. Owned by Kirstae and Spook6 this is one place that is a must see this rev!

Named after the carnivorous plant also known as a pitcher plant, the natural beauty is amazing. Besides the town's namesake, you will also find other larger than life flora.

It is quite easy to spend a lot of time just looking around.

Be sure to visit!

Posted by Cujobear on Sept. 8, 2019 in PvE

Tweak of the Week #5 - September 8, 2019

Tweak of the Week is our weekly roundup of server configuration changes where we list everything we changed during a specific calendar week.

General Change Log

• 2019-09-05: totemo completed initial configuration of a development server for the fundraiser and loaded the world that was provided by the Head Admins.

Web Site Change Log

• 2019-09-04: Deaygo committed more changes to the donation page of the web site.

Creative Change Log

• Nothing much.

PvE Change Log

• 2019-09-02: totemo updated the server to PaperSpigot build 183. In principle, this resolves the bugs we raised for ravager crop grief and a systematic error of 0.1 (i.e. 10% chance) on the drop chance for mob equipment.
• 2019-09-02: totemo updated WorldGuard, resolving a bug where it blocked splash potions as PvP.
• 2019-09-04: totemo audited RecipeManager source code and opened a bug report for an exception on startup that renders the plugin inert.
• 2019-09-05: totemo raised a feature request with the WorldGuard project to protect campfires.
• 2019-09-08: totemo spent some time looking at a pull request to fix some SafeBuckets bugs.
• 2019-09-08: totemo raised a feature request to improve LogBlock logging of bamboo.
• 2019-09-08: totemo investigated limitations on /signtext not allowing owners of parent regions to edit signs. Getting a region parent is not possible with CommandHelper's SKCompat. That has to be added (per a feature request), or /signtext written as a regular Java plugin command.

Posted by totemo on Sept. 8, 2019 in Tech Admin

Tweak of the Week #4 - September 1, 2019

Tweak of the Week is our weekly roundup of server configuration changes where we list everything we changed during a specific calendar week.

General Change Log

• 2019-08-29: totemo fixed /i and /give to handle numeric IDs as well as new material names and namespaced material names. Tested on P and C.
• 2019-09-01: totemo enabled GC logging on BungeeCord. It's buttery smooth. Garbage collection is never triggered by low heap space. It runs at the 5 minute timeout and only pauses for about 0.2ms.

Web Site Change Log

• 2019-08-27: Deaygo made numerous changes to the website:
  • Started rewriting the donation page for the new web site.
  • Added commands to manage the list of past staff.
  • Add delete and underline tags for Markdown rendering.
  • Fixed blog absolute url.
  • Fixed minutes in last edited readout.

Creative Change Log

• Nothing much.

PvE Change Log

• 2019-08-27: totemo raised a Spigot bug concerning WorldGuard being unable to detect Ravagers breaking crops.
• 2019-08-28: totemo fixed a NullPointerException on player login and numerous concurrency issues in TPControl.
• 2019-08-28: totemo fixed Doppelganger to treat CAVE_AIR (Minecraft 1.13) as equivalent to AIR when checking the border around block shapes.
• 2019-08-30: totemo reported a Spigot bug affecting equipment drop chances.
  • The BeastMaster work-around from 6 months ago for this bug in Spigot for 1.13.2 is still in effect and equipment should drop as expected if mob types explicitly set drop chances. There is a subtle latent bug in BeastMaster mob equipment drops if a mob is configured to not drop vanilla items at all (doesn't apply in our configuration). That will be fixed in due course.
  • Doppelganger was re-tested and it was confirmed that doppelgangers do drop wither skeleton skulls as expected. Praise RNGesus. Doppelganger includes a work-around for a presumed vanilla drop chance bug affecting head drops from 6 years ago. Seen in that light, the current Spigot bug may in fact be very, very old.
• 2019-08-31: totemo opened a feature request with the Spigot project to implement API events for flame arrows igniting extinguished campfires, so that they can be denied by WorldGuard.
• 2019-08-31: totemo did more investigation of Java 12 garbage collector performance. GC pauses on PvE are now logged. Shenandoah does most of the GC work concurrently (while the JVM is running Minecraft code) and final pauses (where Minecraft stops executing) typically only take about 1.5 ms. With 19 players online, garbage collection happens every 50 seconds or so. This all looks fine to me. At higher loads (40+ players) the GC triggers every 20 seconds or so. Raising the heap size from 12GB to 16GB puts them back to about once every 50-60 seconds. Not sure if it's really necessary, but the RAM is available so we'll use it.
• 2019-09-01: totemo fixed permissions of the /pinfo and /petinfo CH aliases.

Posted by totemo on Sept. 1, 2019 in Tech Admin

Tweak of the Week #3 - August 25, 2019

Tweak of the Week is our weekly roundup of server configuration changes where we list everything we changed during a specific calendar week.

General Change Log

• 2019-08-21: totemo upgraded the BungeeCord (and lobby) NuVotifier plugins to resolve its failure to load in BungeeCord following the upgrade to Java 12.
• 2019-08-22: totemo explicitly configured the MariaDB timezone to UTC, to match the system timezone. This resolved an issue with LogBlock recording edits 4 hours earlier than chat logs and ModReqs, perhaps due to the database having observed a different system timezone setting when the host booted. The current time at the server can be checked with /date.

Web Site Change Log

• 2019-08-20: Deaygo added extra Markdown formatting options for the web site (blogs etc).

Creative Change Log

• 2019-08-23: Dumbo52 investigated lighting bugs in FastAsyncWorldEdit and is working to fix them. Hopefully this will be resolved next week.
• 2019-08-24: Dumbo52 enabled our LastSeen plugin and migrated our existing /seen data to the new storage format. Creative was previously using a CommandHelper script to handle the /seen command and data collection, but this was disabled in the 1.13 update. Unfortunately this means that /seen has no knowledge of players' logins over the last 3 weeks.

PvE Change Log

• 2019-08-19: Deaygo fixed NerdyDragon mobs picking up player drops (and not dropping them again). (ModReq 1362).
• 2019-08-19: totemo worked on reducing false positives for NoCheatPlus CreativeFly checks and realised that the Y375 elytra ceiling was configurable. Now set at aprox 255+1280.
• 2019-08-19: totemo investigated a possible bug in either the PaperSpigot Timings v2 facility or entity ticking.
• 2019-08-20: totemo prototyped [REDACTED], but don't worry. It's just a decoy!
• 2019-08-21: totemo updated PvE to the latest PaperSpigot build, which includes the asynchronous chunk loading patch. They still have more work to do, but it is supposedly a big improvement in our case, where the map is pre-generated.
• 2019-08-22: totemo enabled the new per player mob spawning option in PaperSpigot. This makes mob spawning fairer in that a player cannot starve other players of mobs by exhausting the global mob cap with a grinder. Each player has their own local mob cap.
• 2019-08-22: totemo reported a possible bug in PaperSpigot's Timings v2 performance profiling facility.
• 2019-08-23: totemo defaulted seating to off in the spawn region with a WorldGuard flag due to players warping through the ship's hull and opened a feature request to allow seating to be disabled by default.
• 2019-08-24: totemo wrote an internal tech note on JDK 12, Shenandoah and ZGC and is investigating better monitoring tools.

Posted by totemo on Aug. 25, 2019 in Tech Admin

Tweak of the Week #2 - August 18, 2019

Tweak of the Week is our weekly roundup of server configuration changes where we list everything we changed during a specific calendar week.

General Change Log

• 2019-08-14: totemo fixed the /dynmap wrapper so admins can use all commands.
• 2019-08-14: Deaygo backed up dev host in preparation for OS distro upgrade.

Web Site Change Log

• Nothing much.

Creative Change Log

• Nothing much.

PvE Change Log

• 2019-08-12: totemo updated Minecraft Overviewer to 1.14 and enabled PvE 24 cartos to render once per day.
• 2019-08-12: totemo updated CommandHelper with a potential fix for WorldEdit failures, per this bug report.
• 2019-08-14: Deaygo updated NerdyDragon for PaperSpigot 1.14.4.
• 2019-08-14: totemo fixed the visibility limits on the mapworld live map and updated the dynmap plugin to 3.0-beta5. We may need to fly around and do spot renders to fill in the map. The full render only seems to show chunks that have been loaded by a player at some time.
• 2019-08-14: totemo updated EasyRider to allow arbitrary saddle disguises.
• 2019-08-16: totemo adjusted NoCheatPlus to allow elytras and slow fall potions.
• 2019-08-16: totemo diagnosed a bug with NerdyDragon mobs not dropping picked up items. (ModReq 1362)
• 2019-08-17: totemo improved documentation and added new features to EasySigns, as version 2.3.0.
• 2019-08-17: totemo did initial investigation and plugin configuration of PAdmin secret project [REDACTED].
• 2019-08-17: totemo fixed HelpHelp for PaperSpigot 1.14.4 and added conveniences for tracking the raw help text.
• 2019-08-17: totemo fixed /myregions (use sudo() in CH, not runas()) and also granted permission to list own regions with /region list.
• 2019-08-18: totemo reported a WorldGuard bug whereby ravagers can trample crops. Looks like it's caused by an omission in Spigot.
• 2019-08-18: totemo installed and configured Java 12 and Shenandoah GC on dev and production hosts, for PvE and BungeeCord. Shenandoah is a low-latency, parallel garbage collector that avoids long garbage collection pauses.
• 2019-08-18: totemo investigated a Y375 ceiling on elytras. It's NoCheatPlus, it's not configurable and it's not obvious where it occurs in the 2500-line SurvivalFly check source code. We'll report a bug, but don't get your hopes up. Actually it's a false positive in the CreativeFly check and it's fixed. :)

Posted by totemo on Aug. 18, 2019 in Tech Admin

Padmin Meeting

Life has slowed down a bit and we were finally able to have our first "official" padmin meeting of Rev 24. Here are some of the things we talked about:

New Rev

We want to thank totemo for getting the rev launched pretty much on time. He had a few tricks up his sleeve which made the change over take a bit less time on launch night and I believed it worked well enough to use those tricks again in the future.

We also want to thank the moderators that helped spruce up the ship before our launch into the new rev. The decorations and secrets are amazing.

Finally, thank you for being patient while we continue to deal with issues as fast as we can.

Suggestion Box

We took some time to go through the current suggestions on PvE and posted the responses on the forum here. Keep those suggestions coming. We do like to hear from you.

Upcoming Events

It is still early rev, and the 3 of us have barely had time to build in our towns but we are excited to start hosting some events. Our first big event will be an Admin Hunt. We haven't decided on a date and time yet but we do ask for your help. We are in need of a good arena. In the past players have liked when builds or towns from previous revs were used. So check out this post if you would like to suggest a build from a previous revision to use for admin murderizing.

But while we work out the details of the first admin hunt, don't forgt to go hunting for those spawn secrets. (No one has found cheezy's yet and he's happy he his it so well) I did not have a chance to place a spawn secret before the launch, so I will be adding a small scavenger hunt this weekend.

We also had a discussion about PvP events. Many revs ago I remember player run PvP fights being hosted almost every weekend, but recently it seems that players wait and expect the admins to set up and host these events. I, for one, would love to see the return of more player run events. I enjoyed heading to the town early to check the place out before heading to the arena to be killed over and over again. So how do we bring back player run get togethers? Is it a matter of newer players not knowing they can host events? Or do players just not like traveling to other towns and would rather be teleported to an arena not on the map? Cheezy is planning something, but we do want to encourage everyone - If you have an idea for an event (PvP, spleef, thimble, anything) go for it! If you need help getting the word out, we will help.

Posted by Cujobear on Aug. 17, 2019 in PvE

Tweak of the Week #1 - August 11, 2019

Tweak of the Week is our weekly roundup of server configuration changes where we list everything we changed during a specific calendar week.

General Change Log

• Nothing much.

Web Site Change Log

• 2019-08-07: Deaygo added a Tech blog.
• 2019-08-11: Deaygo added past mods to the staff page and related tools.

Creative Change Log

• 2019-08-04: Dumbo52 installed LightCleaner, a plugin that will allow admins to fix lighting in certain areas of the map. We previously used FastAsyncWorldEdit's //fixlighting for this, but as of 1.13, this command breaks lighting (as do many other WorldEdit operations) instead of fixing it. We'll investigate the root cause behind this in the coming weeks to find a better solution. Until then, players can submit a modreq to fix lighting in their region.
• 2019-08-07: Dumbo52 modified /i to handle old (numeric) item IDs.
• 2019-08-08: Dumbo52 re-enabled Creative's overviewer map. Because of the time it takes to render, this will only be updated every 3 days.
• 2019-08-11: Dumbo52 restored the wooden shovel as the WorldEdit wand item. Reverting to the wooden axe was a symptom of a larger issue caused by an undocumented change in how FastAsyncWorldEdit reads its configuration files.

PvE Change Log

• 2019-08-09: totemo forked WorldBorder and added a work-around for SPIGOT-5252 that fixes portals on PvE. Code submitted to WorldBorder as PR #152.
• 2019-08-09: totemo raised a bug on WorldEdit failing to load and taking down WorldGuard and other dependent plugins.
• 2019-08-10: totemo fixed NerdSpawn's handling of beds when returning from the end.
• 2019-08-11: Deaygo made PvE 24 dynmap tiles visible.
• 2019-08-11: totemo raised a bug on CommandHelper, which may be the root cause of WorldEdit failures.

Posted by totemo on Aug. 11, 2019 in Tech Admin

Talking Tech #1 - August 2019

First Post

Deaygo has been working on the new nerd.nu web site, with some help from other techs, and we now have the technology to give you regular blog updates on the state of the servers. Thanks, Deaygo!

New Hardware

If you've been playing regularly in the last couple of months, it will not have escaped your attention that the new server hardware is much snappier than the old box. However, we'll make a note of it here for the sake of posterity. PvE restarts are now so quick that there is only just enough time to get from the lobby spawn to the PvE lobby portal before the server is open again.

The new server specs are documented on the Server page of the nerd.nu wiki. The main highlights are double the amount of RAM, a faster CPU with 2 more physical cores and an SSD that has doubled in size.

Creative Rev 34 and Minecraft 1.13.2

Creative got stuck on Minecraft 1.12.2 for some time while we waited for FastAsyncWorldEdit to update. But by mid June, a development server was running on 1.13.2 with the current Rev 34 worlds and updated plugins, courtesy of totemo with some help from Challenger2. The final step was for the CAdmins to do some testing. By the time they were satisfied with the state of it, at the end of July, Dumbo52 had rejoined the tech team, and his first major act was to copy the dev server configuration over to the main host. He's currently doing minor tweaks on the configuration to fix issues that slipped through the net during the testing phase.

PvE Rev 24 and Minecraft 1.14.4

Chunk I/O and Lag

Minecraft 1.14 was a very buggy update with serious performance problems. We run PaperSpigot, (a.k.a. Paper) which is an optimised server derived from Spigot, which is the continuation of the Bukkit project. The PaperSpigot devs describe 1.14 thus:

MC 1.14.x was a rushed release from Mojang and has significant performance issues in how it handles chunk I/O. It is not recommended for large servers. Extensive backups and patience are recommended. [Source]

The PaperSpigot project has an official statement on 1.14 performance on their web site here.

That's certainly been our experience on PvE so far as we approach the end of the second week of Rev 24. We seem to be able to run at 20 TPS (Ticks Per Second) up to about 50 players, but beyond that we fall off some kind of performance cliff. Chunks taking a long time to load and long (7-8 seconds) saves saves are the most obvious symptoms of our current performance woes. Chunk I/O is also apparently blocking the main server thread at times, causing low TPS.

The problem is certainly not due to lack of RAM. We have allocated perhaps a little too much RAM to the server just to rule that out. It is possible that the problem is due to some programming error in Mojang's chunk I/O code which has the effect of serialising multithreaded I/O - effectively making it single-threaded. Or perhaps we have hit the limit of what the CPU can do in each I/O thread. We don't have much visibility into the internal state of the server in these situations, other than that timings report (image above). The situation requires more research and better tools.

It's possible that the chunk I/O issues may be exacerbated by Garbage Collector (GC) churn, whereby many short-term object allocations are overwhelming the GC's ability to free up memory in a timely manner, resulting in a costly full collection pause. So we're keen to try out a Java Virtual Machine (JVM) with the Shenandoah GC algorithm soon.

However, we're skeptical that Shenandoah will make a major difference to performance. We're mostly reliant on the PaperSpigot developers to improve chunk I/O. Mojang have stated that there will be no further 1.14 releases - 1.14.4 is the last - but have vowed to deliver performance improvements in 1.15. Unfortunately, that is a long way down the track.

Major Bugs

WorldEdit has been failing on startup, taking down WorldGuard, LWC and various other plugins with it and necessitating world restoration from the most recent backup. This has happened about 3 times now. It doesn't make any sense. We've held off on reporting it until now in order to try the latest WorldEdit snapshot. We'll be reporting it to the WorldEdit project ASAP and in the meantime will add a small plugin to detect the failure and automatically stop the server before any damage can be done.

Horses, donkeys and apparently pigs were not dropping their inventories on death. That problem was fixed in Spigot build 2437 and made it's way onto PvE in PaperSpigot build 158.

Beds coming back from the end are unreliable. That turns out to be a bug in NerdSpawn. It's randomly selecting either your bed or world spawn as the spawn location. It's a dirty little plugin with a lot of undocumented, legacy code that's no longer needed and complicates maintenance. We'll fix that soon.

Portals coming back from the nether to the overworld don't quite get there, with a message about the player hitting the world border. This issue is reported in WorldBorder issue #131 and at least 3 duplicate reports so far. But it's not exactly a WorldBorder bug. The Spigot server is raising nonsensical teleport events when the player goes through a portal:

onPlayerPortal(): NETHER_PORTAL from (world_nether,296,68,321) to (world,2370,68,2568)
onPlayerTeleport(): NETHER_PORTAL from (world_nether,296,68,321) to (world,2370,68,2568)
onPlayerTeleport(): UNKNOWN from (world_nether,2370,68,2568) to (world_nether,2332,65,2466)
onPlayerTeleport(): UNKNOWN from (world,2332,65,2466) to (world,2332,65,2466)

The two PlayerTeleportEvents with the reason UNKNOWN are the main problem. The first of these is completely nonsensical. It uses the overworld X, Y and Z coordinates but says the teleport happens in the nether. That's a problem for us because our nether world border is at +/-1500. Clearly there is a some kind of a bug in Spigot. We're currently liaising with those developers to get that fixed and we'll also talk to the WorldBorder plugin developer about whether WorldBorder can implement a work-around.

Annoyances

PaperSpigot build 160 fixes MC-156852, "phantom" blocks that the client thinks are deleted but the server does not. So that's good news for people mining ice, sandstone, netherrack and when using haste beacons.

The Case of the Missing Drops

There's a rumour going around in PvE chat that player death drops are despawning too fast - in under 5 minutes. It's the same rumour that was going around last rev, and probably the rev before that. It will probably be going around next rev.

I've personally looked at four such claims in recent memory. In the first case, on PvE Rev 23, the chunks were proven to be loaded by virtue of a player nearby. In the three cases that I have looked at in Rev 24 so far, every time, much more than 5 minutes had elapsed. The minimum time period between death and a modreq was 8 minutes. However, all three players stated that less than 5 minutes had elapsed, and in some cases that the chunks were definitely not loaded because nobody appeared on the live map.

I'm beginning to think we have an X File on our hands. Time dilation or "missing time" is a well known feature of alien abduction. Perhaps it is a feature of this alien planet we have landed on in Rev 24.

If you look on the live map and don't see anyone in the area, that does not mean that nobody is there. There could be invisible staff members or players who have hidden themselves from the live map with /dynmap hide. It's very unlikely that drops are despawning too fast. There is nothing in the server configuration that removes drops. Whenever we test it, the drops stay on the ground for exactly 5 minutes (subject to tick rate scaling). Every time items do disappear, there is an alternate explanation that constitutes normal gameplay.

Posted by totemo on Aug. 7, 2019 in Tech Admin