c45y Posted September 13, 2015 Report Share Posted September 13, 2015 Starting to outline a quick process for this disk replacement, feel free to edit this post to include details or format nicely ( Make sure to cross out items if you complete work ) Create new BungeeCord instance on creeper, pointing to the existing servers on newnerd Change DNS entry for c & p to point to creeper - leave s and nerd.nu webserver on newnerd Wait for DNS TTL to expire for entries so all of our players will be operating on the new bungee Schedule downtime for PvE ( as it is most affected currently ) and rsync the server files across, export all pve database tables and import on creeper mysql instance change bungeecord entry for PvE to point to local server Start PvE with whitelist enabled and verify that permissions, LWC, logblock are all functioning correctly - disable whitelist after testing Repeat steps 4-6 for Creative Turn survival OFF, it is due to be shutdown anyway from memory Ensure any remaining server directories on /ssd/ are no longer required, archive any maps that have not been Kill webserver Export forum/website database tables from mysql and backup on the SATA drive - possibly copy to creeper tarball and copy the website directory(s) to the SATA drive - possibly copy to creeper Power off server, allow on-site technicians to replace drive ( might need to mkfs once the box comes back up ) Repeat process 2-7, swapping target servers from creeper to newnerd Untar server files to the web directory and import mysql dump of forums, starting webserver I'll clean the formatting and start work on it in a few days once my index finger is usable again 3 Link to comment Share on other sites More sharing options...
redwall_hp Posted September 14, 2015 Report Share Posted September 14, 2015 For clarification, if it's not obvious, newnerd is the primary server—the naming being a relic from when we migrated servers ~3 years ago—and creeper is our secondary box. Link to comment Share on other sites More sharing options...
Sir_Didymus Posted September 14, 2015 Report Share Posted September 14, 2015 I think it was mentioned in the meeting that E might be taken down during this change over, but it's not mentioned here that I can see? Is there any confirmation of that (just for planning reasons)? Link to comment Share on other sites More sharing options...
Mrloud15 Posted September 14, 2015 Report Share Posted September 14, 2015 Question for the techadmins, would you guys be ok with me making the post below to update everyone on what's going on? Hello everyone, We currently believe one of the main sources of the lag affecting the servers is a result of a failing SSD on our main box. The techadmins are in the process of moving the necessary data from the failing SSD to our secondary box before contacting the host to replace the SSD. We don't know how long this will take since it depends on the availability of our techadmins and our host. This may result in some downtime for some of our serveries including pve, creative, CS:GO, website, etc., so make sure to keep an eye on the subreddit and forums for any further announcements. Other things are also being done and looked into and we will make further announcements once there is something to update you guys on. Link to comment Share on other sites More sharing options...
c45y Posted September 14, 2015 Author Report Share Posted September 14, 2015 That would be great, thanks mrloud Link to comment Share on other sites More sharing options...
SwitchViewz Posted September 15, 2015 Report Share Posted September 15, 2015 As this seems to be top-priority above everything else right now due to how badly it is effecting the new PvE Rev, is there a aimed timelime to have this all completed by? Obviously the sooner the better, but is there any ETA or goal for it? Link to comment Share on other sites More sharing options...
Mrloud15 Posted September 20, 2015 Report Share Posted September 20, 2015 It's been about a week since this post was made, has any progress been made? Link to comment Share on other sites More sharing options...
redwall_hp Posted September 21, 2015 Report Share Posted September 21, 2015 Well, Dumbo figured out that there's a 4 hour hardware service window, so it seems like it would be wasteful to set everything up on the other server and deal with updating DNS, only for a four hour downtime. A better strategy is: Put the word out that we'll be having a scheduled downtime on a certain day Back up the SSD Power down for service Wait until it's done Copy everything from the backup to the new SSD, make sure everything's working, and unwhitelist We'd lose the website during the four hour window, but reddit, Mumble (which is hosted elsewhere) and IRC would all be available still. Link to comment Share on other sites More sharing options...
Silversunset01 Posted September 21, 2015 Report Share Posted September 21, 2015 If we know about it ahead of time we can also swap out all of the server alerts with a notice. That may help spread the word since not everyone reads the forums :/ Link to comment Share on other sites More sharing options...
SwitchViewz Posted September 21, 2015 Report Share Posted September 21, 2015 Not sure if this is possible but it's an idea I figured I'd pass by you guys. Since you guys are not going to be moving any servers over and just backing it up there won't be a need to use our second box. E is currently hosted on the second box, perhaps we could host a small little event like a snowball fight on E for the 4 hours or so the servers are down? That way we can keep people entertained while waiting. Link to comment Share on other sites More sharing options...
redwall_hp Posted September 23, 2015 Report Share Posted September 23, 2015 A ticket has been opened, and the provider agrees that the drive needs to be replaced. Details haven't been hashed out yet, but the process has started. We've got a timetable we're aiming for ourselves, but we don't want to announce it yet since we have no idea if the provider will be okay with it or not. Basically, 10PM Saturday my time (so early morning Eastern) we want to run down our preparations and have the server powered down for service by the following hour. Then they have a four-hour window where they'll get to it and perform the repairs. But we don't want to announce anything until we have confirmation that it's acceptable with the provider. So, don't go around telling people this date, because if things fall through and we have to change it, the unnecessary fallout will be annoying. (People already are irritable enough about the lag.) Once we've got the schedule set in stone, we'll ensure there's an announcement on the subreddit and forums, tweet it, and change the server MOTDs. Hopefully there will be a few days of lead-up time, but it's not the end of the world if notice is a little short, since the plan is for a reasonably off-peak time. 2 Link to comment Share on other sites More sharing options...
c45y Posted September 24, 2015 Author Report Share Posted September 24, 2015 Not sure if this is possible but it's an idea I figured I'd pass by you guys. Since you guys are not going to be moving any servers over and just backing it up there won't be a need to use our second box. E is currently hosted on the second box, perhaps we could host a small little event like a snowball fight on E for the 4 hours or so the servers are down? That way we can keep people entertained while waiting. The bungee instance that allows people to connect to the private IP of creeper is hosed on the main box, so E will appear down, even if the server isn't actually off We could open up some firewall ports to allow direct connections to E if all the techs think that is a good idea, mcbouncer would need to be set up in that case as Bungee wouldn't be handling our bans Link to comment Share on other sites More sharing options...
Trooprm32 Posted September 24, 2015 Report Share Posted September 24, 2015 The bungee instance that allows people to connect to the private IP of creeper is hosed on the main box, so E will appear down, even if the server isn't actually off We could open up some firewall ports to allow direct connections to E if all the techs think that is a good idea, mcbouncer would need to be set up in that case as Bungee wouldn't be handling our bans Honestly sounds like too much work for you guys just for 4 hours 1 Link to comment Share on other sites More sharing options...
SwitchViewz Posted September 24, 2015 Report Share Posted September 24, 2015 Honestly sounds like too much work for you guys just for 4 hours The bungee instance that allows people to connect to the private IP of creeper is hosed on the main box, so E will appear down, even if the server isn't actually off We could open up some firewall ports to allow direct connections to E if all the techs think that is a good idea, mcbouncer would need to be set up in that case as Bungee wouldn't be handling our bans I don't want to put any large strain on the techs. It was just merely an idea if it was a easy thing to do. Link to comment Share on other sites More sharing options...
Silversunset01 Posted September 24, 2015 Report Share Posted September 24, 2015 Switch just set up a TF2 Saturday Morning Gaming session ;) 2 Link to comment Share on other sites More sharing options...
Recommended Posts