New Job As A System Administrator? Here Are 5 Things To Do First!

When I walked into the corporation I currently work for 20+ years ago, one thing I noticed was the backups for the office I was at; one of 17 satellite offices plus one main office. The backups succeeded at about a 30% rate. I started to troubleshoot and fix this system; It became apparent to the boss that something happened; for that particular office was now the only remote office that had nearly 100% backup success rate. So she handed me the other 16 remote office; all installed and configured by a different tech, for each office. Those too were at about 30% success rate; mostly due to Exchange backups using bricks backup ( PITA ) . Each remote office had a file server, a Domain Controller, and an Exchange server.

Over the next several months, I elevated each to nearly 100% success rate, monitored them daily, and corrected failed backups so the manager only saw 100%. Retention was a consistent 30 days, which was acceptable for their purposes. These backups were for file servers, and Exchange mainly. At about 2 1/2 years in, their Exchange system in the main office went down; I didn't handle their backups; but recognized the "symptom" immediately. I learned about it during a phone call, on monday, after about 30 seconds, told her I knew what the issue was; she suggested that I could not know, because they had spent Friday night, all day Saturday and Sunday with a number of techs, and Microsoft support on the line troubleshooting; no way I could know the answer in 30 seconds. I proposed to her to let me give it a try; I'd take a backup and fix one of the two ailing front ends. I quickly pointed out that they only backed up 1 mailbox server and 2 front ends daily; I managed the backups of 17 file servers and more importantly 17 Exchange servers; thus actually "more experienced" .. After I fixed the first front end, she called me back and asked me to fix the 2nd, and while at it, develop and put something in place so it would never happen again. I did..

Answer: They weren't backing up the front ends; thus not doing a full backup. Yes, I knew the answer in 30 seconds, actually less; Their team was backing up one Exchange server; I was backing up 17, all remote. Turns out the System Admin there was not backing up the front ends which led to the log files filling the disks. Answer was to use NT backup to back up the front ends ONCE with a FULL to purge the files; simple enough; took about 30 minutes; she could not believe that they had Microsoft on the line for 2 days and never came up with the answer: In reality, the System Admin misrepresented his backups to the MS tech; which puzzled the MS tech. And yes, at the time, if you backed up an Exchange system but did not perform a full backup, the log files folder will grow till it runs out of space; and there is really only one way to address the failure; run a full backup of the system. Nothing else, period.

BTW: Managers will NEVER believe the 30 second or less solutions to fix an issue that some other tech has been working on for months or years; I addressed numerous issues caused by this one particular System Admin, and another System Admin that was let go for other reasons; but if you know you are right, stick to your guns; after a while the boss will figure out who knows their stuff and who is full of BS. However, once they start finding out that tech A was working on something for 6+ Months or a year+, and tech B can nail it in literally a few minutes, well.. Sometimes they have trust in the wrong person and don't realize it. Personally, I hate cleaning up the messes of others that aren't worthy of the term "System Admin" or "Domain Admin" ..

I eventually, at about year 3, got offered the System Admin job at the main office, replacing the troublesome System Admin that had put in his notice. I wasn't really surprised to find out that he was the one managing the remote backups earlier; that the main site's backups succeeded at about 30% as well, but was surprised to find out that backup retention was an entire 48 hours; metadata removed at 49.. The boss told me that I was the ONLY one that she had considered as a replacement; simply because of my work ethics, and success with numerous aspects of the 17 remote offices, including various trips to all 17 ( nation wide ) to fix different things and working on improvements.

Backups will always be one of the most important challenges for system administration; it also helps you understand the other systems, their names of the servers, the functions, and which ones are key to the business. It's a chore, but it's where you can go from a zero to a hero rather easily; but if not taken seriously, you can go from Hero to ZERO just as fast. Also note that if you come in fresh, and want to work in a difficult area like backups, and volunteer for it, and succeed, that's an accomplishment that is going to be a "golden ticket" on your career path moving forward.

This type of content is very very good, Linux users are more likely to want to see this sort of thing.

good change of pace video, Jay. A lot of this is stuff that us long-tenured admins could get back to for improvement, too!

your hosting voice has gotten better with time, Jay. Now it's almost as good as your normal voice (like from the homelab show podcast, for example). Cool video. Keep up the good work!

ROFLMAO !! The first thing that I did was…..Kicked everyone off that had Sudo access to the Linux servers, that thought they knew what to do on a Linux servers. Including managers.

Really needed this! Hope to see more like it &/O greater depth! Keep it up Jay, Linux and Us thank you endlessly!

Useful!

I like your first point best ! … and your second point … WOW !

An untested backup is called Schrödinger backup! (Not my idea, but I love it.)

❤❤❤