VMware

Question on shutting down iscsi network.

I’ve got 11 hosts in a blade chassis that talk to a few different sans. Next weekend our network guy needs to do a LONG over due firmware upgrade on our switches in the chassis that handle this traffic. Right now we can’t fail over to one if the other is down (part of the reason we are doing this upgrade), so all the VM’s (about 300 in total) will need to be shutdown. My plan is to do that, then shut all the affected hosts down as well (and make sure the vcenter vm is documented to which host it’s running on). Once the firmware is done, bring the host up, check the datastores then start brining the vm’s back up.

Has anyone else ever had to do anything like this? Any gotcha’s or caveats I should be mindful of?


View Reddit by admlshakeView Source

Related Articles

9 Comments

  1. Yeah…been through something similar before. Your plan is about the best that you can do. Don’t be lured into thinking that you can use this downtime to sneak in some patches…you don’t want to add any other variables into the process in case things don’t come up as planned. Just a suggestion, create a host affinity rule that says the VCSA should run on host #1 and/or #2 (or whatever you determine is most appropriate in your environment). That way if you have some unexpected outage in the future, you already KNOW where to find it and can bring those hosts online first.

  2. Story time: had a very similar scenario a few years ago. To log on the switch it needed RADIUS / TACACS which depended on AD being available. All servers down mean no AD.

    AD was also required to access the password vault to connect using local creds on the switches.

    So I moved 2 VMs on a local datastore for just the vault and AD.

    Rest of them were down.

  3. This sounds like the start of an IT horror story, this isn’t a casual question. this requires a strong battle plan.

    Make sure you have our of band access to everything including your laptop/desktop.

    Have backups of everything, preferably ones that don’t live on your SAN.

    Make sure you understand your shutdown and startup of your env. For example does your Vcenter
    require auth from your AD env? Who runs dns. What breaks If it can’t get a dhcp reservation? Who really needs to start first in your environment.

    Make sure you have app owners or power users confirm the state of their apps prior to the shutdown and confirm it after it turns back on. Have a testing plan for each application. You want to avoid a, ‘well this worked before your upgrade’ no it didn’t Karen.

    Have all the old config from your network gear backed up and know what firmware versions they are running. For that matter know the same info about VMware and your blade chassis.

    Open proactive cases if possible with your vendors. Provide them the info as well as the proposed changes, make sure you google them to see if there are any gotchas out there

    Make sure you have all the passwords you need locally

    I’d you don’t feel comfortable with this put a stop to the work and get in the people to do it right. Upgrading a old SAN can go so badly. Better to be cautious then reckless. Also save the emails you send to cover your ass if they make you do it anyways.

  4. If you’ve got local storage in any if your hosts, migrate your critical infrastructure to that (vcenter, AD, DNS, DHCP, etc.) It’ll make bringing it all back up again much easier.

  5. With everything powered off there shouldn’t be any issues unless something goes wrong with the FW upgrade. If the switch he’s upgrading is bricked due to the upgrade can you still operate?

  6. Sounds like you’ve got a solid plan. Should be okay. Hopefully the network guy has saved the configs / write memory or whatever term is used else that could be a fun conversation post reboot

  7. I did a similar thing a few years ago where I needed to physically relocate servers, storage, and switching to a new rack. Once everything is down, the biggest thing for us was to make sure we 1. Still had some network connectivity (Ever tried to google a problem with no internet?) and 2. Documented the exact order we needed to start things back up in.

    For us, we still used an external (and physical) Windows vCenter and a physical DC at the time, so the order was easy. Network, Domain, Storage, vCenter, hosts, VM’s.

    Good luck!

  8. Make sure you build extra time into your shutdown. I’ve been caught by half applied windows updates before that take extra time that apply during shutdown. If you have that, make sure you note which ones so you can check them on the way back up.

    When you are booting the VMs after the switch upgrade make sure you boot the VMs in batches and not all at once and create a boot storm that grinds your storage arrays to a crawl.

  9. Might be a good idea to reboot the sans and/or vm hosts before the firmware upgrade happens and turn on one vm just as a sanity check before proceeding.

    Read oliland1 comment!!

    Always have your hardcoded logins handy if AD is going down!

Leave a Reply

Your email address will not be published. Required fields are marked *

Close