Looking for help deploying ESXi on two different subnets

Hello all, obligatory first post remark

Let me start by saying that I have an extensive VMware (regarding setup and terminology) and networking background, so don’t pull any punches. However, I’m attempting something I’ve never tried before.

A coworker and I are attempting a side project in which we are setting up a server farm, all of which donate their resources to COVID-19 research. In the interest of keeping things easily manageable, we have deployed ESXi on to each of these servers, and have spun up simple ubuntu VMs on each that can reach out to their intended targets (across the internet) and receive their workload.

However, we quickly realized that we were eating up far too many IP’s on the routable subnet we tagged for this project. We were using 3 IPs per server: one for server management (i.e. iDRAC, iLo), one for ESXi, and one for the VM.

We decided to deploy a private non-routable subnet to resolve this issue. This way, we could migrate the server management and ESXi to the private subnet, and only have the VM be routable. This would cut our IP usage by two-thirds. We then decided to create one server that would have access to both networks. This server would have a VM on it that could manage both the servers and ESXi, as well as be able to monitor the statistics of the farming VMs. This same server also has vCenter running in a separate VM.

We have two separate physical switches that these devices are connected to. “Switch A” being for the private subnet, and “Switch B” being the routable network with internet access. I configured vSwitch0 to use both vmnic0 and vmnic1. vmnic0 is connected to Switch A, and vmnic1 is connected to Switch B.

Cue the issues.

I cannot for the life of me figure out how to configure ESXi to be able to work the way I need it to without jumping the two switches together. As far as I can tell, we shouldn’t be having issues. Assume the IPs are appropriately configured.

The last thing that I thought before leaving the office for the day was that the Gateway configuration was causing me issues. Naturally, the VMs get set to a Port Group, which is assigned a switch, which has the physical NICs connected to it. The port group also has a particular VMKernel attached to it. The VMKernel is what applies the Gateway to the Port Group. Realizing that I cannot set two Gateways to a single port group, I used the CLI to create a new Port Group with a Gateway in the private subnet, but now I cannot figure out how to get a VM to use the Port Group that has the proper Gateway assigned to it. It literally doesn’t appear in my networks when I try to select the network the vNIC is trying to attach to.

Can anyone assist? I really appreciate the insight.

View Reddit by BrokenGQView Source

Related Articles

One Comment

  1. Make it easy and create 2 separate vSwitches. Assign a NIC to each. The vmkernel port is the management interface for esxi and has a gateway. Those settings have no affect on VMs in a port group.

    VSwitch 0 – vmk for esxi mgmt

    vSwitch 1 – port group for VMs (VMnetwork)

Leave a Reply