August 22

How to Add a Hard Disk to an ESXi Virtual Machine

VMware ESXI Hard Drive

In an effort to consolidate my homelab, I recently deployed an ESXi host to my main server. There’s been a bit of a learning curve but overall it’s been a pleasant experience. In this tutorial, I will show you how to add a new hard disk to an ESXi virtual machine. (If you’re looking for hard drives, I usually use WD Reds that I shuck from a WD Easystore or Elements drive. They’re high capacity, well-priced, and reliable.)

Now, theoretically, this should be an easy task where you can just edit the VM and click the Add hard disk > New standard hard disk, specify the hard disk location and be done. Unfortunately, though, there’s a bug in the ESXi host web client and so instead this will be a tutorial on adding an additional hard disk to an ESXi guest via the command line (esxcli).

Starting with the bug in the ESXi host web client, if you add a new hard disk to a VM, and the primary datastore where the VM is located is full, you’ll be greeted with the following message and the save button will gray out:

The disk size specified is greater than the amount available in the datastore. You cannot overcommit disk space when using a thick provisioned disk. You must increase the datastore capacity, or reduce the disk size before proceeding.
ESXi error message

The disk size specified is greater than the amount available in the datastore. You cannot overcommit disk space when using a thick provisioned disk. You must increase the datastore capacity, or reduce the disk size before proceeding.

ESXi host web client bug: Save button is always disabled even despite changing location of hard disk. The save button being disabled will persist even after removing the offending "New Hard disk".
ESXi host web client bug: Save button is always disabled even despite changing location of hard disk. The save button being disabled will persist even after removing the offending “New Hard disk”.

Now, that’s all fine and good, but the problem is, the “Save” button stays that way even when you choose a datastore that does have space. Heck, the save button even remains disabled when you remove the new hard disk! It appears that the web client only evaluates once, cripples the save button, and never re-evaluates the settings ever again.

Another thing I’ve learned along the way is that vSphere != ESXi host web client and unfortunately almost all of VMware’s documentation refers explicitly to vSphere, not the ESXi web client, so in this tutorial I’ll be showing you how to use the command line if you don’t have vSphere set up.

Creating the Virtual Disk via the Command Line (esxcli and vmkfstools):

1. Enable SSH on the ESXi host by right-clicking on the host in the web client > Services > Enable Secure Shell:

Enabling SSH service in ESXi.
Enabling SSH service in ESXi

2. SSH into ESXi host. If you’re using Ubuntu, open a terminal and type the following:

ssh root@<insert IP address of ESXi host here>

3. Find the location of your datastore with the following command:

esxcli storage filesystem list
esxcli storage filesystem list
Note the Mount Point (i.e. the location of the volume)

4. Change directory into that location with the following command:

cd <copy Mount Point from above here>

5. Create your hard disk with the following command:

vmkfstools --createvirtualdisk 256G --diskformat thin <nameYourDriveHere>.vmdk

Note that the above creates a 256 GB virtual drive. If you want a different size, just change it.

vmkfstools --createvirtualdisk 256G --diskformat thin <nameYourDriveHere>.vmdk
vmkfstools –createvirtualdisk 256G –diskformat thin .vmdk

Adding the New Virtual Disk to Your VM:

At this point, we’ve created our virtual disk, but it currently only exists in isolation in the datastore. It isn’t connected to any VM so we need to add it to a VM.

6. Add the hard disk to your VM:

Here is the “choose your own adventure” part of the guide. I originally assumed that the majority of readers will want to add the virtual disk to their VM using the ESXi host’s GUI. That’s fine since this part of the ESXi Embedded Host Client isn’t broken. If that’s you, use the steps outlined in 6a.

However, based on reader feedback, I have learned that some readers are interested in also adding the new hard disk to the VM via the command line, so that both disk creation and hot-adding the disk to the VM are 100% command line. If that’s you, refer to 6b.

6a. Add the hard disk to your VM via the ESXi Embedded Host Client (i.e. the ESXi web client GUI):

We’re done with the command line now. Go back to your ESXi web client and edit your VM. Now click “Add a hard disk” but this time instead of a new hard disk, we’re going to select “Existing hard disk” and navigate to the .vmdk you created in step 5:

Add existing (new) hard disk in ESXi web client
Add existing (new) hard disk in ESXi web client

Defaults are fine. Don’t panic if your new hard disk shows 0 GB. The virtual disk just hasn’t been formatted yet. Save and boot up your VM. As you can see I’m using a Windows VM here.

6b. Hot Adding the Disk to the VM via the Command Line:

To hot-add our new virtual disk to the VM via the command line, we first need to find the ID of our VM. To do so, still SSH’d into the ESXi host, we issue the following command:

vim-cmd vmsvc/getallvms
Result of  vim-cmd vmsvc/getallvms showing Vmid of VMs
Result of vim-cmd vmsvc/getallvms showing Vmid of VMs

I want to add this to my Windows VM, so I want Vmid 1. Now to hot-add our new virtual disk, we simply issue the following command:

vim-cmd vmsvc/device.diskaddexisting <vmid> </vmfs/volumes/pathToDisk.vmdk> 0 1

Replace <vmid> and </vmfs/volumes/pathToDisk.vmdk> with the Vmid and path to your new virtual disk, respectively. The path has to be the full path to the disk you created in steps 3/4/5 above. The 0 at the end refers to your SCSI controller (typically 0) and the 1 refers to your SCSI target number (the 0 slot is occupied by the primary virtual disk your VM is using, so 1 is just the next available target. If you already have a second hard disk on your VM, you would increment this number and use 2 instead. You get the idea). In my case, the command would look something like this:

vim-cmd vmsvc/device.diskaddexisting 1 /vmfs/volumes/5d5d560e-962037c1-16f8-fcaa142fc77d/Windows10Storage/win10aux.vmdk 0 1

7. Add the Hard Drive to the OS:

When you boot up your VM and navigate to “This PC” in File Explorer, you still won’t see the hard drive. Again, don’t panic, this is just because we haven’t added the drive in Windows yet.

Open up Disk Management (you can search for it in Cortana) and you should see your new drive, albeit unformatted. Right-click on the hard drive and select “New Simple Volume”:

Disk Management - Adding unallocated disk to Windows
Disk Management – Adding unallocated disk to Windows

Follow the wizard and when you go back to “This PC”, voila, your new hard disk appears!

New disk installed on VM
New disk installed in VM
August 14

How To Set Up VLANs On An L3 Switch (HP 1910) With pfSense

This site is getting more traffic than I had ever anticipated and, in order to support this self-hosted site and my homelab, I have upgraded to a 1 gigabit internet connection. In doing so, the bottleneck in my homelab network has shifted from the internet connection to the router itself.

As a reminder, my previous homelab consisted of a DMZ interface on pfSense firewall with a second DMZ router sitting directly behind it (i.e. the WAN of the DMZ router connected to the interface of the DMZ pfSense interface port). This obviously led to a double NAT situation that I had to handle but aside from that, this network topology had worked well for me. With the upgraded internet package though, I noticed that I was only getting half (~500 mbps) of my theoretical download speed. I was unsure if the bottleneck was in my pfSense router or my DMZ router (or if my ISP was lying to me). I had thought my pfSense router might be the bottleneck, but when I did my download speed tests, none of my CPU cores pegged to 100% (in a pfSense router with a dedicated quad port NIC, the bottleneck is most likely to occur in the CPU with heavy traffic as the firewall rules are evaluated), so I reasoned that the pfSense firewall being the bottleneck was unlikely. As a quick, field expedient test, I set up another interface on a free port on pfSense. Speedtesting directly on that interface gave me my full download speed, so I knew the bottleneck was downstream of the connection and was on my DMZ router (not surprising since that was a cheap router running DD-WRT).

As my homelab has grown considerably since I first started out, I decided this was the perfect opportunity to replace my DMZ router and upgrade to an L3 switch.

VLANs and Switching

First, let’s start off with what a switch is. You are probably most familiar with a cheap unmanaged switch that allows you to plug it into another ethernet port and effectively split it into an addition 4-8 ports. This is known as an L2 switch – basically all the devices connected to the switch are on the same network and can communicate with each other. The L2 switch accomplishes this by keeping a MAC address table to keep track of what device is on which port. It then switches packets based on their MAC address/port.

One of the greatest features of a managed switch is that it allows you to create what’s called a VLAN (a virutal LAN). A VLAN lets you divide up (subnet) your physical network into separate logical ones.

Why would you want to do this? Well, my biggest motivation is security. By separating out the network into VLANs, you can cordon off the individual VLANs from each other. As an example, let’s say you had a business and you offered guest wifi. You obviously wouldn’t want your guests to be able to connect to your network and be able to access all of your business PCs. With VLANs, you could create two separate VLANs: one for your guest wifi, and another for your business network. With VLANs on a simple L2 switch, devices on those two networks can’t talk to each other since the devices on the your guest wifi effectively live on a separate network from your business network and there’s no routing between them.

What’s an L3 Switch?

An L3 switch allows you to take this functionality of a managed L2 switch a step further. In all but the most simple scenarios (like the guest wifi/business one above), there are scenarios where you want those VLANs to be able to communicate with each other. An L3 switch takes the routing functionality of a router and offloads it to the switch itself, allowing the switch to route traffic between VLANs. So an L3 switch gives you the security through the principle of compartmentalization (via VLANs), while also allowing you to route traffic between those VLANs on the switch itself.

I headed off to Ebay and picked up the following HP 1910-24G for a whopping $30:

HP 1910-24G L3 Switch
HP 1910-24G

Even though this is the lowest price I’ve ever seen for this switch, it turns out I still overpaid (we’ll get to that later).

While you’re there, be sure to pick up a rollover (console) cable. Trust me, you’ll be glad you did, especially when the seller fails to factory reset the switch before shipping and won’t give you the password. I selected this particular cable because it had the best reviews and the most compatibility with other vendors, not just HP. This particular console cable worked great with my HP 1910.

1. Set Up Our Switch

So now we have our switch. Power it up and connect an ethernet cable between your PC and the switch. You’ll need to assign your computer a static IP since a DHCP server isn’t connected to the switch. The exact IP address you need to assign is based on the default IP listed on the back of the switch. For example, if the default IP on the switch is 192.168.254.12, assign your local PC an address of 192.168.254.20. The default subnet is 255.255.0.0. Navigate to the router’s default IP address in a web browser.

Alternatively, you can connect the switch directly to your pfSense firewall/router and the DHCP server running on pfSense will assign it an IP address. You can then just navigate to that assigned IP address (which can be found on the pfSense GUI or by running nmap over your local network).

Once you navigate to the switch IP address, you’ll be presented with a login page. The default username is admin with a blank password:

HP Login Page at Switch's IP Address
HP Login Page at Switch’s IP Address

2. Set Up Static Routing

The first thing we’ll do is set the switch up so that our VLANs can reach the internet. You see, by using the switch as an L3 switch, all routing is handled locally by the switch. The problem is, our switch’s router doesn’t know how to get to the internet- it isn’t directly connected to the internet, it’s never seen the internet before, it doesn’t know what the internet is. That’s what our pfSense router does, so we need to tell our switch about the pfSense router. We accomplish this by creating a static route. To do so, navigate to Network > IPV4 Routing > Create:

Static routing settings - routes IPv4 traffic (that isn't local to switch's VLANs) out to the pfSense router. Note that this IP address is the IP address of the pfSense router on the interface the HP switch is plugged into.
Static routing settings – routes IPv4 traffic (that isn’t local to switch’s VLANs) out to the pfSense router. Note that this IP address is the IP address of the pfSense router on the interface the HP switch is plugged into.

Enter a destination IP address of 0.0.0.0 with a mask of 0.0.0.0. The next hop should be the IP address of your pfSense router (in my case, it is 192.168.2.1).

So what have we done here? What we’ve said to our switch is that if we’re trying to access any IP address (0.0.0.0 with a mask of 0.0.0.0), get there by sending the traffic through our next hop at 192.168.2.1 (i.e. get there by going through our pfSense router). Preference is inversely related to how strongly we should use that route so by giving it a high preference, we’ll continue to use the other dynamic routes the switch’s router has picked up on if at all possible (i.e. so VLAN-to-VLAN traffic will continue to be routed through the switch).

Keeping along this same train of thought, the pfSense router has no idea how to get response traffic from the internet back to the VLANs since it has no idea of the VLAN’s existence. Therefore, we also need to define static routes on the pfSense route for the return traffic.

To do so, we need to first define a gateway in pfSense by going to System > Routing > Gateways. When you click add you will be prompted for an interface and a gateway IP address. But what interface and gateway IP address should we use? Thinking about the return flow of traffic, pfSense doesn’t know how to get to our VLANs. It just knows it has traffic for an IP address it’s never heard of. But there is one device that does know where the hosts that belong to these IP addresses live- the switch. And where does that switch live from pfSense’s perspective? It lives in the DMZ. So we use the DMZ as our interface and we use the L3 switch’s DMZ IP address as the gateway:

Defining our HP 1910 gateway in pfSense
Defining our HP 1910 gateway in pfSense.

So now that we have a gateway specified, we can define the rest of our static route. Go to the “Static Routes” tab. You’ll then specify the destination network, the interface, and the gateway (that we just created):

Static routes for return traffic from pfSense to L3 Switch
Static routes for return traffic from pfSense to L3 Switch

Let’s summarize what we’ve done. The above static routes tell pfSense that for traffic destined for these networks, send the traffic through the DMZ interface, and use the L3 switch that’s located on that DMZ interface as the “next hop” to route the traffic.

Test it out by changing the gateway (of your IPV4 settings) on your PC to the IP address of your switch. You should now be able to navigate to the web and ping outside servers. N.B. You may also need to update your firewall rules (see end of article).

3. Create Our VLANs

Now we’re finally ready to create our VLANs. Go to Network > VLAN > Create:

Creating a VLAN on the HP 1910; here we are simply telling the switch that a VLAN with this ID exists.
Creating a VLAN on the HP 1910; here we are simply telling the switch that a VLAN with this ID exists.

4. Assign Ports to Your VLANs:

We now need to tell the ports which VLANs they apply to. To do so, go to Network > VLAN > Modify Port:

Network > VLAN > Modify Port; Assign a port as untagged and enter the applicable VLAN ID.
Network > VLAN > Modify Port; Assign a port as untagged and enter the applicable VLAN ID.

Select a port. Since the devices we’re going to connect to our switch aren’t “VLAN aware”, we are going to use an untagged port (meaning there is no VLAN tag added outbound on the port). In the VLAN ID box, specify the VLAN the device listed should be on.

5. Create VLAN Interface:

If we were going to use this switch as a simple L2 switch with our VLANs routed by our pfSense router (known as a “router on a stick” configuration), we wouldn’t need to complete this step. Creating the VLAN interface is what tells our HP 1910 switch to enable L3 routing. To create our VLAN interface, go to Network > VLAN Interface > Create:

Creating a VLAN Interface on the Switch. This tells the switch to route VLAN traffic and by assigning the switch an IP address, we are also defining the gateway for that VLAN.
Creating a VLAN Interface on the Switch. This tells the switch to route VLAN traffic and by assigning the switch an IP address, we are also defining the gateway for that VLAN.

Input the VLAN ID you created in the previous step, select Manual for the IPv4 address (unless you have an external DHCP server you want to use). The IPv4 address you specify here will be the IP for switch on this VLAN. Since we’re using it as an L3 switch, this also means this address will be your gateway address for any devices you connect to this VLAN.

Don’t forget to click “Apply” and save often.

That’s it!

Don’t forget to assign the correct IP addresses and gateway for the devices you have connected to the switch. As a reminder, they should be in the same subnet as the VLAN specified on that port and their gateway should be the IP address of the switch itself on that VLAN (the one we specified in step 5). Example: If the VLAN interface is defined as 10.0.20.0/24 (worded another way 10.0.20.0 with a subnet of 255.255.255.0), then your IP addresses should fall within the 10.0.20.1-10.0.20.254 range.

You may also have to change your pfSense firewall rules on the interface that the L3 switch is plugged into. For example, you likely have a rule that says allow traffic originating from LAN net to anywhere. For our traffic to reach the internet, you would need to reconfigure this rule would need to allow traffic originating from anywhere to anywhere since our VLANs lie within a separate subnet.

Category: Servers | LEAVE A COMMENT