Common VMware Hypervisor IaaS Cloud Problems and Solutions
Have you ever encountered a situation where a virtual machine (VM) refused to start through a remote console? Or were there difficulties with a slow-running VM? In this article, we will talk about the problems that you may encounter when working with a cloud environment, and their solutions. / photo David Cruz CC
Suppose the linux-base-01 virtual machine is used to transfer business critical information from a research laboratory to the cloud using a clipboard. A typical task is when a copy operation is called on one side and an insert is called on the receiving side. The user of the system found that the clipboard is not working correctly, therefore it does not allow copying data from the laptop to the VM directory.
To solve the problem, you need to open a browser and connect to vSphere Web Client. In the control panel window that opens, after authentication, we turn to the user directory. The user provided us with a fragment of the study in the form of a readme.txt file. Now we need to solve the following problems: restore the clipboard, copy the text from the user file on the desktop and paste it into the protocol.txt document of the linux-base-01 machine in the cloud.
You can enable or disable the use of the buffer both for a separate virtual machine and for all ESXi host machines. To activate the copy and paste functions for a specific VM, you must complete these steps:
In the window that opens, enable the use of the copy / paste functions:
Then we save the changes. Note that after this you need to restart the VM, transferring it to the status of power on. For more detailed information on resolving problems with the clipboard, you can use the materials from the VMware knowledge base.
Suppose that a large number of requests from users for various operations have accumulated in the queue. The employee believes that this was due to the low performance of the virtual machine that could not cope with the influx of requests. To solve this problem, you should first turn to the troubleshooting methodology in order to clearly identify the source.
Most often, performance problems are caused by disks (if you are interested, then this is a good study on the topic of "disk dependency"). While the network causes this much less. However, you should still analyze the performance of the subsystems, identify bottlenecks and make a plan for their elimination or optimization.
In our case, first connect to the problem virtual machine using terminal access. We assume that the user VM already has tools for collecting performance statistics, so we see information about the host name, CPU utilization, RAM, IP address, etc.
We will also use the SPECjbb utility, which is a benchmark for measuring performance created by the Standard Performance Evaluation (SPEC). The results obtained during testing and presented in the figure below indicate that the perf-01 system is experiencing performance problems.
This was due to overuse of processor resources. It turned out that on the esx-01a host (dual-core processor), in addition to perf-01a, another virtual machine was deployed - perf-02a. At the same time, perf-01a uses 2vCPU, and perf-02a uses 1vCPU, which exceeds the number of host cores available and clogs the system.
There are several ways to solve the problem. Firstly, you can transfer perf-01a to a less busy ESXi host, for example esx-02a, which is practically not loaded. Secondly, do the same, but transfer perf-02a. Thirdly, you can use the redistribution of computing resources for the machine perf-02a.
The established limit will reduce the load on the system. Also note that increasing the number of virtual processors can improve performance. To do this, there is a special line in the Virtual Hardware tab for adding additional processor resources.
However, it must be remembered that if the host hosting the virtual machine has only two physical processors, an increase in the number of CPUs can lead to resource overruns and reduced performance. You should also make sure that the operating system and applications can work in multi-threaded mode.
Configuring virtual machines with more virtual CPUs than is actually required for the workload can lead to increased resource utilization, potentially affecting the performance of busy systems.
Most guest operating systems idle during idle hours, often following HLT or MWAIT instructions, while some operating systems use the busy-waiting mechanism. All this leads to the consumption of resources that could be available for other tasks.
There are two reasons why a user may encounter this problem:
To determine the cause of the problem and fix it, we connect to vSphere Web Client - we use an account with administrator rights for access. In the control window that opens, go to the Administration> Single Sign-On> Users and Groups tab, after which we find the problem user account. From the image below it follows that the Murphy account has the Disabled attribute, that is, it is turned off and cannot be used for access.
To fix the “malfunction”, simply turn on the account by calling the context menu and setting the Enable attribute.
There are situations when you or system users cannot delete existing snapshots of a virtual machine. This problem manifests itself if the backup server or another VM blocks the base disk or the previous snapshot file, preventing the possibility of merging. This also happens if the process freezes when creating a snapshot.
To solve the problem, you can combine the snapshot by removing .vmdk from the backup system. Another option is to clone the last snapshot disk to a new one. You can read more about these operations here .
Another common problem when working with snapshots is the error that occurs when creating a snapshot for ESXi / ESX virtual machines, which statesthat the created file exceeds the maximum permissible size. It appears when the total size of the snapshot file and the additional space required for the operation exceeds the maximum file size allowed by the data storage of the virtual machine.
To solve this problem, either change the location of the VM configuration files, or install workingDir in a data warehouse with enough space for snapshots. Snapshots are created in workingDir, and by default this directory contains .vmx configuration files. To move virtual machine disks and / or configuration files, use Storage vMotion or the cold migration method with moving files.
PS Stay tuned for new content on our IaaS blog., because in the following articles we will continue to consider options for solving common problems of a virtual environment based on the VMware hypervisor.
PPS What else are we writing on our blog:
Clipboard problem: copy / paste does not work
Suppose the linux-base-01 virtual machine is used to transfer business critical information from a research laboratory to the cloud using a clipboard. A typical task is when a copy operation is called on one side and an insert is called on the receiving side. The user of the system found that the clipboard is not working correctly, therefore it does not allow copying data from the laptop to the VM directory.
To solve the problem, you need to open a browser and connect to vSphere Web Client. In the control panel window that opens, after authentication, we turn to the user directory. The user provided us with a fragment of the study in the form of a readme.txt file. Now we need to solve the following problems: restore the clipboard, copy the text from the user file on the desktop and paste it into the protocol.txt document of the linux-base-01 machine in the cloud.
You can enable or disable the use of the buffer both for a separate virtual machine and for all ESXi host machines. To activate the copy and paste functions for a specific VM, you must complete these steps:
- Verify VMware Tools
- Connect to vSphere Web Client
- Connect to vDC with linux-base-01
- Turn off the virtual machine, turning it into power off status
- Select a virtual machine and go to the Summary tab
- Go to the Edit Settings window
- Open virtual machine options: VM Options> Advanced> Configuration Parameters> Edit Configuration
In the window that opens, enable the use of the copy / paste functions:
Then we save the changes. Note that after this you need to restart the VM, transferring it to the status of power on. For more detailed information on resolving problems with the clipboard, you can use the materials from the VMware knowledge base.
Lack of performance
Suppose that a large number of requests from users for various operations have accumulated in the queue. The employee believes that this was due to the low performance of the virtual machine that could not cope with the influx of requests. To solve this problem, you should first turn to the troubleshooting methodology in order to clearly identify the source.
Most often, performance problems are caused by disks (if you are interested, then this is a good study on the topic of "disk dependency"). While the network causes this much less. However, you should still analyze the performance of the subsystems, identify bottlenecks and make a plan for their elimination or optimization.
In our case, first connect to the problem virtual machine using terminal access. We assume that the user VM already has tools for collecting performance statistics, so we see information about the host name, CPU utilization, RAM, IP address, etc.
We will also use the SPECjbb utility, which is a benchmark for measuring performance created by the Standard Performance Evaluation (SPEC). The results obtained during testing and presented in the figure below indicate that the perf-01 system is experiencing performance problems.
This was due to overuse of processor resources. It turned out that on the esx-01a host (dual-core processor), in addition to perf-01a, another virtual machine was deployed - perf-02a. At the same time, perf-01a uses 2vCPU, and perf-02a uses 1vCPU, which exceeds the number of host cores available and clogs the system.
There are several ways to solve the problem. Firstly, you can transfer perf-01a to a less busy ESXi host, for example esx-02a, which is practically not loaded. Secondly, do the same, but transfer perf-02a. Thirdly, you can use the redistribution of computing resources for the machine perf-02a.
The established limit will reduce the load on the system. Also note that increasing the number of virtual processors can improve performance. To do this, there is a special line in the Virtual Hardware tab for adding additional processor resources.
However, it must be remembered that if the host hosting the virtual machine has only two physical processors, an increase in the number of CPUs can lead to resource overruns and reduced performance. You should also make sure that the operating system and applications can work in multi-threaded mode.
Configuring virtual machines with more virtual CPUs than is actually required for the workload can lead to increased resource utilization, potentially affecting the performance of busy systems.
Most guest operating systems idle during idle hours, often following HLT or MWAIT instructions, while some operating systems use the busy-waiting mechanism. All this leads to the consumption of resources that could be available for other tasks.
Denied access to the vSphere Web Client console
There are two reasons why a user may encounter this problem:
- The user account has been disabled by the administrator.
- The user several times specified incorrect authorization data, and the account was blocked.
To determine the cause of the problem and fix it, we connect to vSphere Web Client - we use an account with administrator rights for access. In the control window that opens, go to the Administration> Single Sign-On> Users and Groups tab, after which we find the problem user account. From the image below it follows that the Murphy account has the Disabled attribute, that is, it is turned off and cannot be used for access.
To fix the “malfunction”, simply turn on the account by calling the context menu and setting the Enable attribute.
Common Snapshot Issues
There are situations when you or system users cannot delete existing snapshots of a virtual machine. This problem manifests itself if the backup server or another VM blocks the base disk or the previous snapshot file, preventing the possibility of merging. This also happens if the process freezes when creating a snapshot.
To solve the problem, you can combine the snapshot by removing .vmdk from the backup system. Another option is to clone the last snapshot disk to a new one. You can read more about these operations here .
Another common problem when working with snapshots is the error that occurs when creating a snapshot for ESXi / ESX virtual machines, which statesthat the created file exceeds the maximum permissible size. It appears when the total size of the snapshot file and the additional space required for the operation exceeds the maximum file size allowed by the data storage of the virtual machine.
To solve this problem, either change the location of the VM configuration files, or install workingDir in a data warehouse with enough space for snapshots. Snapshots are created in workingDir, and by default this directory contains .vmx configuration files. To move virtual machine disks and / or configuration files, use Storage vMotion or the cold migration method with moving files.
PS Stay tuned for new content on our IaaS blog., because in the following articles we will continue to consider options for solving common problems of a virtual environment based on the VMware hypervisor.
PPS What else are we writing on our blog:
