Musings from security guide –Part 02

in ESX3i all logs are lost during reboot. Therefore a syslog server is required to record and archive all ESX Server 3i logs since a simple reboot will erase all activity of all users.

ISO Images consumes a lot of disk space since they are not compressed when they are created.

Users should create hash checksums on all ISO OS images on the ESX server before utilizing the ISO OS images for virtual machines.

Shares and Resource allocation: Minimum and Maximum resource settings within ESX Servers are absolute values, whereas shares are used to give preference to a guest OS when resource is scare. Minimum guarantee a specific amount of a resource to the virtual machine but deny that much of the resource to other virtual machine, While Maximum deny the virtual machine a portion of the resource while allowing other virtual machine more access to that resources. Do not configure the minimum virtual machine CPU and memory setting equal the total physical amount available. Use reservations,shares and limits to allocate resources.

Time management, synchronizing the virtual machine with the ESX Server is the preferred method for time synchronization.

Musings from security guide –Part 01

• The ESX server virtual switch port groups will be configured with any value between 2 and 4094. Utilizing VLAN1 will cause a denial of service since the ESX Server drops this traffic. The maximum port group that may be configured on a virtual switch is 512. Each port group is identified by a network label and a VLAN ID.

• Ports Groups may have VLAN ID between  0 –4095.

• VLAN ID 4095 specifies that the port group should use trunk mode or Virtual Guest Tagging (VGT) mode.

• A value of Zero or blank VLAN ID is default value for External Switch Tagging(EST). EST is default configuration for all virtual switches within ESX Server. EST mode has 1-to-1 relationship, the number of VLAN’s are limited to the number of physical network adapter ports assigned to ESX.

• Virtual Switch Tagging(VST) allows virtual switch to handle it’s own VLAN tagging. This processing is handled by Pnic and this overhead never comes to VMkernel. Each physical switch port that connects to virtual switch is configured in trunk mode. VLAN’s can span across multiple PSwitch. VLAN is enabled by trunked link connecting the virtual switch and PSwitch thru frame tags. Trunk links can carry the traffic of multiple VLANs simultaneously.Within Switch fabric, switches uses frame tagging to direct frames to the appropriate switch and port. Frame tagging assigns frame id prior to traversing trunked link. After the frame reaches the access link, VLAN ID is removed and the end device receives the frame.

• Each Virtual Nic (VNic) has two MAC Address. effective and initial MAC address. Both the MAC address are same when they are first created.

• Forged Transmits (set to accept by default): When effective MAC Address and initial MAC address are different, which means effective MAC address is always compared with initial MAC address.it is considered as forged transmits.

• MAC Address Changes (set to accept by default): When effective MAC address is changed compared to initial one.

• Promiscuous mode: When promiscuous mode is applied, all virtual machine connected to virtual switch have potential of reading all packets.

• STP is not supported on vSwitch.Spanning Tree Protocol (STP) is either needs to disabled or Port fast needs to be enabled on PSwitch.

The vpxuser has privileges of a root user on the ESX server host,, but has no file privileges on the ESX server console. The vpxuser is created when the ESX server host is attached to Virtual Center. It is not present on the ESX Server host unless the host is being managed through VirtualCenter.

Virtual Center has two default roles defined, system roles and sample roles. System Roles are permanent and the permissions associated with these roles cannot be changed. All changes made to permissions of custom roles are effective immediately not requiring users to log off and log back in.

ESX3.5 Notes -Part02

Lab Manager 2.5.1 does not support ESX Server 3.5.
All hosts in a VMware HA cluster must have DNS configured so that the short host name (without the domain suffix) of any host in the cluster can be resolved to the appropriate IP address from any other host in the cluster.

If a host is added to a cluster, you can no longer create child resource pools of that host. You can create child resource pools of the cluster if the cluster is enabled for Distributed Resource Scheduler (DRS).

You cannot use VMotion to migrate a virtual machine with a guest operating system with 16GB of memory or more to ESX Sever 3.5 hosts or earlier. Resize the guest operating system memory or migrate to a compatible version of ESX Server 3.

Using VI Client or VI Web Access ensures that the starting sectors of partitions are 64K aligned, which improves storage performance.

In centralized license server mode, license files are located at the following default location on the machine running the VMware license server: C:\Program Files\VMware\VMware License Server\Licenses. This is different from VirtualCenter 2.0, where the default location of the license file was C:\Documents and Settings\All Users\Application Data\VMware\VMware License Server\vmware.lic. which no longer exists.

The VI Client installer installs Microsoft .NET Framework 2.0 on your machine. If you have an older version, the VirtualCenter Server installer upgrades your version to version 2.0.

While installing ESX Server 3.5, the option to create a default network for virtual machines is selected by default. If you proceed with installing ESX Server 3.5 with this option selected, your virtual machines share a network adapter with the service console, which does not provide optimal security.

Manage remote console connections—You can now configure VirtualCenter 2.5 to set the maximum number of allowed console connections (0 to 100) to all virtual machines.

VirtualCenter 2.5 provides an unlicensed evaluation mode that doesn’t require that you install and configure a license server while installing VirtualCenter 2.5 and ESX Server 3.

Virtual Center 2.5 can Manage up to 200 hosts and 2000 virtual machines
ESX Server 3.5 supports 256GB of physical memory and virtual machines with 64GB of RAM.
ESX Server hosts support for up to 32 logical processors
SATA support—ESX Server 3.5 supports selected SATA devices connected to dual SAS/SATA controllers
ESX Server 3.5 introduces support for N-Port ID Virtualization (NPIV) for Fibre Channel SANs. Each virtual machine can now have its own World Wide Port Name (WWPN).

VMotion migration of virtual machines with local swap files is supported only across ESX Server 3.5 hosts and later with VirtualCenter 2.5 and later

Enhanced HA provides experimental support for monitoring individual virtual machine failures. VMware HA can now be set up to either restart the failed virtual machine or send a notification to the administrator.

Storage VMotion simplifies array migration and upgrade tasks and reduces I/O bottlenecks by moving virtual machines to the best available storage resource in your environment.Migrations using Storage VMotion must be administered through the Remote Command Line Interface (Remote CLI)

VirtualCenter 2.5 provides support for batch installations of VMware Tools where VMware Tools can now be updated for selected groups of virtual machines. VMware Tools upgrades can now be scheduled for the next boot cycle

Logs,vpxd

Where is the vm-support dumping it’s data ?

= > /var/lib/vmware/hostd/docroot/downloads

Vpxd-#.log is circular in nature, which means they start over again with vpxd-0.log when the size limit is reached. You either check the vpxd-index file for checking what the current log index is or easier is to sort by date.

Virtual center logs rotate at 5MB and also when vpxd is started

How to’s

How to find version of virtual center you are running?

D:\Program Files\VMware\VMware VirtualCenter 2.0>vpxd.exe -v
VMware VirtualCenter 2.0.1 build-32042

---

Few ESX Commands

vmkping – use vmkernel to ping a device

esxcfg-vswif for configuring network interface

vm-support to run diagnostic commands

esxupdate -l query query patches installed on ESX Server

esxcfg-rescan vmhba1 to rescan HBA’s

ps -ef | grep hostd To check hostd is running

esxcfg-mpath -l to list path details

---

Important Log and their locations

VMKernel Logs = > /var/log/vmware/

VIClient logs = > /var/log/vmware/hostd.log

Virtual Center interaction logs = > /var/log/vmware/vpx/vpxa-*.log

VMWare Patch activity logs = > /var/log/vmware/esxupdate.log

[root@esx2007a config]# cat vmware-sites
FULLTIME_SITES_TID 00000061
+ 1:8042,8042,8043 esx2007e vmware #FT_Agent_Port=8045
+ 2:8042,8042,8043 esx2007d vmware
+ 3:8042,8042,8043 esx2007b vmware
+ 4:8042,8042,8043 esx2007a vmware
+ 5:8042,8042,8043 esx2007c vmware

License server logs = > %ALLUSERSPROFILE%\Application Data\VMware\VMware License Server\lmgrd.log

---

STORAGE

When ESX Server scans the SAN, each HBA reports all LUNs visible on the storage network; each LUN reports an ID that uniquely identifies it to all nodes on the storage network. After detecting the same unique LUN ID reported by the storage network, the VMkernel automatically enables multiple, redundant paths to this LUN, known as multi-pathing.ESX Server uses a single storage path for a particular LUN until the LUN becomes unavailable over this path. After noting the path failure, ESX Server switches to an operational path.

---

VIRTUAL CENTER:

SQL Server authentication will work predictably on local and remote database servers. NT authentication requires VC to run with a service account that has access to the DB; “Local System”account does not have access to remote servers.

vpxd.exe VirtualCenter Server service

vpxd.cfg VirtualCenter Server configuration

vpxa VirtualCenterAgent

If you create partition from virtual center then that partition is automatically aligned, in case you are creating partition using vmkfstools then you will have to manually align the partition using fdisk. But this is not enough you also have to align partition at guest level as well, which can be easily achieved using diskpart if the os is windows.

Virtual Center Database size calculator

How to calculate DB size for Virtual Center. This is quick excel sheet calculator from VMware, hope you will find it useful. I got this link from Advance technical design guide

www.vmware.com/support/vi3/doc/vc_db_calculator.xls

Basic System Administration -Part 04

If you remove users from the VirtualCenter domain, they lose permissions to all objects in the VMware Infrastructure and will not be able to log on again. Users who are currently logged on and are removed from the domain retain their Vmware Infrastructure permissions only until the next validation period (the default is every 24 hours)

Vmware doesn’t not explicitly restrict users from with same login and password from accessing and taking action within the VC.

If you rename user domain account, it becomes invalid in VC and same applies to group but before that(for groups only) you need to restart virtucal center.

Following activities can be scheduled as Tasks

• Change the power state of a VM
• Create a VM template
• Move a VM with Vmotion
• Create a VM
• Make snapshot of VM
• Customize VM
• Add a Host

When you remove an object (such as a folder, datacenter, cluster, or resource pool), VirtualCenter removes all child inventory objects (such as datacenters, clusters, hosts, and virtual machines contained within the object). All the associated tasks and alarms are also removed. Assigned processor and migration licenses are returned to available status. Virtual machines that were on a managed host remain on the host, but are no longer managed by VirtualCenter.

This was the last part of the series from basic administration task,this pdf is in more details and end in 364 pages. There should be more information which be might be useful for VCP. I might add more to this series soon.

VMWARE HA

Clustering in VMWare is based upon customer requirements.

Cluster-in-a-Box: Both the Nodes in same Physical Hosts, this type of configuration is suitable in case there is possibility of data crashes or administrative errors, but there is no cover if ESX host fails on hardware front.

Cluster-across-Boxes: Both the nodes are placed on seperate ESX host, and this takes of ESX host’s hardware failure.

Physical-to-Virtual Cluster: Here Node A is actually physical box and Node B is Virtual Machine in ESX host, acting as standby host.

VMWARE HA solutions has some advantages which not very obvious. But we should any case apply VM HA for one simple reason, if the ESX host fails, all VM’s at least get started at other host. You don’t have to manually do that. Downtime will be Non-Zero

VMHA and VC 2.0 deals only with Host failures, for VM’s (Node failure) you monitor Heart Beat using Alarm

PRE-REQUISITES VMHA:

• Each host must be able to poweron VM’s i.e. Each host must have access to VM’s files, in other words all VMotion requirements are met.
• ESX server is reachable when you type it’s fully qualified domain name

For VMHA heartbeats it is recommended to set

• Two service console port on different virtual switch
• One service console with NIC teaming enable at virtual switch level

VMHA is fully integrated with DRS, which means when your host fails and all VM’s are moved to different hosts, DRS takes care of resource management. VMHA is reactive solution, which means it will act only when one or more host fails but VMDRS is proactive solution, it is always best to implement both VMHA & VMDRS

Failover capacity: When you enable cluster, two important configurations you need to do and they are again dependant upon client’s requirement.

1. Number of host allowed failures allowed
   

   Maximum is 04 and Minimum is 01. This configuration help HA to determine if there are enough resources to power on VM in the cluster. But it is we who decided how much redundant capacity to be made available.

2. Admission Control
   1. Do not power ON VM if they violate availability constraints (Selected as default option)
   2. Allow virtual machines to be powered on if they violate availability constraints

Depending upon adminission control option you select, VM will be either powered ON or NOT. These values help VMHA to balance and calculated enough resource across hosts in case there is any host failures. Current failover capacity under Cluster’s summary tab informs how many hosts are available at that time to hold the VM’s

We only need to provide number of host, rest like resources required to power on VM’s across these host or only 1 host is alive, decision like this is taken by VMHA. If resources are not enough VMHA wouldn’t all VM’s to be powered ON(default option). You can force VMHA to start VM’s(when you like the constraints to be voilated), in this case Cluster will show RED sign, which means failover might not be guaranteed. It is not recommended that you work with red clusters. Also if you have 3 hosts and 2 fails cluster will turn RED.

So when you enable VMHA, you should design in such a way that hosts in ESX will be able to handle additional VM’s without any over utilization of resource.

For example: Two ESX Host having equal capacity handling 50 VM’s each. We should design in way that each Host should be able to handle 100 VM’s.

---

VCB -Backup Strategies

There are two things when you think of backup of virtual machines

• Application backup which is called File Level Backup
• Entire VM backup Image Level Backup, ( which is quite easy, cause you just need to back VMDK)

File Level Backup: It is recommended that you put all your data in Non-System Disk, it brings it’s own advantages.

Backup Proxy Server is required for carrying out this task, this has been implemented especially to remove backup overheads from ESX/VM’s. This would be Windows 2003 server, with Backup software installed(for example netbackup) which has VCB plugin to carry out the task.

COMPONENTS involved in VCB back process are:

• Hostd: On ESX Server and interacts with Virtual center
• VM to be backup:
• Backup Proxy server with 3rd party software installed on it.
• VCB Framework which consists of
  • vcbMounter
  • vLUN driver
  • Integration module
    • Pre & Post-backup scripts which ties with
  • Backup application (for example Netbackup)

---

VCB WORKFLOW:

• Backup application starts backup job as per pre-schedule time
• Pre-backup script is intiated by backup software 
  • Quiesces NTFS/FAT (only in case of MS Guest OS),this ensure no write operations are pending
  • Puts VM in snapshot mode
  • Snapshot is taken and put’s VM into normal opertion
  • Backup software mounts this Snapshot for File Level Backup, and selected files are copied. (Done by Backup client)
  • For Image level back, entire disk is export to Backup proxy server.(Done by backup client)
• Post-backup script is called
  • which unmounts VM snapshot from backup proxy
  • Takes VM out of snapshot mode, commits any changes made to the disk during the snapshot mode.

Restoring backups done using VCB approach

Restoring file/Images taken via backup is not straight forward. There are three approaches for this

• Self-Service restore : Backup agent Installed on each and every VM
• Per-group restore : Select VM’s which will do restore work (i.e. install backup agent only these VM’s) and then get someone to restore files of those specific VM’s
• Centralized restore: Backup agent is installed only on Backup Proxy and restore file/Image on backup proxy. After you can uses windows share to copy data over the location

These approachs differs from each other at one level i.e. present of backup agents. Backup agents here are only doing restore work.

For Image level backup you can use VCBMounter to backup entire virtual machine in the service console. VCBMounter quiesce the snapshot of the VM and export the setfiles which can be later on used to restore using VCBRestore.  This can be done only from Service Console.For file level back you have to use third party backup software.

Creating resource Pools

VMKernel manages all memory, except the memory that is allocated to Service console. VM will only power on when Server is sure to allocate that VM’s it’s reservations. This is calculated by Server by checking how much unreserved resources are available and if that meets the reservation of the VM.

Let’s learn how to create resource pools.Resource pool can be created on individual hosts, only if hosts are not clustered. In order to create resource pool, you need to have following information in hand.

1. Name: Name of the resource pool
2. Shares: Number of shares to be allocated to every VM
3. Reservation: Minimum resources guranteed to VM’s
4. Expandable reservation: There are two options YES/NO, if say YES, if reservation of existing resource pools are not available they can be used from Parent resource pool, If select NO, then host won;t be able to power ON the VM.
5. Limit: Maximum resources any VM, under this resource pool would get.

Similiar option you would get for memory configuration. Once this step is completed you might get yellow triangle, which suggest something is incorrect. 

Now resource pool is created, let’s play around with it.

Select resource pool(left hand-side), select on summary tab (right-side), you see below mentioned screen

I’ve underline things which could be of worth observation when you see them in Virtual center.

Now select resource allocation, resource allocation gives what reservation have been used, what is available, In order understand this clearly, let take look at picture below

You can see currently there are no reservation defined in this resource pool, but you still see CPU/Memory reservation used,

because VM-Marketing has been allocated reservation which is consumed from total reservation available

Other important information you get from here is the unreserved status, this unreserved status helps you in allocating resources in future. You also see resources are set to unlimited field, there is custom shares defined for particular VM.

Of course it is true, you can certainly change values under resource pools by editing it after you create resource pools

Lets discuss another important topic, How to add VM to existing Resource pool or move vm to another resource pool. You can vm to existing pools in two ways

• When you create VM, you can assign VM to any specific resource pool
• You can also drag and drop VM to any resource pool even after it is created

NB: you don;t need to power off VM in order to just move it to different resource pool.

When you move drag & drop VM to resource pool, its Memory/CPU/Shares/Limit don’t change, this change is reflected only when the VM is powered ON, in power off state resource pool’s resources are not affected by it or not being allocated to this new VM. Though you will observe in resource allocation tab Reservations for both CPU/Memory will reflect the changes in value. Also a point to be noted if Resource pool is not in a position to allocate the reservation for this VM, it generate error and move will fail.

---