Lets Design, Implement and do Administration of ESX3

Virtualization with VMWare Infrastructure 3.0

Archive for the ‘Advance Concepts’ Category

How to find Queue depth of HBA ?

Posted by Preetam on April 30, 2009

Here is way

How to find queue depth for HBA ?

[root@pz-vmware07 scripts]# tail /etc/vmware/esx.conf
/vmkmodule[0003]/enabled = “true”
/vmkmodule[0003]/module = “tg3.o”
/vmkmodule[0003]/type = “nic”
/vmkmodule[0004]/enabled = “true”
/vmkmodule[0004]/module = “cciss.o”
/vmkmodule[0004]/type = “scsi”
/vmkmodule[0005]/enabled = “true”
/vmkmodule[0005]/module = “lpfcdd_732.o”
/vmkmodule[0005]/options = “lpfc0_lun_queue_depth=16 lpfc1_lun_queue_depth=16 lpfc_tgt_queue_depth=31”
/vmkmodule[0005]/type = “fc”

 

Or

[root@pz-vmware07 scripts]# esxcfg-module -l
Device Driver Modules
Module Type Enabled Loaded
vmkapimod vmkapimod true true
vmklinux linux true true
bnx2 nic true true
tg3 nic true true
cciss scsi true true
lpfcdd_732 fc true true
[root@ci-vmware07 scripts]# esxcfg-module -g lpfcdd_732
lpfcdd_732 enabled = 1 options = ‘lpfc0_lun_queue_depth=16 lpfc1_lun_queue_depth=16 lpfc_tgt_queue_depth=31’

Posted in Advance Concepts, How to, Storage, VMWare | Leave a Comment »

How to check which is your Primary node in HA ?

Posted by Preetam on September 22, 2008

[root@my-vmware01 root]# cat /opt/LGTOaam512/log/aam_config_util_listnodes.log
KEY: -z VAL: 1
KEY: domain VAL: vmware
KEY: cmd VAL: listnodes
CMD: hostname -s
RESULT:
————-
my-vmware01

CMD: /opt/LGTOaam512/bin/ft_gethostbyname my-vmware01 |grep FAILED
RESULT:
————-

list_nodes
CMD: /opt/LGTOaam512/bin/ftcli -domain vmware -connect my-vmware02 -port 8042 -timeout 60 -cmd listnodes
RESULT:
————-
        Node              Type              State
———————–  ————    ————–
  my-vmware01            Primary      Agent Running
  my-vmware02            Primary      Agent Running

Total time for script to complete:  0 minute(s) and 1 second(s)

Courtesy: Yellow Bricks

Posted in Advance Concepts, How to, Logs, VMWare | Leave a Comment »

VMware launches VI Ops portal

Posted by Preetam on September 14, 2008

Amazing site (http://viops.vmware.com/home/index.jspa) is launched by VMWare. And great stuff is over here. Proven practices. I like the concept and i can assure it is going to help us all. Just have a look Blades vs. traditional Rack servers. Meaningful discussion is going on here. And many more.

Found a good blog there http://virtuallycrazy.blogspot.com/ 

Good Redbook http://www.redbooks.ibm.com/abstracts/redp3953.html

And again good stuff here http://perspectives.mvdirona.com/2008/09/11/WhyBladeServersArentTheAnswerToAllQu estions.aspx

What else ? Thanks to VMWare again !!

Posted in Advance Concepts, Tips, VMWare | Leave a Comment »

What are Thin Disks,Thick Disks

Posted by Preetam on September 13, 2008

Excellent explanation of various storage technologies, especially Thin disks from vmmba.

 There are three main technologies that can accomplish storage oversubscription:

  1. Linked clones
    • This feature is available in VMware Lab Manager and VMware Workstation at the virtual disk level.  When a linked clone is used, the new VM uses pointers to the original VM for all common data.
    • The additional advantage of linked clones is that whitespace is not stored – for example if an empty data disk is part of a clone operation, the new disk will act as a "thin" disk and only consume the storage that it really requires for data
    • Linked clones can also be accomplished at the datastore level using technologies such as NetApp FlexClone (useful when cloning many VMs at once)
    • Keep in mind: linked clones pay a performance penalty on write operations (using copy-on-write), and put added stress on the source disks on read operations
  2. Thin Disks
    • Thin-provisioned disks are virtual disks that "appear" to the VM as one size, but only consume up to the amount of data that is required by that disk.  So, a 10 GB drive that is 50% utilized will only store 5 GB on disk (a traditional "thick" virtual disk would consume the entire 10 GB on disk)
    • Thin disks are options in VMware Workstation, and are the default disk type when using NFS storage in VMware ESX Server – however, VMs cloned from templates are always thick
    • Storage vendors such as Hitachi and NetApp have LUN-level thin provisioning, but that would only apply to VMware if using RDMs
  3. Deduplication
    • Deduplication is a technology similar to memory page sharing (above), where common data is stored only once.  It is done "after the fact" (ex poste), meaning de-duplication opportunities are scanned using a background process
    • Deduplication is primarily used for backups (e.g. Symantec PureDisk, EMC Avamar, or Quantum DXi-Series), but can also be used on the filesystem itself (today, using NetApp Deduplication, formerly A-SIS)

REFERENCE: www.vmmba.com

Posted in Advance Concepts, Storage, VM Provisioning, VMFS, VMWare | 1 Comment »

Musings from security guide –Part 02

Posted by Preetam on September 9, 2008

in ESX3i all logs are lost during reboot. Therefore a syslog server is required to record and archive all ESX Server 3i logs since a simple reboot will erase all activity of all users.

ISO Images consumes a lot of disk space since they are not compressed when they are created.

Users should create hash checksums on all ISO OS images on the ESX server before utilizing the ISO OS images for virtual machines.

Shares and Resource allocation: Minimum and Maximum resource settings within ESX Servers are absolute values, whereas shares are used to give preference to a guest OS when resource is scare. Minimum guarantee a specific amount of a resource to the virtual machine but deny that much of the resource to other virtual machine, While Maximum deny the virtual machine a portion of the resource while allowing other virtual machine more access to that resources. Do not configure the minimum virtual machine CPU and memory setting equal the total physical amount available. Use reservations,shares and limits to allocate resources.

Time management, synchronizing the virtual machine with the ESX Server is the preferred method for time synchronization.

Posted in Advance Concepts, Security, Virtual Center, VMWare | Leave a Comment »

Musings from security guide –Part 01

Posted by Preetam on September 9, 2008

  • The ESX server virtual switch port groups will be configured with any value between 2 and 4094. Utilizing VLAN1 will cause a denial of service since the ESX Server drops this traffic. The maximum port group that may be configured on a virtual switch is 512. Each port group is identified by a network label and a VLAN ID.

  • Ports Groups may have VLAN ID between  0 –4095.

  • VLAN ID 4095 specifies that the port group should use trunk mode or Virtual Guest Tagging (VGT) mode.

  • A value of Zero or blank VLAN ID is default value for External Switch Tagging(EST). EST is default configuration for all virtual switches within ESX Server. EST mode has 1-to-1 relationship, the number of VLAN’s are limited to the number of physical network adapter ports assigned to ESX.

  • Virtual Switch Tagging(VST) allows virtual switch to handle it’s own VLAN tagging. This processing is handled by Pnic and this overhead never comes to VMkernel. Each physical switch port that connects to virtual switch is configured in trunk mode. VLAN’s can span across multiple PSwitch. VLAN is enabled by trunked link connecting the virtual switch and PSwitch thru frame tags. Trunk links can carry the traffic of multiple VLANs simultaneously.Within Switch fabric, switches uses frame tagging to direct frames to the appropriate switch and port. Frame tagging assigns frame id prior to traversing trunked link. After the frame reaches the access link, VLAN ID is removed and the end device receives the frame.

  • Each Virtual Nic (VNic) has two MAC Address. effective and initial MAC address. Both the MAC address are same when they are first created.

  • Forged Transmits (set to accept by default): When effective MAC Address and initial MAC address are different, which means effective MAC address is always compared with initial MAC address.it is considered as forged transmits.

  • MAC Address Changes (set to accept by default): When effective MAC address is changed compared to initial one.

  • Promiscuous mode: When promiscuous mode is applied, all virtual machine connected to virtual switch have potential of reading all packets.

  • STP is not supported on vSwitch.Spanning Tree Protocol (STP) is either needs to disabled or Port fast needs to be enabled on PSwitch.

The vpxuser has privileges of a root user on the ESX server host,, but has no file privileges on the ESX server console. The vpxuser is created when the ESX server host is attached to Virtual Center. It is not present on the ESX Server host unless the host is being managed through VirtualCenter.

Virtual Center has two default roles defined, system roles and sample roles. System Roles are permanent and the permissions associated with these roles cannot be changed. All changes made to permissions of custom roles are effective immediately not requiring users to log off and log back in.

Posted in Advance Concepts, Security, Virtual Center, VMWare | Leave a Comment »

ESX,Storage and HBACMD’s -02

Posted by Preetam on September 1, 2008

[root@FirstESX hbanyware]# ./hbacmd PortStat 10:00:00:00:c9:4f:47:6f

Port Statistics for 10:00:00:00:c9:4f:47:6f

Secs Since Last Reset   :  700883
Exchange Count          :  5729859
Responder Exchange Count:  23
TX Seq Count            :  6842854
RX Seq Count            :  10893170
TX Frame Count          :  6845254
RX Frame Count          :  108920755
TX Word Count           :  223818240
RX Word Count           :  643851008
TX KB Count             :  874290
RX KB Count             :  2515043
LIP Count               :  1
NOS Count               :  N/A
Error Frame Count       :  0
Dumped Frame Count      :  N/A
Link Failure Count      :  0
Loss of Sync Count      :  46
Loss of Signal Count    :  1
Prim Seq Prot Err Count :  0
Invalid TX Word Count   :  235
Invalid RX Frame CRC Cnt:  0

******************************************************************************************************************************

[root@SecondESX hbanyware]# ./hbacmd ServerAttrib 10:00:00:00:c9:49:06:da

Server Attributes for 10:00:00:00:c9:49:06:da

Host Name       : FirstESX.vmzare.com
FW Resource Path: /usr/sbin/hbanyware/RMRepository/
DR Resource Path: /usr/sbin/hbanyware/RMRepository/
HBAnyware Server Version: 2.1a35

******************************************************************************************************************************

[root@SecondESX hbanyware]# ./hbacmd AllNodeInfo 10:00:00:00:c9:49:06:da

All Node Info for 10:00:00:00:c9:49:06:da

Node Type      : READY
FCP ID         : D2
SCSI Bus Number: 0
SCSI Target Num: 0
Node WWN       : 50:80:02:00:00:03:19:98
Port WWN       : 50:80:02:00:00:03:19:9B
OS Device Name : /proc/scsi/lpfc/00,0

Node Type      : READY
FCP ID         : B5
SCSI Bus Number: 0
SCSI Target Num: 3
Node WWN       : 50:80:02:00:00:03:19:98
Port WWN       : 50:80:02:00:00:03:19:9C
OS Device Name : /proc/scsi/lpfc/00,3

Node Type      : READY
FCP ID         : C6
SCSI Bus Number: 0
SCSI Target Num: 10
Node WWN       : 20:00:00:20:37:17:D4:8E
Port WWN       : 22:00:00:20:37:17:D4:8E
OS Device Name : /proc/scsi/lpfc/00,10

Node Type      : READY
FCP ID         : CD
SCSI Bus Number: 0
SCSI Target Num: 9
Node WWN       : 20:00:00:20:37:17:83:92
Port WWN       : 22:00:00:20:37:17:83:92
OS Device Name : /proc/scsi/lpfc/00,9

Node Type      : READY
FCP ID         : CC
SCSI Bus Number: 0
SCSI Target Num: 11
Node WWN       : 20:00:00:20:37:17:C5:14
Port WWN       : 22:00:00:20:37:17:C5:14
OS Device Name : /proc/scsi/lpfc/00,11

Node Type      : READY
FCP ID         : CB
SCSI Bus Number: 0
SCSI Target Num: 12
Node WWN       : 20:00:00:20:37:17:D0:E4
Port WWN       : 22:00:00:20:37:17:D0:E4
OS Device Name : /proc/scsi/lpfc/00,12

Node Type      : READY
FCP ID         : C9
SCSI Bus Number: 0
SCSI Target Num: 13
Node WWN       : 20:00:00:20:37:17:C6:11
Port WWN       : 22:00:00:20:37:17:C6:11
OS Device Name : /proc/scsi/lpfc/00,13

Node Type      : READY
FCP ID         : C7
SCSI Bus Number: 0
SCSI Target Num: 14
Node WWN       : 20:00:00:20:37:17:C2:3D
Port WWN       : 22:00:00:20:37:17:C2:3D
OS Device Name : /proc/scsi/lpfc/00,14

Node Type      : READY
FCP ID         : EF
SCSI Bus Number: 0
SCSI Target Num: 1
Node WWN       : 20:00:00:20:37:17:E5:F1
Port WWN       : 22:00:00:20:37:17:E5:F1
OS Device Name : /proc/scsi/lpfc/00,1

Node Type      : READY
FCP ID         : E8
SCSI Bus Number: 0
SCSI Target Num: 2
Node WWN       : 20:00:00:20:37:17:90:6F
Port WWN       : 22:00:00:20:37:17:90:6F
OS Device Name : /proc/scsi/lpfc/00,2

Node Type      : READY
FCP ID         : E4
SCSI Bus Number: 0
SCSI Target Num: 4
Node WWN       : 20:00:00:20:37:17:DA:7C
Port WWN       : 22:00:00:20:37:17:DA:7C
OS Device Name : /proc/scsi/lpfc/00,4

Node Type      : READY
FCP ID         : E2
SCSI Bus Number: 0
SCSI Target Num: 5
Node WWN       : 20:00:00:20:37:11:E8:CD
Port WWN       : 22:00:00:20:37:11:E8:CD
OS Device Name : /proc/scsi/lpfc/00,5

Node Type      : READY
FCP ID         : E1
SCSI Bus Number: 0
SCSI Target Num: 6
Node WWN       : 20:00:00:20:37:51:46:DD
Port WWN       : 22:00:00:20:37:51:46:DD
OS Device Name : /proc/scsi/lpfc/00,6

Node Type      : READY
FCP ID         : E0
SCSI Bus Number: 0
SCSI Target Num: 7
Node WWN       : 20:00:00:20:37:17:BB:A8
Port WWN       : 22:00:00:20:37:17:BB:A8
OS Device Name : /proc/scsi/lpfc/00,7

Node Type      : READY
FCP ID         : DC
SCSI Bus Number: 0
SCSI Target Num: 8
Node WWN       : 20:00:00:20:37:24:83:26
Port WWN       : 22:00:00:20:37:24:83:26
OS Device Name : /proc/scsi/lpfc/00,8

********************************************************************************************************

[root@FirstESX log]# cat /etc/logrotate.d/vmkernel
/var/log/vmkernel{
    create 0640 root patrol
    missingok
    nocompress         =======> change it to compress
   # keep a history over 3 years.
    monthly
    rotate 36
   #   max log size of 200k  (thus limiting total disk usage to under 8megs)
    size 200k          =======> change it to 2046
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

********************************************************************************************************
Check the nodes in a cluster

[root@FirstESX config]# cat vmware-sites
FULLTIME_SITES_TID 00000022
+ 1:8042,8042,8043 FirstESX    vmware #FT_Agent_Port=8045
+ 2:8042,8042,8043 SecondESX vmware

Posted in Advance Concepts, ESX-CMDs, HBACMD, Storage, VMWare | Leave a Comment »

ESX,Storage and HBACMD’s -01

Posted by Preetam on September 1, 2008

Found good article on how manage HBA and listed some command output

[root@FirstESX hbanyware]# ./hbacmd listhbas

Manageable HBA List

Port WWN   : 10:00:00:00:c9:4f:47:6f
Node WWN   : 20:00:00:00:c9:4f:47:6f
Fabric Name: 00:00:00:00:00:00:00:00
Flags      : 8000fa00
Host Name  : FirstESX.vmzare.com
Mfg        : Emulex Corporation

Port WWN   : 10:00:00:00:c9:49:06:da
Node WWN   : 20:00:00:00:c9:49:06:da
Fabric Name: 00:00:00:00:00:00:00:00
Flags      : 0000f902
Host Name  : FirstESX.vmzare.com
Mfg        : Emulex Corporation

******************************************************************************************************************************

[root@FirstESX hbanyware]# ./hbacmd HBAAttrib 10:00:00:00:c9:4f:47:6f

HBA Attributes for 10:00:00:00:c9:4f:47:6f

Host Name      : FirstESX.vmzare.com
Manufacturer   : Emulex Corporation
Serial Number  : MS53963943
Model          : LP10000
Model Desc     : Emulex LP10000 2Gb PCI-X Fibre Channel Adapter
Node WWN       : 20 00 00 00 c9 4f 47 6f
Node Symname   :
HW Version     : 1001206d
Opt ROM Version:
FW Version     : 1.90A4 (T2D1.90A4)
Vender Spec ID : 10DF
Number of Ports: 1
Driver Name    : lpfcdd_732
Device ID      : FA00
HBA Type       : LP10000
Operational FW : SLI-2 Overlay
SLI1 FW        : SLI-1 Overlay 1.90a4
SLI2 FW        : SLI-2 Overlay 1.90a4
IEEE Address   : 00 00 c9 4f 47 6f
Boot BIOS      : Boot Bios Firmware5.00a7
Driver Version : 7.3.2_vmw; HBAAPI(I) v2.0.f, 12-01-03

******************************************************************************************************************************

[root@FirstESX hbanyware]# ./hbacmd HBAAttrib 10:00:00:00:c9:49:06:da

HBA Attributes for 10:00:00:00:c9:49:06:da

Host Name      : FirstESX.vmzare.com
Manufacturer   : Emulex Corporation
Serial Number  : MS51195140
Model          : LP9002
Model Desc     : Emulex LP9002 2Gb PCI Fibre Channel Adapter
Node WWN       : 20 00 00 00 c9 49 06 da
Node Symname   :
HW Version     : 2002606d
Opt ROM Version:
FW Version     : 3.92A2 (C2D3.92A2)
Vender Spec ID : 10DF
Number of Ports: 1
Driver Name    : lpfcdd_732
Device ID      : F900
HBA Type       : LP9002
Operational FW : SLI-2 Overlay
SLI1 FW        : SLI-1 Overlay 3.92a2
SLI2 FW        : SLI-2 Overlay 3.92a2
IEEE Address   : 00 00 c9 49 06 da
Boot BIOS      : Disabled
Driver Version : 7.3.2_vmw; HBAAPI(I) v2.0.f, 12-01-03

******************************************************************************************************************************

[root@SecondESX hbanyware]# ./hbacmd PortAAttrib 10:00:00:00:c9:49:06:da

Port Attributes for 10:00:00:00:c9:49:06:da

Node WWN            : 20 00 00 00 c9 49 06 da
Port WWN            : 10 00 00 00 c9 49 06 da
Port Symname        :
Port FCID           : 0001
Port Type           : Private Loop
Port State          : Operational
Port Service Type   : 12
Port Supported FC4  : 00 00 01 20 00 00 00 01
                      00 00 00 00 00 00 00 00
                      00 00 00 00 00 00 00 00
                      00 00 00 00 00 00 00 00
Port Active FC4     : 00 00 01 00 00 00 00 01
                      00 00 00 00 00 00 00 00
                      00 00 00 00 00 00 00 00
                      00 00 00 00 00 00 00 00
Port Supported Speed: 2 GBit/sec.
Port Speed          : 1 GBit/sec.
Max Frame Size      : 2048
OS Device Name      : /proc/scsi/lpfc/0
Num Discovered Ports: 16
Fabric Name         : 00 00 00 00 00 00 00 00

Posted in Advance Concepts, ESX-CMDs, HBACMD, Storage, VMWare | Leave a Comment »

VMware: VI Toolkit (for Windows)

Posted by Preetam on August 18, 2008

Best website to start with powershell on vmware

VMware: VI Toolkit (for Windows)

and for quick start guide refer http://www.rtfm-ed.eu/docs/vmwdocs/whitepaper-powershell.pdf

Posted in Advance Concepts, powershell, VMWare | Leave a Comment »

ESX3.5

Posted by Preetam on August 4, 2008

Below paper provides good analysis of which Storage protocol to choose from.

storage protocol performance

see VMware knowledge base article 1003469 “Tuning ESX Server 3.5 for Better Storage Performance by Modifying the Maximum I/O Block Size”

Tuning ESX Server 3.5 for Better Storage Performance by Modifying the Maximum I/O Block Size

Posted in Advance Concepts, ESX3.5, Storage, VMWare, VMWARE_KBs | Leave a Comment »