EMC Symmetrix Architecture

This document will be using the EMC symmetrix configuration. There are a number of EMC Symmetrix configurations but they all use the same architecture as detailed below.

Front End Director Ports (SA-16b:1)
Front End Director (SA-16b)
Cache
Back End Director (DA-02b)
Back End Director Ports (DA-02b:c)
Disk Devices

Front End Director
A channel director (front end director) is a card that connects a host to the symmetrix, each card can have upto four ports.

Cache
Symmetrix cache memory buffers I/O transfers between the director channels and the storage devices. The cache is divided up into regions to eliminate contension.

Back End Director
A disk director (back end director) transfers data from disk to cache. Each back-end director can have upto four interfaces (C,D,E and F). Each back-end director interface can handle seven SCSI ids (0-6)

Disk Devices
The disk devices that are attached to the back-end directors could be either SCSI or FC-AL.

Interconnect
The direct matrix interconnect is a matrix of high speed connections to all componentswith bandwidth up to 64Gb/s

SAN Components

The are many components to a SAN Architecture. A host can connect to a SAN via direct connection or via a SAN switch.

Host HBA Host bus adaptor cards are used to access SAN storage systems
SAN Cables There are many types of cables and connectors:

Types: Multimode (<500m), single mode (>500m) and copper
Connectors: ST, SC (1Gb), LC (2Gb)

SAN Switches The primary function of a switch is to provide a physical connection and logical routing of data frames between the attached devices.

Support multiple protocols: Fibre channel, iSCSI, FCIP, iFCP
Type of switch: Workgroup, Directors

SAN Zoning Zoning is used to partition a fibre channel switched fabric into subsets of logical devices. Each zone contains a set of members that are permitted to access each other. Members are HBA’s, switch ports and SAN ports.

Types of zoning: hard, soft and mixed

Zone set s This is a group of zones that relate to one another, only one zone set can be active at any one time.
Storage arrays Storage array is were all the disk devices are located.
Volume access control This is also know as LUN masking. The storage array maintains a database that contains a map of the storage volumes and WWN’s that are allowed to access it. The VCM database in a symmetrix would contain the LUN masking information.

SAN Login

The below table documents the various proccesses that occur when a fibre channel device is connected to a SAN

Information/process FLOGI (fabric login) PLOGI (port login) PRLI (process login)
What is need ? – Link initialization
– Cable
– HBA and driver
– Switch Port
– FLOGI
– Zoning
– Persistent binding
– Driver setting
– PLOGI
– Device masking (target)
– Device mapping (initiator)
– Driver setting (initiator)
What information is passed – WWN
– S_ID
– Protocol
– Class
– Zoning
– WWN
– S_ID
– ULP
– Class
– BB Credit
– LUN
Who does the communication ? – N_port to F_port – N_port to N_port – ULP( scsi-3 to scsi-3)
where to find the information ? Unix
– syslog
– switch utilites

Windows
– Event viewer
– Switch viewer

Unix
– Syslog
– Driver Ulitities

Windows
– Driver utilities

Unix
– Syslog
– Host based volume management

Windows
– Driver Utilities
– Host based volume management
– Device Manager

If any one of the above were to fail then the host will not be allowed to access the disks on the SAN.

VCM Database

The Symmetrix Volume Configuration Management (VCM) database stores access configurations that are used to grant host access to logical devices in a Symmetrix storage array.

The VCM database resides on a special system resource logical device, referred to as the VCMDB device, on each Symmetrix storage array.

Information stored in the VCM database includes, but is not limited to:

  • Host and storage World Wide Names
  • SID Lock and Volume Visibility settings
  • Native logical device data, such as the front-end directors and storage ports to which they are mapped

Masking operations performed on Symmetrix storage devices result in modifications to the VCM database in the Symmetrix array. The VCM database can be backed up, restored, initialized and activated. The Symmetrix SDM Agent must be running in order to perform VCM database operations (except deleting backup files).

Switches

There are three models of switchs M-series (Mcdata), B-series (Brocade) and the MDS-series (Cisco). Each of the switch offer a web interface and a CLI. The following tasks can be set on most switches:

  • Configure network params
  • Configure fabric params (BB Credit, R_A_TOV, E_D_TOV, switch PID format, Domain ID)
  • Enable/Disable ports
  • Configure port speeds
  • Configure Zoning
BB Credit Configure the number of buffers that are available to attached devices for frame receipt default 16. Values range 1-16.
R_A_TOV Resource allocation time out value. This works with the E_D_TOV to determine switch actions when presented with an error condition
E_D_TOV Error detect time out value. This timer is used to flag potential error condition when an expected response is not received within the set time

Host HBA’s

The table below outlines which card will work with a particular O/S

Solaris Emulex PCI (lputil)
Qlogic
HPUX PCI-X gigabit fibre channel and ethernet card
AIX FC6227/6228/6239 using IBM native drivers
Windows Emulex (HBAnyware or lputilnt)
Linux Emulex PCI (lputil)

How do I extend the terminal width or set term variables with ksh? / Not Getting the full command info from ps in Unix terminal ?

To identify processes to kill we need to view the FULL output from the ps command (we use the comm field). But by default we are not able to view full command.  So here is the solution..

Solaris :

/usr/ucb/ps -awux

You may need two “w” options. From ps(1b):

-w Uses a wide output format (132 columns rather than 80). If the option letter is repeated, that is, -ww, uses arbitrarily wide output. This information is used to decide how much of long commands to print.

AIX / Linux :

I have a putty remote session or any terminal session with ksh shell. When I insert a long command I can see ‘<‘ suddenly appear on the line and my command begins to scroll horizontally between prompt and ‘<‘ sign while I keep on writing it.

I want to make it clear: a ‘<‘ sing does not appear at the end of the visible line (ie. limited by putty window) but in the middle of it. Is there a way to change it to higher value? I tried TERM, but did not set it.

Answer: You have to define $COLUMNS after you login or in .kshrc. Similarly, you can also define $LINES.

Open your .kshrc or edit .kshrc, and add

# echo $SHELL
/usr/bin/ksh
# export COLUMNS=1000
# ps -ef

 

How do I extend the terminal width or set term variables with ksh? / Not Getting the full command info from ps in Unix terminal ?

To identify processes to kill we need to view the FULL output from the ps command (we use the comm field). But by default we are not able to view full command.  So here is the solution..

Solaris :

/usr/ucb/ps -awux

You may need two “w” options. From ps(1b):

-w Uses a wide output format (132 columns rather than 80). If the option letter is repeated, that is, -ww, uses arbitrarily wide output. This information is used to decide how much of long commands to print.

AIX / Linux :

I have a putty remote session or any terminal session with ksh shell. When I insert a long command I can see ‘<‘ suddenly appear on the line and my command begins to scroll horizontally between prompt and ‘<‘ sign while I keep on writing it.

I want to make it clear: a ‘<‘ sing does not appear at the end of the visible line (ie. limited by putty window) but in the middle of it. Is there a way to change it to higher value? I tried TERM, but did not set it.

Answer: You have to define $COLUMNS after you login or in .kshrc. Similarly, you can also define $LINES.

Open your .kshrc or edit .kshrc, and add

# echo $SHELL
/usr/bin/ksh
# export COLUMNS=1000
# ps -ef

 

TSM Policy Settings

POLICY DOMAIN – This is a container for policy and scheduling info
*BACKRETention* is a fallback value for any files which
have been backed up under the specified policy domain,
but for which there is now a lack of an active policy
set.
*ARCHRETention* is a fallback value for any files which
have been archived under the specified policy domain,
but for which there is now a lack of an active policy
set.

POLICY SET – This is a set of management classes within a POLICY
DOMAIN.
Only the active version has any effect.  The active
version is created by issuing ACTIVATE POLICYSET
<set-name-you-made>.
After activating, you may edit the original set without
affecting the active copy.

MANAGEMENT CLASS – This is a classification for particular sets
of objects.  Minimal set to be bound is an entire
archive for archiving (ARCHMC option) or a single file,
directory,  filespace or everything for a backup
(INCLUDE option).
There can be many management classes per policy set,
but only ones in the active policy set may be
referenced
**The additional parameters for this structure are used
only for HSM migration.

BACKUP COPYGROUP – This is the structure which defines
retention of backup data.  These settings are the most
confusing because their operation overlaps and may
require some to be set to “NOLIMIT” to attain the
desired effect.

*VERSIONS DATA EXISTS* – This value specifies the maximum
number of versions of a file which may be retained.
If this is exceeded by additional backups or imports,
then the next expiration run on the server will remove
the oldest versions until there are only this many
left.
**This may be “NOLIMIT” or a whole number.

*RETAIN EXTRA VERSIONS* – This value specifies the maximum
number of days to retain an inactive copy when there is
an active copy in the database.
**This may be “NOLIMIT” or a whole number.

*RETAIN ONLY VERSION* – This value specifies the maximum
number of days to retain the only inactive version of
a file once there are no active versions.
**This may be “NOLIMIT” or a while number.
**This does not affect active copies which are always
retained.

*VERSIONS DATA DELETED* – This is the maximum number of
inactive versions of a file to retain once the file no
longer has an active version.


*DESTINATION* – This specifies the storage pool to which
newly backed up data will go if it is bound to this
management class during backup.
**Rebound data will not move to a new pool
automatically.
**Data may be moved from this pool through migration or
MOVE DATA commands.

Archive Copygroup – This is the structure which defines
retention of archive data.

*RETAIN VERSION* – This is the only retention parameter
for archives, and is the number of days that the entire
archive will be kept.

*DESTINATION* – This specifies the storage pool to which
newly archived data will go if it is bound to this
management class during archive operation.
**Rebound data will not move to a new pool
automatically.
**Data may be moved from this pool through migration or
MOVE DATA commands.

Schedule – This is a time/date/frequency setting for when commands
may be automatically run by the tsm server either on the
tsm server (administrative) or on the tsm client (client).

Client Association – a separate entity that binds clients to schedules.

Node – A separate entity which defines a client and allows access.

——-
CAVEATS
——-

NOTE: Once an archive is made, it may not be rebound.  In otherwords,
you may not specify a different management class for it. It IS possible
to change the copy-group retention period for this management class, and
then re-activate the policyset.  This will change it’s retention;
however, be cautious with this, as you will affect ANY data in the same
management class.

NOTE: Management classes with the same name but which are within a
different policy domain will NOT conflict with eachother.

NOTE: Changes to a policy set will NOT have ANY effect unless that
policy set is copied into the active policy set with ACTIVATE POLICYSET.
Only the active policyset, and not it’s source policyset or any other,
will have an effect.

NOTE: Active versions are converted to inactive only by the client,
through client-side expiration.  See the particular client code manual
for details. This may happen during a backup with an exclude list, or
during a backup with missing files, or through an API call to expire the
object.

NOTE: There are other parameters for all of these structures.  Please
see the ADSM/TSM Administrator’s Reference Guide for details.

——-
EXAMPLE
——-

Backup copygroup is set as follows:
VDE = 3
VDD = 1
RETEX = 100
RETO = 1

Explaination:
– Versions 1 through three are kept up to 100 days FROM WHEN THEY
WERE BACKED UP.
– Any versions past 3 will be expired immediately upon the next
server-side EXPIRE INVENTORY process.
– NOTE: NO MORE THAN THREE VERSIONS WILL BE KEPT AFTER EXPIRE
INVENTORY, REGARDLESS OF THE NUMBER OF DAYS SPECIFIED IN
RETAIN-EXTRA AND RETAIN-ONLY!
– If the active version is made inactive (client side expiration),
then only the most recent copy will be kept, and it will only
be kept for one day FROM WHEN IT WAS BACKED UP.

——-
EXAMPLE
——-

Backup copygroup is set as follows:
VDE = NOLIMIT
VDD = NOLIMIT
RETEX = 14
RETO = 20

Explaination:
– All versions will be kept for 14 days FROM WHEN THEY ARE BACKED UP.
– When no active version remains (client-side expiration), the last
inactive version will be kept for 20 days FROM WHEN BACKED UP.

——-
EXAMPLE
——-

Backup copygroup is set as follows:
VDE = 5
VDD = 0
RETEX = NOLIMIT
RETO = NOLIMIT

Explaination:
– 5 versions will be kept while an active copy exists.
– Files will not expire by age.
– When the active version is made inactive, the next EXPIRE INVENTORY
will remove ALL versions of the file.

Compaq: Troubleshooting

 

If you suspect that hardware problems are the reason for errors in your installation, analyze the system log. To do this, enter the command:
/usr/sbin/uerf -R | more

Solaris: Mounting a CD-ROM

In the Solaris operating system, a layer of software manages CD-ROM and other media.
This is the volume manager, which automates the interaction between you and your CDROMs.
Do not use this volume manager to mount CD-ROMs needed for the installation, as
the manager does not use the nomaplcase option. Without this option, some files on CDs can
not be found. Mount CD-ROMs for the installation manually as described in the following

Procedure

For more information on configuring the volume management, see the SunOS Routine
System Administration Guide.
Procedure
1. Log on as user root.

 

2. Check if vold (volume daemon) is running:
ps -ef | grep vold
3. Kill vold if it is running.
4. Insert the CD-ROM into your disk drive.
5. Create a mountpoint:
mkdir /sapcd
6. Mount the CD-ROM:
mount -F hsfs -o nomaplcase,ro /dev/dsk/c0t6d0s2 /sapcd
After the installation, you can start vold with the command /usr/sbin/vold.

 

Linux: How to Mounting a CD-ROM / DVD

1. Log on as user root.
2. Create a mount point for the CD-ROM with the command:
mkdir <CD-mountdir> (for example, <CD-mountdir> is /sapcd)
3. Mount the first CD-ROM device with the command:
mount -t iso9660 -r <device> <CD-mountdir>
where <device> is /dev/cdrom for non-SCSI CD-ROM devices and
/dev/scd<n> for SCSI drives with the device number <n>.
If the file names on the mounted CD-ROM are written in lowercase letters, remount the
CD-ROM with the following commands:
umount <device>
mount -t iso9660 -r -omap=off <device> <CD-mountdir>

 

>

Compaq Tru64 UNIX: Setting up Standard File Systems

The following table shows the variables and their corresponding values.


Variables in Standard File Systems

 
Variable Description
<diskname> Full path of a disk, for example, /dev/rz11c
<rawdiskname> Full path of a raw disk, for example, /dev/rrz11c
<disktype> Disk type, for example, rz28
<mountpointname> Full path of a mount point, for example,
SAP DB: /sapdb/<SAPSID>/sapdata1
Informix: Not applicable
Oracle: /oracle/<SAPSID>/sapdata1


To set up a standard file system:

1. Ask the customer which disks are available and make sure that they do not belong to
one of the categories listed in the following table:


Avoidable Disk Categories

Disk Category How to Check
Mounted disks grep <diskname> /etc/fstab
Swapdevices swapon -s | grep <diskname>
Advanced FS cd /etc/fdmns; showfdmn * | grep <diskname>
Volumes within LSM (Logical Storage manager) volprint -Ath | grep <diskname>


2. Create disk label, if necessary:
disklabel -w -r <rawdiskname> <disktype>
3. Create empty file system:
newfs -m 0 <rawdiskname>

4. Create mount point:
mkdir <mountpointname>
5. Add line to /etc/fstab:
<diskname> <mountpointname> ufs
6. Mount file system:
mount <mountpointname>