1. SAN (Storage)
To rescan all fiberchannel devices to be able to use them and to see the result, issue a "cfgadm -al" command:
To rescan all fiberchannel devices to be able to use them and to see the result, issue a "cfgadm -al" command:
# cfgadm -al
Ap_Id Type Receptacle Occupant Condition
c3 scsi-bus connected configured unknown
c3::dsk/c3t0d0 disk connected configured unknown
c4 fc-fabric connected configured unknown
c4::2100001b321fb79d unknown connected unconfigured unknown
c4::210000d0231c50ab disk connected configured unknown
c4::210000e08b0d82c6 unknown connected unconfigured unknown
c4::220000d0232c50be disk connected configured unknown
c5 fc-fabric connected configured unknown
c5::2100001b321f9eb3 unknown connected unconfigured unknown
c5::210000d0231c50be disk connected configured unknown
c5::220000d0232c50ab disk connected configured unknown
In this example we have two storage device with the following fiberchannel addresses:
Device 1:
WWNN: 200000d0230c50abWWPN: 210000d0231c50ab,220000d0232c50ab
Device 2:
WWNN: 200000d0230c50beWWPN: 210000d0231c50be,220000d0232c50be
"c4" in the cfgadm output represents Fiberchannel Interface Card 1, c5 Card 2. Each storage device is seen by c4 and c5 - but with different WWPNs - this seems to be a rendundant fiberchannel setup.
For disks to show up, just type:
# devfsadm -c disk
This command creates new /devices and /dev-entries if needed. The storage devices are now available to the Solaris operating system and they should appear when invoking the "format" command:
# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c3t0d0
/pci@7b,0/pci1022,7458@11/pci1000,3060@2/sd@0,0
1. c4t210000D0231C50ABd0
/pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0/disk@w210000d0231c50ab,0
2. c4t210000D0231C50ABd1
/pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0/disk@w210000d0231c50ab,1
3. c4t210000D0231C50ABd2
/pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0/disk@w210000d0231c50ab,2
4. c4t220000D0232C50BEd0
/pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0/disk@w220000d0232c50be,0
5. c4t220000D0232C50BEd1
/pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0/disk@w220000d0232c50be,1
6. c4t220000D0232C50BEd2
/pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0/disk@w220000d0232c50be,2
7. c5t210000D0231C50BEd0
/pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0/disk@w210000d0231c50be,0
8. c5t210000D0231C50BEd1
/pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0/disk@w210000d0231c50be,1
9. c5t210000D0231C50BEd2
/pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0/disk@w210000d0231c50be,2
10. c5t220000D0232C50ABd0
/pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0/disk@w220000d0232c50ab,0
11. c5t220000D0232C50ABd1
/pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0/disk@w220000d0232c50ab,1
12. c5t220000D0232C50ABd2
/pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0/disk@w220000d0232c50ab,2
First, look at the device names:
c5t22000D0232C50ABd0:
Controller 5: c5
Target 22000D0232C50AB (WWPN): t22000D0232C50AB
LUN 0: d0
The storage device is divided in multiple LUNs. SCSI allows to define different Logical Units (LUNs) available at the same target. RAID devices very often use this to diffentiate between created volumes on the storage device.
The target specification is just the WWPN of the device.
Another thing is apparent: The devices are appearing twice. Example: It is quite obvious that
5. c4t220000D0232C50BEd1
/pci@7b,0/pci10de,5d@d/pci1077,142@0/fp@0,0/disk@w220000d0232c50be,1
represents the same disk volume as
8. c5t210000D0231C50BEd1
/pci@7b,0/pci10de,5d@e/pci1077,142@0/fp@0,0/disk@w210000d0231c50be,1
They belong to the same WWNN, the same device. And the LUN is the same as well: d1. It's the same "disk".
So we need a software layer to correct this - otherwise this leads to chaos. Imagine you format disk 5, get data on it and you think disk 8 is available and you format this disk as well and mount it - the system will crash or immediately stops from using disk 5. The Operating System does not now that disk 8 is the same as 5 but at the point you begin to write on disk 8 it will notice that disk 5 has changed data without its knowledge - stop.
Now the scsi_vhci driver gets on his track.
First, we need to allow "fp" (the fiberchannel driver) to be able to use multipathing. Look at the configuration file
/kernel/drv/fp.conf :
Just make sure that MPxIO (Multipathing) is enabled. Normally you see that line in fp.conf:
mpxio-disable="yes";
If you see such a line, replace "yes" by "no":
mpxio-disable="no";
Next step: Be sure that scsi_vhci supports your device. Look at
http://southbrain.com/south/2008/02/solaris-10-with-mpxio-scsi-vhc.html (for Solaris 10) or
http://southbrain.com/south/2009/09/sun-mpxio-storage-scsi-vhci-so.html (for OpenSolaris)
In my case (using Solaris 10 and a non-supported non-Sun array), I had to write this in my
/kernel/drv/scsi_vhci.conf :
device-type-scsi-options-list =
"ADVUNI OXYGENRAID", "symmetric-option";
symmetric-option = 0x1000000;
In case you want to use an asymmetric link (only one link is active and the others are standby) with such a custom device (e.g. the device does not accept symmetric usage), just write this:
device-type-scsi-options-list =
"ADVUNI OXYGENRAID", "asymmetric-option";
asymmetric-option = 0x2000000;
Many devices are supported out of the box with scsi_vhci, so you won't need to configure it. Setting mpxio-disable="no"; in fp.conf as mentioned above is sufficient in this case.
Do a reconfiguration reboot:
# reboot -- -r
After boot, "format" will give you another output:
# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c3t0d0
/pci@7b,0/pci1022,7458@11/pci1000,3060@2/sd@0,0
1. c6t600D0230006B66680C50AB0F36ADF100d0
/scsi_vhci/disk@g600d0230006b66680c50ab0f36adf100
2. c6t600D0230006B66680C50AB4F92F61000d0
/scsi_vhci/disk@g600d0230006b66680c50ab4f92f61000
3. c6t600D0230006B66680C50AB5675F91300d0
/scsi_vhci/disk@g600d0230006b66680c50ab5675f91300
4. c6t600D0230006C1C4C0C50BE4DFE511B00d0
/scsi_vhci/disk@g600d0230006c1c4c0c50be4dfe511b00
5. c6t600D0230006C1C4C0C50BE16FF1FE200d0
/scsi_vhci/disk@g600d0230006c1c4c0c50be16ff1fe200
6. c6t600D0230006C1C4C0C50BE57396E9F00d0
/scsi_vhci/disk@g600d0230006c1c4c0c50be57396e9f00
The links are aggregated now. And - the controller number has changed!
MPxIO has replaced c4 and c5 by a virtual host controller interface (v h c i ...) c6! In fact c4 and c5 are hidden by the scsi_vhci driver.
Now it is safe to use these "disks" - despite of their long device name! You will love a terminal with copy & paste support to enter commands involving these disk devices.
To create a zpool on disk 6 above just type:
# zpool create mypool c6t600D0230006C1C4C0C50BE57396E9F00d0
Like you would do with a physical hardware scsi disk.
2. IP (Network)
To set up networking, just plumb the fcipX interfaces (X = fiberchannel port, X=0,1,2,3,....):
# ifconfig fcip0 plumb
Result:
# ifconfig fcip0
fcip0: flags=1000842
inet 0.0.0.0 netmask 0
ether 0:1b:32:1f:bd:9c
You're ready to assign an IP address:
# ifconfig fcip0 192.168.47.11 netmask 0xffffff00 broadcast 192.168.47.255 up
# ifconfig fcip0
fcip0: flags=1001843
inet 192.168.47.11 netmask ffffff00 broadcast 192.168.47.255
ether 0:1b:32:1f:bd:9c
Alias interfaces are also possible:
# ifconfig fcip0:1 plumb
# ifconfig fcip0:1 10.1.1.1 netmask 0xffffff00 broadcast 10.1.1.255 up
# ifconfig fcip0:1
fcip0:1: flags=1001843
inet 10.1.1.1 netmask ffffff00 broadcast 10.1.1.255
The high MTU value (65280) makes Fiberchannel IP very efficient to transfer high data rates (up to 8 Gbit/sec fiberchannel).
Be careful to use proper zoning when using fiberchannel storage across firewalled networks. Firewalls are easily circumvented with Fiberchannel IP when all storage is in one zone...
NETWORK MULTIPATHING IS NOT DONE VIA SCSI_VHCI! EVEN IF YOU USE MPXIO FOR SAN (AS ABOVE) YOU'LL HAVE FCIP0 AND FCIP1. USE NETWORK MULTIPATHING IF YOU PLAN TO LINK TWO FIBERCHANNEL IP LINKS TOGETHER.
No comments:
Post a Comment