NetApp cabling and hardware basics

NetApp cabling and hardware basics

In this post we will review some of the NetApp backend connectivity and hardware basics, like disk ownership, single path and multipath, partner cabling, etc. I decided to write this article since I’ve been working with NetApp for some time, but always managing them remotely, never had the chance to stand in front of one of these amazing storage boxes, and never found a NetApp document explaining this stuff, so learning and figuring this has been like a challenge to me and I think it would be nice to share my progress with other ones starting working with NetApp storage.

Let’s see what we are going to talk about:

  • Head FC ports
  • Shelves and modules
  • Disk ownership (hardware and software)
  • Cabling (single path and multipath, single nodes and clusters)

Head FC ports

Filers need FC ports in order to be able to connect to fabrics and shelves (there is a chance of having SAS shelves but we will only cover FC shelves on this post). This is the rear view of a FAS3020, as you can see, it comes with 4 FC ports (marked in orange):

image

These ports are named as 0a, 0b, 0c and 0d, and as said they can be used to connect shelves (configure them as initiators) or to connect to hosts (configure them as targets). You can define whether a FC port is  target or initiator using the fcadmin config command.

As you can see there are 4 expansion slots, here you can place new HBAs to provide the filer increased FC port availability, in this case the ports will be named ‘#X’, where ‘#’ stands for the number of slot where the HBA has been installed and ‘X’ stands for the port index which can be ‘a’ or ‘b’ since most of the HBAs are dual ports.

You can use sysconfig command in order to see how many ports your system has and what it is connected to them. What is connected under each port is known as loop or stack, for example, if under port 0a you have 3 shelves, it would be a loop or a stack composed by 3 shelves. Here it is an example from a NetApp simulator, in this case, as it is a simulator, different adapters will be named as v1, v2, v3, etc.:

image

Shelves and Modules

As any storage system besides having the controllers or heads, it has shelves where disks should be contained. NetApp has different types of disks, shelves and modules. Depending on the type of disk you select you will be able to use certain types of shelves and certain types of modules. Check the following diagram to understand the possible combinations (as we already said, we won’t cover SAS shelves, but you can take a look at http://www.netapp.com/mx/products/storage-systems/disk-shelves-and-storage-media/disk-shelves-tech-specs-la.html for further information):

image

Let’s see what each box means:

Disk type:

  • ATA Disks: well… you must be already used with these types of disks.
  • SATA Disks: Also for this one, already well known.
  • FC disks: Fiber channel disks.

To read more on disks types, specially about supported types and sizes, there is a very good NOW article on disk supported types: Available disk capacity by disk size (you might require registering to read the article).

Shelf Models

  • ds14mk2: This shelf accepts 14 FC or ATA disks, it takes 3 rack units and accepts ESH2 and AT-FCX disks.
  • ds14mk4: This shelf accepts 14 FC disks, it takes 3 rack units and only accepts ESH4 disks.

Modules

Modules allow different disk shelves to be connected to the storage FC loops.

  • ESH2: This module is used for FC disks and it is connected to a FC 2GB bus.
  • ESH4: This module is used for FC disks and it is connected to a FC 4GB bus.
  • AT-FCX: This module is used for ATA or SATA disks and it is connected to a FC 2GB bus.

This is a general guide so you can understand the concepts and differences between disks, shelf types and modules, for further information you can see the technical specs of you storage box. Here you can see the rear view of a disk shelf with only one module installed. Each shelf has space for 2 modules, module A (in the top slot) and module B (in the bottom slot, empty in the next picture).

image

As we said before, shelves are connected to a loop or stack, each module has an IN port and an OUT port which allow shelf being chained, and between the modules you will see a little green display which will allow you to set the number of shelf inside the loop. In the following picture you can see a single FAS3020 controller with 2 shelves in loop 0a:

image

This is a single path configuration, but we will talk on that later. The picture shows how rx (receive) and tx (transfer) fibers from 0a port on the filer head are connected to IN port in module A in shelf 1, the from module A in shelf one rx and tx fibers are run from OUT port to IN port in shelf 2 module A.

Disk ownership

Well, here we will make a quick stop and just say there are two types of ownership for disks. But first, let’s define ownership, disks are owned by a filer which is the one that manages the LUNs, shares, exports, snapshots and the rest of the operations over the volumes it hosts, in a cluster scenario the filers must know it’s partner’s disks and only take ownership of those resources in case of a takeover.

About the types of ownership, we have:

  • Software based: The less common one, the ownership of the disks in managed with the disk assign command, disks owned by a filer might be distributed across all shelves belonging to the cluster.
  • Hardware based: The most common one, the filer connected to the A module of the shelves owns the disks.

Cabling

NetApp supports lot’s of cabling configurations, let’s start reviewing from the simplest one, to the more complex ones. As said in the previous sections of this post there are other supported configurations, such as metro cluster, but here we will cover the most common ones:

  • Single node, single path
  • Single node, multipath
  • Cluster, single path
  • Cluster, multipath

Single node, single path

The most simple configuration, this is the less redundant solutions since it has many points of failures, such as a single controller, only 1 loop to each stack and only one module per shelf. In this configuration, rx (receive) and tx (transfer) fibers are connected from any of the FC ports on the controller to IN port in module A in the first shelf of the loop or stack, then from module A in that shelf rx and tx fibers are run from OUT port to IN port in shelf 2 module A, and so on for all the shelves in each stack.

image

Single node, multipath

This configuration is more secure than the previous one since it reduces all the single point of failures but one (there is still only one controller):

image

As you can see, in this case shelves have 2 modules each and there are two loops connected to the same stack of shelves (0a, solid line, and 0c, dotted line).

Why have  we used 0a for primary loop and 0c for secondary? well… there is no actual limitation, you can use any adapter you like, but NetApp recommends to use 0a as primary and 0c as secondary and same for 0b / 0d, even across nodes, will talk on this later.

Remember when we talked about disk ownership? If hardware disk ownership has been set (which is in 99% of the cases) 0a loop, connected to the A modules, will own the disks, in case of a faulty module or fiber cable, resources would start being accessed using 0c loop. If you run sysconfig as we have seen early on this post, you would see there are 6 shelves on the system, 3 attached to 0a and three attached to 0c,  using environment shelf command and storage show disk –p command you can identify which shelves are duplicated and which is the loop connected to A module and which to B modules.

For example if you run environment shelf command, you would obtain something like this for each shelf on the system, if you have 3 shelves in 2 loops then you would see this 6 times:

Channel: v0
Shelf: 1
SES device path: local access: v0.17
Module type: LRC; monitoring is active
Shelf status: normal condition
SES Configuration, via loop id 17 in shelf 1:
logical identifier=0x0b00000000000000
vendor identification=XYRATEX
product identification=DiskShelf14
product revision level=1111
Vendor-specific information:
Product Serial Number:
         Optional Settings: 0x00
Status reads attempted: 844; failed: 0
Control writes attempted: 3; failed: 0
Shelf bays with disk devices installed:
13, 12, 11, 10, 9, 8, 6, 5, 4, 3, 2, 1, 0
with error: none
Power Supply installed element list: 1, 2; with error: none
Power Supply information by element:
[1] Serial number: sim-PS12345-1  Part number: <N/A>
Type: <N/A>
Firmware version: <N/A>  Swaps: 0
[2] Serial number: sim-PS12345-2  Part number: <N/A>
Type: <N/A>
Firmware version: <N/A>  Swaps: 0
Cooling Unit installed element list: 1, 2; with error: none
Temperature Sensor installed element list: 1, 2, 3; with error: none
Shelf temperatures by element:
[1] 24 C (75 F) (ambient)  Normal temperature range
[2] 24 C (75 F)  Normal temperature range
[3] 24 C (75 F)  Normal temperature range
Temperature thresholds by element:
[1] High critical: 50 C (122 F); high warning 40 C (104 F)
Low critical:  0C (32 F); low warning 10 C (50 F)
[2] High critical: 63 C (145 F); high warning 53 C (127 F)
Low critical:  0C (32 F); low warning 10 C (50 F)
[3] High critical: 63 C (145 F); high warning 53 C (127 F)
Low critical:  0C (32 F); low warning 10 C (50 F)
ES Electronics installed element list: 1, 2; with error: none
ES Electronics reporting element: none
ES Electronics information by element:
[1] Serial number: sim-LS12345-1  Part number: <N/A>
CPLD version: <N/A>  Swaps: 0
[2] Serial number: sim-LS12345-2  Part number: <N/A>
CPLD version: <N/A>  Swaps: 0

The first record marked in bold is the serial number for the shelf, since this is an output obtained from a simulated filer the serial number here is missing, and the other two serial numbers marked in bold identifies A and B modules in that shelf respectively. So serial numbers will help to understand which shelves are connected to which loops, then storage show disk –p command will help you to identify which is the primary loop:

image

As you can see the 3 simulated shelves we saw in the sysconfig output are connected only to v0 adapter (since NetApp simulator does not emulate multipathing to shelves), and also you can see primary port is A.

Cluster, single path

Take a look at the following configuration:

image

we have 2 nodes interconnected by an infiniband cable (this interconnect cable is used for heartbeat and other cluster related operations and checks), and then we have 2 stacks, one with 3 shelves, and another one with 2 shelves. The first stack has all the A modules connected to 0a loop in controller 1 while B modules are connected to 0c modules in controller 2, as in a single node multipath configuration 0a/0c is used, but the difference now resides in 0a loop belongs to the owning filer and 0c to the partner, not 0a and 0c in the same node. And then we have the very same configuration for node 2, 0a loop is connected to A modules in the second stack and 0c adapter in controller 1 is used to connect the partner to this stack.

In this configuration there is no single point of failure, but there is still one down side, if an A module, or a fiber between primary loops fails a takeover of the resources would be executed by the partner. For example, if module A in shelf 1 (upper one) on 0a loop in controller 1 fails (I know it might sound confusing, read it twice if necessary, I had to hehehe), controller 1 would lost connectivity to the whole stack, in this case, controller 2 would have to takeover resources from controller 1 in order to be able to continue servicing storage. Unfortunately, the takeover process implies cifs service to be restarted, all cifs connections are dropped, so it can be really disruptive.

Cluster, multipath

Ok, now take a look at the following graph and go crazy!

 

image

Let’s do some writing to describe connections because it is really hard to follow the lines hehe:

Controller 1 (left) has:

  • 0a loop connected to A modules in stack 1 (3 shelves, left).
  • 0c adapter is connected to B modules in stack 2 (2 shelves, right).
  • 0b loop is used to provide a second patch for controller 1 to stack 1, it is connected to B module OUT port in last shelf on the loop, this way if A module fails in any of the shelves on this stack Controller 1 would still have access to the disks on this stack without need to failover resources over the partner.
  • 0d port is connected to shelf 2 in stack 2 module A OUT port, this way if resources have been failed from controller 2 to controller 1 you still don’t have a single point of failure.

Controller 2 (right) has:

  • 0a loop connected to A modules in stack 2 (2 shelves, right).
  • 0c adapter is connected to B modules in stack 1 (3 shelves, left).
  • 0b loop is used to provide a second patch for controller 2 to stack 2, it is connected to B module OUT port in last shelf on the loop, this way if A module fails in any of the shelves on this stack Controller 2 would still have access to the disks on this stack without need to failover resources over the partner.
  • 0d port is connected to shelf 3 in stack 1 module A OUT port, this way if resources have been failed from controller 1 to controller 2 you still don’t have a single point of failure.

As you might have already guessed this is the most redundant configuration (at least among the standard ones, never worked with metro cluster for example so can’t talk about it), then only down side of this configuration is you have to use 2 FC ports per head in order to provide access to a stack of shelves, this might become really expensive in FAS6240 environment which might have LOTS of shelves in different stacks..

Hope you liked the article and helped you to understand some of the basics, as always, any questions or comments are welcomed.

Updates and Contributions!

Luckily this article had lots of replies via LinkedIn groups and here, and I received some feedback which I think sharing is a must:

From Rajdeep Sengupta (Sr. Manager IT at Atrenta): He suggested to include SAS cabling information which I did not include in the post, He referenced to NetApp Unerversal SAS and ACP Cabling Guide document (part number: 215-05500_A0), it is a short but very good document with all you need, you can find it here (you might require registering at NOW to see the document):

  • http://now.netapp.com/NOW/knowledge/docs/hardware/filer/215-05500_A0.pdf

Thanks Rajdeep for the information!

From Chantal Callow (Technical Training Developer at NetApp): She made a very interesting comment, I talk across all the document about Clusters, this is because when I wrote the document I was thinking about Ontap 7, but as Chantal states, since Ontap 8 what was called Cluster in Ontap 7 is now called an ‘HA pair’. Thanks Chantal for your comment!

From Logan: He also mentions hardware disk ownership (which I stated was the most common ownership method) was deprecated after Ontap 8 leaving only software ownership. Thanks for your comment Logan, I should have specified that I was talking about Ontap 7. Any way I really appreciate your comment.

Thanks to all, readers and contributors who help this blog and its posts become richer day after day.

Hernán J. Larrea

62 Replies to “NetApp cabling and hardware basics”

  1. Hi Larrea,

    Just curious as I read your write up and real good stuff here. You mentioned that hard disk based ownership is more common vs software disk based ownership? Is that true? Moving forward , ONTAP 8.0 ONLY supports software disk based ownership, so I am confused. Please advise, thanks!

    Logan R

    1. Hi Logan, first of all thanks for your comment and sorry for answering you so late, I’ve been to the top of my eyes in the office. Well, your are totally right, I’ve oriented my post talking about Ontap 7 but haven’t detailed that on the post. I’ve received other comments from NetApp people saying Cluster in Ontap 8 is not called cluster anymore but HA. I’m going to edit the post including this Ontap 8 discrepancies and mention the contributors 🙂

      Thanks again Logan for passing by.

      Take care,
      Hernán

  2. Hi Hernan

    Great article. I didn’t quite completely get one cabling part from the multipathing.

    Controller2 :”0d port is connected to shelf 3 in stack 1 module B OUT port, this way if resources have been failed from controller 1 to controller 2 you still don’t have a single point of failure”.

    I assume module B OUTport is already cabled with 0b port from controller 1 . This is as per this line in your article:

    Controller1:”0b loop is used to provide a second patch for controller 1 to stack 1, it is connected to B module OUT port in last shelf on the loop”

    Sorry about my ignorance. Just wanted to check if I missed anything..

    Thanks in advance

    Regards
    Gaurab

    1. Hi Gaurab, you are totally right!! Is a typo, controller 2 0d port should be connected to shelf 3 in stack 1 but to module A’s OUT port.

      Thanks for noting that! I’ll correct the article.

      Take care!
      Hernán

  3. Hi Hernan,

    Was passing by and thought to ask you a question. With the cabling of “0b loop is used to provide a second patch for controller 1 to stack 1, it is connected to B module OUT port in last shelf on the loop, this way if A module fails in any of the shelves on this stack Controller 1 would still have access to the disks on this stack without need to failover resources over the partner”

    Consider this scenario, if A module of shelf 1 (stack 1) fails, would data access still be possible without a failover?Can you please explain ? I would assume the 0c module from stack 2 (controller 2) would see the shelf from B module .
    Regards
    Rahul

    1. Hi Rahul, thanks for passing by and sorry for taking so long to reply but I’ve been really busy.

      If A module of shelf 1 in stack 1 fails data would still be accessible without failingover to the other node. This is because as you mentioned, 0b loop from controller 1 is connected to Module B in the last shelf in the stack (to OUT port), and this module is connected via the IN port to the module B on shelf 2, and shelf 2 is connected to module B in sheld 3. Might be this is not so clear in the drawing but that is the idea.

      Hope this helps!
      Hernán

  4. 0b loop is used to provide a second patch for controller 1 to stack 1, it is connected to B module OUT port in last shelf on the loop, this way if A module fails in any of the shelves on this stack Controller 1 would still have access to the disks on this stack without need to failover resources over the partner.

    I have a scenerio, If A module of Shelf1 (controller 1) fails then would data access still be possible without a failover? I would assume the shelf wil be accessed from 0c port of stack 2(controller 2) from module B (shelf 1)

    Cheers
    Rahul

    1. Hi Rahul, actually you got your answer, in the first parragraff you wrote:

      0b loop is used to provide a second patch for controller 1 to stack 1, it is connected to B module OUT port in last shelf on the loop, this way if A module fails in any of the shelves on this stack Controller 1 would still have access to the disks on this stack without need to failover resources over the partner.

      That is right, data can stil be accessed even if a module fails. That is the idea in a Multipath configuration 🙂

      Cheers,
      Hernán

  5. This is a great post. Thank you very much. We had a FAS 3140 that we gave to a sister company for lab equipment since it was off warranty and we did not have room for it anymore.

    NetApp wired it up when we first got in 2008. They asked me to wire up the shelfs again….and the NetApp documentation is horrible for this.

    You saved me a ton of time!

  6. Hi,

    First Word: Awesum job !!! Secondly, we do see a lot of documentation if we google up. But its quite rare to get the basic documentation, since everything starts up with the “BASICS”. Again, great work !!!

    Cheers,
    Sudheer.

    1. Thanks a lot man for your comment!!! Glad you liked it 🙂

      If you have any suggestion or question, or you would like to see an article on an specific matter just let me know and we can work on it 🙂

      Take care and thanks again man!!
      Hernán

  7. HI Hernan

    Could you please elaborate the significance in connecting 0b to the out port of moduleB on shelf3 on controlllerA and controller B.

    BTW your document reeally rocks thanks a lott!!!!!:)

    1. Thanks for your comment Javed!! That provides a second path from the Controller A to that stack, so if module A fails, you don’t have to take over on Controller B, Controller A still can access the shelves using that path. In other words, if 0a look is broken at any point, shelves can still be accessed via 0b loopp without failover.

      Please let me know if you have any doubts or questions 🙂

      Thanks again for passing by and leaving your comment!
      Hernán

  8. Hi Hernan,

    in case we dont have physical view to the disk shelves , and we have a A module failure on one of the disk shelves then how would we then figure out using command line whether we have a dual path to the disk from the same controller well from sysconfig -a we will know whether it is multipath or single path and also storage show disk -p is another command. but i had a bad experience when one of the A module went bad . we had single path HA to the disk shelves and in that scenario the filer was not takenover by its partner. when the CE came to replace the module we gave a filer takeover command which accidentally caused the filer to panic.so my question is how do we figure out whether the module which failed was the primary path or secondary path

    Regards
    Javed

    1. Now that I think of… “environment show shelf” (I believe it was that way) will also help…. you should see the same shelves SNs connected through different loops.

      Hope this helps 🙂

      Regards,
      Hernán

      1. thanks for your prompt reply . I was just checking the output of storage show disk -pp. icould now figure out the primay and secondary disk path to the local disk shelves . but how can we figure out the partner node port connectivty to th local shelves. If we go by your in diagram 0c connectivity to to partner disk shelves

        1. If I don’t have physical access to the storage frame what I do is to make a diagram, check under the papers section of this blog the Ontap Simulator Cookbook 1.1, under the cluster section you will see a diagram. I always try to draw all the loops for each box (using sysconfig -a you can discover how many ports for disks you have and their id), and then with “environment show shelf” command I place each of the disks under the proper loop. Take a look at the diagraman in that document and you will see what I’m talking about 🙂

          Hope it helps, cheers,
          Hernán

  9. Hi Hernan,
    Great article man, awesome work.
    Got a question here, I want to connect two controllers (3210A) both in the same enclosure to the same disk shelf (DS14mk2) I want controller A to own the first 7 disks and controller B to own the rest, is this possible with software disk ownership?
    If yes, what is the cabling configuration for that.
    Thanks a lot man.

    1. Hi Bassem, thanks for your comment!! Well, if you are running Ontap 7, you will have to use software ownership. If you are running Ontap 8, software ownership is the only type so you can do it bydefault. About the cabling, connect controller 1 to module A, and controller 2 to module B.

      Check page 6 here: http://greg.porter.name/wordpress/wp-content/uploads/2011/02/Data-ONTAP-7G-Cook-Book-v3_1.pdf

      Also if you are running Ontap 7, you will have to go into maintenance mode in order to change the ownership type.

      Cheers,
      Hernán

  10. Hello Hernan,
    Great work…
    I have recently changed my job and would be working for NetApp storage support.
    I have been given autosupport mails from my new manager and he has asked me to draw connectivity diagram of those setup.
    Is it possible from autosupport mails.
    Pl help.

    Regards,
    Nikhil

    1. Hi Nikhil, first of thanks for passing by!

      Well, I don’t remeber exactly the content of the autosupport emails. For sure they contain the sysconfig -a output which is helpful, you can see how many shelves you have per loop. But I would suggest you get the output of the commands I mentioned on the post. Might be if you don’t have access to the posts you can request some one to get them for you.

      Take a look on the Papers section of this blog the cookbook for Ontap simulator 1.1, under the cluster appendix you will see a diagram which might be helpfull to you.

      Cheers,
      Hernán

  11. Have you ever thought about publishing an e-book or
    guest authoring on other sites? I have a blog centered on
    the same topics you discuss and would really like
    to have you share some stories/information. I know my readers would appreciate your work.
    If you’re even remotely interested, feel free to shoot me an e mail.

    1. Hi Maxwell, thanks for your comment! Honestly I would love to, but these days I’ve been short of time, in fact I have many ideas and things I want to write about, some of them are already written down it is I just didn’t have much time to review them between work and college, and don’t want to upload low quality content. But I like the idea, if you want send me an email with your blog’s address and what you have in mind and we can discuss. Here is my email address: hjlarrea@hotmail.com

      Thanks again for passing by!
      Hernán

  12. hi Hernan, how would this connect to a Fabric like a brocade switch? is there an order to the ports? I’m trying to connect a FAS3040 to a brocade switch, zone it with a Dell R610 running esxi 4.1 however while the ports and zoning look ok, the esxi doesn’t see the LUN. I’m wondering if I am connected to the wrong ports. I have 0a connected to the Brocade, port 0, and the esxi to port 3. I also have 2 quad port expansion cards which I used to connect to two DS14MK4 shelves. any help would be great! thanks.

    1. Hi Joe, thanks for passing by, seems this is the first node you connect to your NetApp box? If that is the case there are several things you need to check:
      1) That the port you are using (0a) is set as target and not as initiator, you can do that from the NetApp cli with the fcadmin command
      2) Verify that the port on the switch side where both hosts are connected are actually enabled, from the Brocade side you can use the portshow command
      3) Verify that both WWPNs (Storage’s port and the HBA’s WWPN on the server side) are visible from the fabric side, you can run the seach using the nscamshow or switchshow commands from the fabric side.

      I would start verifying those 3 points, probably you will find the answer while going through them :).

      Hope this helps, regards,
      Hernán

      1. Hi Hernan,

        Man, I though you got it with the netapp port as target and not as initiator, but was disappointed to find out that it not the case, take a look below, the output of the netapp and brocade, I have port 0a of the NetApp connected to the Brocade port 0, and I have port 3 of the Brocade connected to port 1 of a Qlogic HBA on a Dell R610. Very simple at this point, I just want to make sure this is working now, however I can’t get the VMware to see the new storage, I have a 720Gb LUN I’m exporting. All the commands are below, I’m just baffled at what could be causing this, Im startnig to suspect the VMware or something….. anything you can think of? thanks very much!

        Here is pretty much all the information…..

        FAS3040> fcadmin config

        Local
        Adapter Type State Status
        —————————————————
        0a target CONFIGURED online
        0b target CONFIGURED offline
        0c target CONFIGURED offline
        0d target CONFIGURED offline

        FAS3040> aggr status
        Aggr State Status Options
        aggr0 online raid_dp, aggr root
        aggr1 online raid_dp, aggr

        FAS3040> vol status
        Volume State Status Options
        vol0 online raid_dp, flex root
        vol1 online raid_dp, flex

        FAS3040> lun show -v
        /vol/vol1/vol1 718.1g (771008495616) (r/w, online, mapped)
        Comment: “NetApp Aggr1”
        Serial#: HnY6/opv2W5f
        Share: none
        Space Reservation: enabled
        Multiprotocol Type: vmware
        Maps: ESX7=102

        FAS3040> lun show -m
        LUN path Mapped to LUN ID Protocol
        ———————————————————————–
        /vol/vol1/vol1 ESX7 102 FCP

        FAS3040> igroup show
        ESX7 (FCP) (ostype: vmware):
        21:00:00:e0:8b:87:bf:be (not logged in)

        FAS3040> fcp show adapter
        Slot: 0c
        Description: Fibre Channel Target Adapter 0c (Dual-channel, QLogic 2432 (2462) rev. 2)
        Adapter Type: Local
        Status: OFFLINE
        FC Nodename: 50:0a:09:80:87:e9:71:67 (500a098087e97167)
        FC Portname: 50:0a:09:81:87:e9:71:67 (500a098187e97167)
        Standby: No

        Slot: 0d
        Description: Fibre Channel Target Adapter 0d (Dual-channel, QLogic 2432 (2462) rev. 2)
        Adapter Type: Local
        Status: OFFLINE
        FC Nodename: 50:0a:09:80:87:e9:71:67 (500a098087e97167)
        FC Portname: 50:0a:09:82:87:e9:71:67 (500a098287e97167)
        Standby: No

        Slot: 0a
        Description: Fibre Channel Target Adapter 0a (Dual-channel, QLogic 2432 (2462) rev. 2)
        Adapter Type: Local
        Status: LINK NOT CONNECTED
        FC Nodename: 50:0a:09:80:87:e9:71:67 (500a098087e97167)
        FC Portname: 50:0a:09:83:87:e9:71:67 (500a098387e97167)
        Standby: No

        Slot: 0b
        Description: Fibre Channel Target Adapter 0b (Dual-channel, QLogic 2432 (2462) rev. 2)
        Adapter Type: Local
        Status: OFFLINE
        FC Nodename: 50:0a:09:80:87:e9:71:67 (500a098087e97167)
        FC Portname: 50:0a:09:84:87:e9:71:67 (500a098487e97167)
        Standby: No

        DS_4100B:admin> portshow 0
        portIndex: 0
        portName: NetAPP e0
        portHealth: No Fabric Watch License

        Authentication: None
        portDisableReason: None
        portCFlags: 0x1
        portFlags: 0x20b03 PRESENT ACTIVE F_PORT G_PORT LOGICAL_ONLINE LOGIN NOELP ACCEPT FLOGI
        LocalSwcFlags: 0x0
        portType: 10.0
        POD Port: Port is licensed
        portState: 1 Online
        Protocol: FC
        portPhys: 6 In_Sync portScn: 32 F_Port
        port generation number: 0
        state transition count: 1

        portId: 010000
        portIfId: 43020001
        portWwn: 20:00:00:05:1e:04:ed:43
        portWwn of device(s) connected:
        21:00:00:1b:32:0e:a6:92
        Distance: normal
        portSpeed: N4Gbps

        LE domain: 0
        FC Fastwrite: OFF
        Interrupts: 12 Link_failure: 0 Frjt: 0
        Unknown: 0 Loss_of_sync: 2 Fbsy: 0
        Lli: 12 Loss_of_sig: 4
        Proc_rqrd: 445 Protocol_err: 0
        Timed_out: 0 Invalid_word: 0
        Rx_flushed: 0 Invalid_crc: 0
        Tx_unavail: 0 Delim_err: 0
        Free_buffer: 0 Address_err: 0
        Overrun: 0 Lr_in: 2
        Suspended: 0 Lr_out: 2
        Parity_err: 0 Ols_in: 2
        2_parity_err: 0 Ols_out: 2
        CMI_bus_err: 0

        Port part of other ADs: No

        DS_4100B:admin> portshow 3
        portIndex: 3
        portName:
        portHealth: No Fabric Watch License

        Authentication: None
        portDisableReason: None
        portCFlags: 0x1
        portFlags: 0x20b03 PRESENT ACTIVE F_PORT G_PORT LOGICAL_ONLINE LOGIN NOELP ACCEPT FLOGI
        LocalSwcFlags: 0x0
        portType: 10.0
        POD Port: Port is licensed
        portState: 1 Online
        Protocol: FC
        portPhys: 6 In_Sync portScn: 32 F_Port
        port generation number: 0
        state transition count: 1

        portId: 010300
        portIfId: 43020007
        portWwn: 20:03:00:05:1e:04:ed:43
        portWwn of device(s) connected:
        21:00:00:e0:8b:87:bf:be
        Distance: normal
        portSpeed: N4Gbps

        LE domain: 0
        FC Fastwrite: OFF
        Interrupts: 11 Link_failure: 0 Frjt: 0
        Unknown: 0 Loss_of_sync: 2 Fbsy: 0
        Lli: 11 Loss_of_sig: 4
        Proc_rqrd: 32 Protocol_err: 0
        Timed_out: 0 Invalid_word: 0
        Rx_flushed: 0 Invalid_crc: 0
        Tx_unavail: 0 Delim_err: 0
        Free_buffer: 0 Address_err: 0
        Overrun: 0 Lr_in: 2
        Suspended: 0 Lr_out: 1
        Parity_err: 0 Ols_in: 1
        2_parity_err: 0 Ols_out: 2
        CMI_bus_err: 0

        Port part of other ADs: No

        DS_4100B:admin> cfgshow
        Defined configuration:
        cfg: LA_CLOUD1
        LA_CLOUD
        zone: LA_CLOUD
        Dell_esx7_port_1; NetApp_Port_e0
        alias: Dell_esx7_port_1
        21:00:00:e0:8b:87:bf:be
        alias: NetApp_Port_e0
        21:00:00:1b:32:0e:a6:92

        Effective configuration:
        cfg: LA_CLOUD1
        zone: LA_CLOUD
        21:00:00:e0:8b:87:bf:be
        21:00:00:1b:32:0e:a6:92

        1. Check this out:

          Slot: 0a
          Description: Fibre Channel Target Adapter 0a (Dual-channel, QLogic 2432 (2462) rev. 2)
          Adapter Type: Local
          Status: LINK NOT CONNECTED
          FC Nodename: 50:0a:09:80:87:e9:71:67 (500a098087e97167)
          FC Portname: 50:0a:09:83:87:e9:71:67 (500a098387e97167)
          Standby: No

          Status: LINK NOT CONNECTED

          Seems there is some issue with the conecction between 0a and the switch. Can you do a portshow to the port where the 0a is connected?

          1. Hernan, Port 0a on the NetApp is connected to port 0 of the Brocade. That portshow is:

            DS_4100B:admin> portshow 0
            portIndex: 0
            portName: NetAPP e0
            portHealth: No Fabric Watch License

            Authentication: None
            portDisableReason: None
            portCFlags: 0x1
            portFlags: 0x20b03 PRESENT ACTIVE F_PORT G_PORT LOGICAL_ONLINE LOGIN NOELP ACCEPT FLOGI
            LocalSwcFlags: 0x0
            portType: 10.0
            POD Port: Port is licensed
            portState: 1 Online
            Protocol: FC
            portPhys: 6 In_Sync portScn: 32 F_Port
            port generation number: 0
            state transition count: 1

            portId: 010000
            portIfId: 43020001
            portWwn: 20:00:00:05:1e:04:ed:43
            portWwn of device(s) connected:
            21:00:00:1b:32:0e:a6:92
            Distance: normal
            portSpeed: N4Gbps

            LE domain: 0
            FC Fastwrite: OFF
            Interrupts: 12 Link_failure: 0 Frjt: 0
            Unknown: 0 Loss_of_sync: 2 Fbsy: 0
            Lli: 12 Loss_of_sig: 4
            Proc_rqrd: 445 Protocol_err: 0
            Timed_out: 0 Invalid_word: 0
            Rx_flushed: 0 Invalid_crc: 0
            Tx_unavail: 0 Delim_err: 0
            Free_buffer: 0 Address_err: 0
            Overrun: 0 Lr_in: 2
            Suspended: 0 Lr_out: 2
            Parity_err: 0 Ols_in: 2
            2_parity_err: 0 Ols_out: 2
            CMI_bus_err: 0

            Port part of other ADs: No

          2. Hey Joe, check this out, on the portshow for port 0 you see:

            portWwn of device(s) connected:
            21:00:00:1b:32:0e:a6:92

            But the WWPN of the 0a port in the NetApp seems to be different:

            Slot: 0a
            Description: Fibre Channel Target Adapter 0a (Dual-channel, QLogic 2432 (2462) rev. 2)
            Adapter Type: Local
            Status: LINK NOT CONNECTED
            FC Nodename: 50:0a:09:80:87:e9:71:67 (500a098087e97167)
            FC Portname: 50:0a:09:83:87:e9:71:67 (500a098387e97167)
            Standby: No

            Are you sure the cabling is correct? How many switches you’ve got on the fabric?

  13. Hernan –

    There is one Broacde right now, 16 active ports, of which only port 0 and 3 are connected. There is one
    Fiber going from port 0 of the Brocade to Port 0a of the NetApp.

    There is a second cable going from the Dell R610 to port 3 of the Brocade.

    The WWN of the NetApp is: 50:0a:09:83:87:e9:71:67
    The WWN of the HBA on the R610 is: 21:00:00:e0:8b:87:bf:be

    There are 2 4-port HBA’s in the NetApp, I’m thinking that possibly I have 0a connected to the shelf and not to the Brocade, and one of the ports on the 4-port HBA’s is connected to the Brocade. if this is the case, then possibly the port is configured as initiator and not target…. do you know how to change that on the netapp, is it possible?

    Here is the adapter settings from sysconfig -a

    slot 2: FC Host Adapter 2a (QLogic 2432 rev. 3, N-port, )
    Board name: QLE2464
    Serial Number: GFC0806J55046
    Firmware rev: 4.5.2
    Host Port Id: 0x10000
    FC Node Name: 2:100:001b32:0ea692
    SFF Vendor: FINISAR CORP.
    SFF Part Number: FTLF8524E2KNL
    SFF Serial Number: PD41CR7
    SFF Capabilities: 1, 2 or 4 Gbit
    Link Data Rate: 4 Gbit
    Switch Port: DS_4100B:0
    I/O base 0x0000000000000000, size 0x0
    memory mapped I/O base 0x00000000c0200000, size 0x4000
    slot 2: FC Host Adapter 2b (QLogic 2432 rev. 3, L-port, )
    Board name: QLE2464
    Serial Number: GFC0806J55046
    Firmware rev: 4.5.2
    Host Loop Id: 0
    FC Node Name: 2:101:001b32:2ea692
    SFF Vendor: FINISAR CORP.
    SFF Part Number: FTLF8524E2KNL
    SFF Serial Number: PD4228D
    SFF Capabilities: 1, 2 or 4 Gbit
    Link Data Rate: 1 Gbit
    I/O base 0x0000000000000000, size 0x0
    memory mapped I/O base 0x00000000c0204000, size 0x4000
    slot 2: FC Host Adapter 2c (QLogic 2432 rev. 3, L-port, )
    Board name: QLE2464
    Serial Number: GFC0806J55046
    Firmware rev: 4.5.2
    Host Loop Id: 0
    FC Node Name: 2:102:001b32:4ea692
    SFF Vendor: FINISAR CORP.
    SFF Part Number: FTLF8524E2KNL
    SFF Serial Number: PD43M7E
    SFF Capabilities: 1, 2 or 4 Gbit
    Link Data Rate: 1 Gbit
    I/O base 0x0000000000000000, size 0x0
    memory mapped I/O base 0x00000000c0400000, size 0x4000
    slot 2: FC Host Adapter 2d (QLogic 2432 rev. 3, L-port, )
    Board name: QLE2464
    Serial Number: GFC0806J55046
    Firmware rev: 4.5.2
    Host Loop Id: 7
    FC Node Name: 2:103:001b32:6ea692
    SFF Vendor: FINISAR CORP.
    SFF Part Number: FTLF8524E2KNL
    SFF Serial Number: PD43TE5
    SFF Capabilities: 1, 2 or 4 Gbit
    Link Data Rate: 2 Gbit
    16 : NETAPP X276_HPYTA288F10 NA04 272.0GB 520B/sect (VAX6KALA)
    17 : NETAPP X276_FAL9E288F10 NA05 272.0GB 520B/sect (DH07P8208K5M)
    18 : NETAPP X276_FAL9E288F10 NA05 272.0GB 520B/sect (DH07P8208K4P)
    19 : NETAPP X276_FAL9E288F10 NA05 272.0GB 520B/sect (DH07P8208K4W)
    20 : NETAPP X276_FAL9E288F10 NA05 272.0GB 520B/sect (DH07P8208JV2)
    21 : NETAPP X276_FAL9E288F10 NA05 272.0GB 520B/sect (DH07P8208K36)
    22 : NETAPP X276_FAL9E288F10 NA05 272.0GB 520B/sect (DH07P8208K29)
    23 : NETAPP X279_HVPBP288F15 NA02 272.0GB 520B/sect (JLWDA3NC)
    24 : NETAPP X279_S15K6288F15 NA02 272.0GB 520B/sect (3QP2BJTK00009005K1RU)
    25 : NETAPP X279_S15K6288F15 NA02 272.0GB 520B/sect (3QP2BGWQ000090058DXE)
    26 : NETAPP X279_S15K6288F15 NA02 272.0GB 520B/sect (3QP2BWBW00009005R967)
    27 : NETAPP X279_S15K6288F15 NA02 272.0GB 520B/sect (3QP2BHX000009005RARU)
    28 : NETAPP X279_S15K6288F15 NA02 272.0GB 520B/sect (3QP289JQ00009005N0CG)
    29 : NETAPP X279_HVPBP288F15 NA02 272.0GB 520B/sect (JLWDA2ZC)
    Shelf 1: ESH4 Firmware rev. ESH A: 14 ESH B: 14
    I/O base 0x0000000000000000, size 0x0
    memory mapped I/O base 0x00000000c0404000, size 0x4000
    slot 4: FC Host Adapter 4a (QLogic 2432 rev. 3, L-port, )
    Board name: QLE2464
    Serial Number: GFC0806J55028
    Firmware rev: 4.5.2
    Host Loop Id: 0
    FC Node Name: 2:100:001b32:0e9492
    SFF Vendor: FINISAR CORP.
    SFF Part Number: FTLF8524E2KNL
    SFF Serial Number: PD43UNV
    SFF Capabilities: 1, 2 or 4 Gbit
    Link Data Rate: 1 Gbit
    I/O base 0x0000000000000000, size 0x0
    memory mapped I/O base 0x00000000c1600000, size 0x4000
    slot 4: FC Host Adapter 4b (QLogic 2432 rev. 3, L-port, )
    Board name: QLE2464
    Serial Number: GFC0806J55028
    Firmware rev: 4.5.2
    Host Loop Id: 0
    FC Node Name: 2:101:001b32:2e9492
    SFF Vendor: FINISAR CORP.
    SFF Part Number: FTLF8524E2KNL
    SFF Serial Number: PD43TA4
    SFF Capabilities: 1, 2 or 4 Gbit
    Link Data Rate: 1 Gbit
    I/O base 0x0000000000000000, size 0x0
    memory mapped I/O base 0x00000000c1604000, size 0x4000
    slot 4: FC Host Adapter 4c (QLogic 2432 rev. 3, L-port, )
    Board name: QLE2464
    Serial Number: GFC0806J55028
    Firmware rev: 4.5.2
    Host Loop Id: 0
    FC Node Name: 2:102:001b32:4e9492
    SFF Vendor: FINISAR CORP.
    SFF Part Number: FTLF8524E2KNL
    SFF Serial Number: PD43T7G
    SFF Capabilities: 1, 2 or 4 Gbit
    Link Data Rate: 1 Gbit
    I/O base 0x0000000000000000, size 0x0
    memory mapped I/O base 0x00000000c1800000, size 0x4000
    slot 4: FC Host Adapter 4d (QLogic 2432 rev. 3, L-port, )
    Board name: QLE2464
    Serial Number: GFC0806J55028
    Firmware rev: 4.5.2
    Host Loop Id: 7
    FC Node Name: 2:103:001b32:6e9492
    SFF Vendor: FINISAR CORP.
    SFF Part Number: FTLF8524E2KNL
    SFF Serial Number: PD43T89
    SFF Capabilities: 1, 2 or 4 Gbit
    Link Data Rate: 4 Gbit

  14. Hi Hernan,

    I have a simple question, I am in cluster -single path mode.
    To make my system full HA do I just need to do the cabling as you described for multipath -cluster(HA) mode.
    Or do we need to run any set of commands as well, do complete the cluster-multipath configuration?????

    In other words, is the cabling itself is sufficient to achive multipath. Or do we need enable, disable, modify or set some options too??

    Regards,
    Nishant Kohli
    nishantkohli87@gmail.com

      1. Thanks Hernan, that helped. In my case I will also have to install new set of fc adapters on both controllers, which means a new slots. I guess I will take proper downtime and powerdown both the controllers before doing so.
        Also since these are new adapters, I will need to do fcadmin config to set the new ports as initiators.

        right?

  15. This is good article.
    I have a question hope you can explain this.
    I has 3 shelf stacked and a fas3210 with dual controller in HA.
    .> STORAGE SHOW DISK -P

    PRIMARY PORT SECONDARY PORT SHELF BAY

    0A.00.0 A 0B.00.0 B 0 0

    what is primary and what is secondary?

  16. Hi,

    Thank you for this great article.

    I was wondering if you can make a image of a setup in Multipath modes with 3/3 shelves. I’m not 100% sure how this would end up in cabeling.

    My other question is if it’s possible to a multipath setup with SAS and SATA shelves. As it’s know you cannot mix SAS and SATA in the same loop, so would this limit you to a single path setup per type of shelf ?

    Thanks again!

    1. Hi Matt, thanks for passing by and sorry for the late response, I’m really short on time lately, so I’ll have to own you the graphic, but it should differ too much from the ones included here, just add a shelf in the middle.

      About SAS and SATA, yes, do not mix them, and for max per loop, I have seen most configurations to hold around 3 shelves per loop. You can read the box specs to check the max amount of shelves per loop.

      Regards,
      Hernán

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.