How to turn one JBOD into two JBOD’s

When putting together a storage lab for study, one of the important components is the disk arrays.  For the purpose of CCIE Storage Study, the arrays may seem to be rather generic, and from the MDS perspective they pretty much are.  You don’t need anything fancy.  You can use public or private loop for example.  I prefer public loop simply because its fabric aware, and TL ports are not supported on Gen 2 modules, so I want to future proof my investment, especially since public loop JBOD’s can be had for so little.

There are reasons however for getting a better JBOD, it may in fact save you money, even though it may seem to cost more.  For example, it’s no secret that one of my favorite JBOD’s is the Xyratex RS-1600-FC2.  These are basically what NetApp uses as their DS14mk2 disk shelves.  These are public loop and seemingly basic, however they have some nice options for lab study, particularly the ability to be paritioned. There are many other JBOD’s that support this type of partitioning functionality.  For example the Sun StorEdge A5200 (and most later models) support this functionality.  I prefer the Xyratex unit over the Sun however, since its more readily available, less heavy, less noisy, uses less power, and produces less heat.  This concept of splitting a JBOD can be applied to many different disk arrays however.

Xyratex RS-1600-FC2

The RS-1600-FC2 has the ability to hold 16 drives.  These can be configured as a single 16 drive loop, or as two 8 drive loops.  When your using two loops, they are really completely separate from the MDS’s perspective.  It effectively views these two separate loops as two JBOD’s each with 8 drives.  Because the RS-1600-FC2 has two LRC modules, you end up with two ports for each loop to plug into your MDS’s, just as you would if you had two physically separate JBOD’s.  There is much advantage to this.  When your building a SAN lab, power, space, cooling and noise can really become an issue.  It’s not the type of thing you easily build in your house.  The biggest culprits of these environmentals are in fact the disk shelves, followed by the MDS’s, and lastly the servers and other support equipment.  Being able to reduce the number of power circuits to half, and not introduce any additional power supplies and fans is a big plus.  Also you don’t have to use the redundant power on the JBOD’s, they will work just fine with single power supplies plugged in, you can silence any alarms using the configuration screen or jumpers.  The net result, is that instead of purchasing say four separate JBOD’s for study, you can get away with 2 and have pretty much the same result.
Here is a look at the way this particular JBOD is laid out:

Xyratex RS-1600-FC2 Layout

You can see that the JBOD has redundant LRC’s, redundant power supplies, and redundant fans.  Notice how the drives are laid out left to right.  The drives can be partitioned so that drives 0 through 7 can bee on one loop and drives 8 through 15 can be on a second loop.

Xyratex RS-1600-FC2 Loop Configurations

To the left are the two different loop configurations supported on this particular JBOD.  Configuring these is very simple, via the use of a thumbwheel and set of jumpers on the back of the unit.  Xyratex refers to the 1×16 drive setup as Mode 1, and the 2×8 drive setup as Mode 2.  A decent setup would be to populate the array with at least 12 drives, 6 in each loop.  I find 6 to be a very useful number of drives for a JBOD and studying storage.  You could populate all the drives, but each drive does use more power and generate more heat, so something to take into consideration.  With our setup of 12 drives, we would populate drive slots 0-5 and drive slots 8-13.  Each loop would be setup in the same way.  I prefer to use small drives and typically finding drives that are 7200 or 10000 RPM would be preferable to 15000 RPM, since usually the faster the drives the more heat and power that is used.  There is not much benefit in using larger drives if its just for lab.  You can get small good drives for probably less than $10 each on ebay.  An actual RS-1600-FC2 based shelf unit can sometimes be had for as little as $100-$150.

Xyratex RS-1600-FC2 Switch Settings

You can see that the switch settings clearly show its a breeze to partition the JBOD.  Simply set switch 1 to “Off” and the JBOD will use a split 2×8 configuration the next time it is powered on.  Don’t worry that this is not the factory recommended setting, after all why would Xyratex want you to get two for the price of one :).  There are other settings you may want to configure such as the speed or the drive addressing.   Realize that on most JBOD’s like the RS-1600-FC2, there is no auto-detection between 1Gb/s and 2Gb/s, you have to set this manually.  You have to make sure your GBIC’s at both end support the speed as well, which with 1-2GB this usually is not a problem.

So this is one step toward making your study lab more affordable and more easier to manage.  It is also possible to virtualize the servers.  I am not talking about using NPIV however, as that would not be as appropriate in a SAN lab, but rather multiple physical adapters in a beefy server running ESX for example.  You could say run 3-4 hosts on a box like this, and put 3 dual port HBA’s in it, and have literally 1 server and 2 JBOD’s doing the work of 4 servers and 4 JBOD’s.  Unfortunately there is no way to virtualize the MDS’s or third party switches such as the McData and Brocades you will need.

If you are using a JBOD that supports this type of partitioning please comment.  I can tell you that the JMR Fortra arrays I use do not.  But I have the Sun A5200 and Xyratex arrays which do.  Also to note is there is a difference between the Xyratex RS-1600-FC and Xyratex RS-1600-FC2.  You want the FC2.  There are also some other models which may be RAID, and what you want in a CCIE Storage Lab is JBOD mode, not RAID.  So do careful shopping, ask questions and make some smart purchases.  Good luck in building your lab!

This entry was posted in CCIE Storage, NetApp, Storage and tagged , , , , , . Bookmark the permalink.

29 Responses to How to turn one JBOD into two JBOD’s

  1. Julian says:

    Hey Brian, I’m trying to follow your guide here but I’m still confused on to what combination of SFP ports from the back of both controllers will attach to what JBOD loop. I got two of these off ebay and have been trying to turn them into 4 JBODs with their respective A and B connections as per ieMentor’s workbook, just having a hard time figuring out what physical connection represent which link going to the MDSes. I already turned the dip switch to off to split up the box in two, and six drives already show up as JBOD1A-B connecting fine to the MDS1-2 fc1/4, now when I try the same for JBOD2A-B fc1/5 and only B shows up connected in FM, and as I try combinations of SFP the behavior changes, I’ve been unable to keep all 4 links up, best I’ve been able to get is JBOD1A-B and JBOD2B.

    • brian says:

      You should be patching port A from LRC 1 and port A from LRC 2 for Loop #1, and patching port B from LRC 1 and port B from LRC 2 for Loop #2. The dip switch must be set as well. Can you tell me do you have the RS-1600-FC2? Have you downloaded the manual?

  2. shahid says:

    Hi Brian, just quick one, i have NetApp DS14mk2 with ESH2, It has two esh2 modules per shelf, how can i connect it as JBOD to MDS port 1/4 as per storage workbook?, the port comes up as TL but i dont find disks in flogi database nor Qlogic sansurf manager sees the disk. Both are in same vsan 10. i have 12 disk in shelf. From left to right, 1-6.

    any help to get it up and going would be great.

    cheers
    shahid

    • brian says:

      Do you have default zone permit turned on so that all initiators and targets can see eachother? You would either need that set within the VLAN or the disks properly zoned to the initiators

  3. shahid says:

    Yes, I have default zone permit. How can I connect MDS fc 1/4 to which port on DS14mk2? each ESH2 MODULE has two ports. I need help to connect MDS with Netapp DSKMK2 Please.

    cheers
    shahid

    • brian says:

      I don’t know too much about the DS14mk2, but I do know it can be used as a JBOD. I would start by connecting the fc1/4 directly to each esh port and try them one at a time, you should see drives show up with “sh flogi data”. Make sure the interface comes up, which should just be matching speed and type (TL or FL).

      Make sure the port is in the correct VSAN, the one which you did default zone permit for.

      Its possible the DS14mk2 has some jumpers or something like that which should be set.

  4. shahid says:

    It is in vsan 10 same as qlogic hba port fc1/1. Port only come up as TL, not FL.. Any other thoughts? I tried each port one by one on DS14MK2. MDS port fc1.4 comes up as TL.

    • brian says:

      do you see “anything” at all from the sh flogi? Do you see for example the controllers inside the DS14mk2? I would spend some time on NetApps site reading about the DS14mk2 and any particular jumpers, etc. The Xyratex site as well should have a break down. Make sure your not set to RAID mode, which may be the case, you need to be in a JBOD mode as far as the controllers are concerned.

  5. shahid says:

    Also, which console cable u used for brocade 2400? cisco condole cable doesnot show ant output. i need to config brocade to MDS as E port.
    Thanks

    • brian says:

      Its a null modem cable I believe or regular serial. I likely pinned out my own adapter. Best place for that info would be the brocade support forums, if you search there you will likely get all the info you need about making a console cable or which cable to use.

  6. shahid says:

    I dont see the controller in flogi database..

    • brian says:

      I wish I had some other suggestions for you. There could of course be something wrong with the disk shelf, but I would start with any documentation you can find on it, and look for any basic settings it allows for such as settings wheels or dip switches.

  7. shahid says:

    I have EMC CX-500 Array, can I convert it into JBOD?

    Thanks

  8. Albert says:

    Hey Brain,

    I am going to buy a Xyratex RS-1600-FCs. Is there a easy way to indentify it is FC or FC2. Is there a part number I can check on the box?

  9. Bart Flentje says:

    I have a large lot of Xyratex RS-1600-FC units selling for a fraction of what they’re listed for on eBay. Contact me through my website if you have an interest.

  10. Ashraf says:

    Hi,
    I’m interested to buy Xyratex RS-1600-FC2, give me price

  11. chris says:

    hello,

    i have a rs1602 with sata disks configured. i connected controller A to mds1 and controllerB to mds2 (seems i can just connect one port, if i connect two, mds not let the second port join). but my main problem is, and i have no idea anymore why/how, i see all disks as targets in the mds, i even see each disk in solaris and linux, but i cant use them, in solaris i get “type unavailable” and “condition” failed.

    im not sure if that is related to my switch config or related to the sata disks (with correct rs1602 sata drive caddys/converters). also (strange) if i direct connect the jbod to a server, i see (in nexenta) the jbod enclousre and healt…. but if i have the MDS between – it not shows up as jbod anymore (but the disk show up – as long as i use FC disks instead of the sata disks)

    any idea?

    • brian says:

      Chris,

      There could be alot of different things going on here. You need to make sure all jumpers on the rs1602 are set correctly. Second, what type of configuration do you have on the MDS? I assume you are not using zoning and just have default zoning enabled to allow all? You should make sure your SFP’s and cables are all good as well.

      What does the MDS show as far as the output of it trying to read FC4 targets? fcns? flog?

      I would take nexenta out of the picture until you have basics working between initiators (windows, etc) and the disks. You want to make sure you can read/write the disks, format them, etc. Sometimes, depending on what the disks were being used for before, you may need to wipe out the partition tables, etc on the disks and that can fix some things.

      • chris says:

        hi,

        thanks for your answer.

        well, we use default zoning (and as i mentioned, as long as i use the seagate FC disks, everything works perfect. i now tryed also to use a qlogic switch and i even direct connected the sbod – no success.

        its realy wired, if the sata disks are in, the switch see the correct vendor, name, disk size,.. but my linux and my solaris box see no disks as long as i use sata.

        you mentioned to use correct jumpers, well its extreme hard to find xyratex manuals – i found 3 different ones, each explain the jumpers on the sbod different 🙂 or are you talking about the controller jumpers?

        i tryed 2gb controllers, 4gb controllers, i have already about 6 different xyratex controllers 🙂
        the disks work (so they not show a failue (they show green light).

        i also tryed to get a 3.5 / 2.5 console cable to be able to check the controller, but i was not able to find one to buy. we was able to custom make our own cable to VIEW the console, but any input from my side is ignored,.. so seems our cable is not fully working.

        do you have any ideas? its realy wired.

        thanks
        regards
        chris

        • brian says:

          Yes you need to look at the jumper pins on the actual controller inside the shelf. DIP switches. This will set up the controller in different ways, there should be documentation on xyratex.

    • brian says:

      Chris,

      So let me get this straight, if you connect one port all is good, but two it does not work? How are you connecting the second port, i.e. what is your port configuration?

  12. chris says:

    .. oh one realy interresting additional note:

    if i take a netapp esh2-at contrller (14 disk enclosure!!), remove the right and the left screws, than put it in the rs1602 enclousre (yes, the 16 disk one 🙂 – the sata disks ARE showed to my solaris and to my linux server BUT that is a realy dirty hack and i just have one esh2 controller to test (and the controller have just 1x2gb FC, so realy to slow)

  13. chris says:

    Well, the jumper configurations i found in the manuals are just partialy described – and mostly just if the controlers do failover/agregation and speed related configuration notes. so i have not found anything about my issue..

  14. Sylvie says:

    Hi, I am setting up a RS-1600 with two non-RAID LRCs.

    I have a chance to get some 600 GB drives. But I was wondering if the LRC firmware will allow 16 x 600 GB, even if I break it up into 2 TB volumes?

    Could it be a problem?

  15. Ron says:

    I have two of the xyratex 1600-fc2
    one of them have two of the fc-al modules
    the other one has one fc-al moudle and the other has x24 raid controller
    does anyone have any input on how to loop these together and also the correct swtich settings. if so please send email bigierc617@……g.m.a.i.l.c.o.m

Leave a Reply