How Do I Stack S-Series Units Together Using SFTOS?
 

Overview

The S-Series supports stacking through rear stack ports that enable you to interconnect several units to form a managed, single-IP address system. Stacking is essentially a method to manage a group of switches without the need to assign an IP address to every switch. This document uses the term "stack" to describe the set of switches that make up the interconnected system.

Best Practices

  1. Pre-configure unit numbers for each unit in the stack, prior to adding them to a stack. In SFTOS, use the switch renumber command.

    If you connect a previously configured unit with a saved stack number that does not match the unit number assigned with the switch renumber command, SFTOS will assign a default configuration to the unit. If this occurs, use the switch renumber command to assign a stack number that matches the unit number that you pre-configured.

  2. To control management unit selection, use the switch priority command.

    If the stack management unit were to be removed or become non-functional for some reason, one of the other switches in the stack will be elected manager. It is possible to set the switch that will become the replacement management unit by configuring it to have the highest switch priority. Note that setting this priority in an operating stack will not cause an immediate change to a new management unit. This only occurs when a new manager needs to be elected.

    Make sure that the switch that you want to be the manager of the stack is the manager after the stack boots. A switch that is the manager of the stack will remain the manager of the stack after a stack reboot. So, if necessary, use the movemanagement from-unit to-unit command to force the desired unit to become the manager. This will ensure that it will remain manager if, for example, there is a power outage and the whole stack reboots.

  3. Make sure each unit has the same software version prior to connecting them together.

    Loading a new SFTOS image onto the management unit automatically causes that image to be propagated to all other stack member units (all must run the same SFTOS version.)

  4. Make sure you save the configuration after making changes. The configuration will be saved to all units in the stack. If you do not save changes, and the management unit were to fail, the changes would be lost; the other units in the stack will not learn about, and store, the changes unless the changes are saved.

    Saved configuration changes are propagated to and saved in NVRAM of member units. The propagated configuration includes the management IP address. Therefore, take care to ensure that you do not create duplicate addresses on the network by simply removing the stacking cables from two units connected to the same network.

  5. Connect a new unit to an existing stack prior to powering it up. Failure to do so will not result in physical damage, but a new stack member that was previously configured as a management unit could end up as the management unit of its new stack, disrupting expected stack operation.

These practices will minimize disruption to the stack (and network) when connecting units and during failures.

Note: It is possible, and recommended, to pre-configure the stack for new units. Use the member command on the management unit to accomplish this function. Tthis will logically create the new unit and all the ports connected to that unit. You can define the new unit in advance, and pre-configure the ports even though the ports do not yet physically exist in the stack. If you do so, make sure you pre-assign the unit number to the new unit so that it will get the proper configuration from the management unit when connected to the stack. See Best Practices, above. It is worth noting that pre-configured ports that are not physically present will have a state of "detached".

Number of Units in a stack: Force 10 does not recommend stacking more than three units when running SFTOS, although the software will allow you to stack up to seven.

Step by Step Stacking Procedure

Step 1: Determine the Stacking Role of each Switch.

Switches within a stack have one of two roles:

  • Management Unit – provides the management interface for the entire stack. Each management request goes to the management unit before redirection to the appropriate member unit in the stack.

    S-Series switches have a failover algorithm that automatically selects a new management unit if the current management unit goes off-line or is removed from the stack. You can influence that algorithm with the switch priority command in order to designate a particular unit as the standby management unit. However, unlike FTOS, SFTOS does not specifically identify a stack member as the standby management unit, so SFTOS must take some time to identify a new management unit by running its management unit election algorithm.
  • Member Unit

Step 2: Determine the Physical Stacking Configuration.   

S-Series switches can be configured as a stack using either a ring topology or a cascade topology. An S50 model has a 10G stacking module built in. The S25P, S50N, and S50V models have two rear expansion slots that can be populated with 10G Ethernet modules or stacking modules, either a single-port 24G stacking module or a two-port (12G each) stacking module. You cannot interconnect the types. If you use single-port 24G modules, you can insert one in each expansion slot to accomplish the ring topology.

Although the following diagram shows the stacking cables connecting two systems through Stack Ports A and B, the ports are bi-directional, so you can connect any-to-any. 

Step 3: Pre-Configure Each Unit and Check Its SFTOS Version.

Each unit must run the same software version.

Before attaching the stack together, you can pre-configure the switch that you want to be the management unit, so that, by configuring a management address on it, you can more easily configure the stack.

If you have units that were configured before adding to the stack, you can ensure that the configuration of the management unit replaces the configuration of other units by connecting member units to the stack while they are powered down. Note also that you can power up each unit in the order in which you want them to assume the stack management role in failover situations.

Step 4: Rack-Mount the Units.

As a best practice, Force10 Networks recommends rack-mounting the S-Series units before you begin your stacking connections.

Step 5: Connect the Cables.

The following diagram illustrates the connections between Stack Port B on Switches 1 and 2, and between Stack Port A on Switches 2 and 3.   

Note: You can connect two A ports, two B ports, or port A and port B.  The ports are interchangeable and bi-directional.

The following diagram illustrates the stacking port cable connections for a ring topology.  Note how Stack Port B on Switch 3 is connected to Stack Port A on Switch 1. Such a connection provides a redundant data path in case a cable is removed or a stack port fails.

Step 6: Power On the Units. 

The S-Series is hot pluggable, and you can add units to a stack without powering down the active units. 

Note: Connecting a new unit to a stack will cause the units in the stack to reset as they absorb the new unit. A merge of two stacks will cause a general reset that results in new stack IDs if there are any duplicates.

Step 7: Verify the Unit Number, Stack Ports Status, and Stacking Role of each Unit

Status Panel LEDs: The front panel Status Panel LEDs do not indicate the status of stack ports. The LEDs indicate the presence of a stacking module only to the extent that an LED does not light up if it is normally associated with the 10G port now used by the stack port.

For example, in an S50N or S50V, the LEDs labeled XFP49 and 50 do not light up if a stacking module is inserted in the right-hand expansion bay (as you face the back panel). The LEDs labeled XFP 51 and 52 do not light up if a stacking module is inserted in the left-hand expansion bay.

Use the show switch command to view the status of the stack members.  

(S50) #show switch
        Management    Preconfig     Plugged-in  Switch Code
Switch    Status      Model ID      Model ID    Status Version
------ ------------ ------------- ------------- ------ -------
1      Mgmt Switch  SA-01-GE-48T  SA-01-GE-48T  OK     2.5.1
2      Stack Member SA-01-GE-48T  SA-01-GE-48T  OK     2.5.1

Use the show stack command to verify that the stack ports have a “Link Up” status.  

(S50) #show stack
                      Configured Running
                      Stack      Stack      Link         Link
Unit    Interface     Mode       Mode       Status       Speed (Gb/s)
---- ---------------- ---------- ---------- ------------ ------------
1    Stack Port A     N/A        Stack      Link Down    10
1    Stack Port B     N/A        Stack      Link Up      10
2    Stack Port A     N/A        Stack      Link Up      10
2    Stack Port B     N/A        Stack      Link Down    10

Note: In the above output, Stack Port A of Switch 1 and Stack Port B of Switch 2 have a “Link down” status because the redundant connection to create a ring topology is not being used.

Note: If a 24G stacking module is installed, the show stack-port command output displays the port as 50 if the module is installed in the right-hand expansion bay, port 51 if the module is installed in the left-hand expansion bay.

SFTOS automatically assigns a number to each unit as it comes up. These numbers are stored in NVRAM and remain in the configuration even after a unit is removed from a stack.

Changing a Stack ID (Unit Number)

To change the stack ID of an existing stack member, you can:

  • Plug the unit into a new stack and allow SFTOS to assign a new unit number automatically if it detects a unit in the stack with the same number.

    Note: A previously unconfigured unit will be assigned the lowest unused unit number.

  • Manually change the number with the switch renumber command. Note, however, that changing the stack ID obsoletes the existing configuration, requiring the manual update of each configuration element that includes a port number with the old stack ID. Consider backing up the configuration so that you can edit it as needed before reapplying it.

Use the following steps to renumber the stack ID of a switch. In this example, a stack has a single “Stack Member” switch.  

 (S50) #show switch
        Management   Preconfig     Plugged-in    Switch  Code
 Switch Status       Model ID      Model ID      Status  Version
 ------ ------------ ------------- ------------- ------  --------
  1     Mgmt Switch  SA-01-GE-48T  SA-01-GE-48T  OK      2.5.1
  2     Stack Member SA-01-GE-48T  SA-01-GE-48T  OK      2.5.1

Use the switch current-stackID renumber new-stackID command to change the stack ID of the switch.  

 (S50) #configure
 (S50) (Config)#switch 2 ?
 priority       Set priority of a stack switch.
 renumber       Renumber a stack switch.
 (S50) (Config)#switch 2 renumber ?
 <newunit>      Enter switch ID in the range of 1 to 8.
 (S50) (Config)#switch 2 renumber 3

SFTOS responds with a confirmation request that notes that the changed member unit will be reset. Enter "yes". If you execute the command on the management unit, the whole stack will be reloaded. You will then need to update the configuration with the new slot/port numbers.

Verify your configuration change with the show switch command. Initially, the switch will transition from Stack Member to Unassigned.  

 (S50) #show switch
        Management   Preconfig     Plugged-in    Switch  Code
 Switch Status       Model ID      Model ID      Status  Version
 ------ ------------ ------------- ------------- ------  --------
 1      Mgmt Switch  SA-01-GE-48T  SA-01-GE-48T  OK      2.5.1
 2      Unassigned   SA-01-GE-48T  Not Present           0.0.0

Wait for about 30 seconds and then execute the show switch command to verify that the switch has transitioned from Unassigned to Stack Member with the new unit number.  

 (S50) #show switch
         Management  Preconfig     Plugged-in   Switch  Code
 Switch  Status      Model ID      Model ID     Status  Version
 ------ ------------ ------------- ------------- ------ ----------
 1      Mgmt Switch  SA-01-GE-48T  SA-01-GE-48T  OK     2.5.1
 2      Unassigned   SA-01-GE-48T  Not Present          0.0.0
 3      Stack Member SA-01-GE-48T  SA-01-GE-48T  OK     2.5.1

Step 8: Configure and Manage the Stack

You manage the stack from the management unit. When connected to the console port of the management unit, you will see the expected prompt and have access to all CLI commands.  

If you connect to the console port of a member unit (not the management unit), you will see the prompt '(Unit number)>', where 'number' is the assigned unit number of this physical unit. No user commands can be executed from non-management units.

Note: Executing ‘?' at the CLI on a non-management unit returns the word 'devshell', which refers to an advanced engineering and diagnostic shell that only should be used under the direction of a Force10 Networks Technical Assistance Center (TAC) engineer. Unauthorized use of this shell could disrupt the functioning of your unit.

All of the forwarding protocols run on the management unit. The forwarding database resides on the management unit, which then synchronizes the tables to the other member units. The member units then make individual forwarding decisions based on their local copy of the forwarding table.

If a management unit powers down, resets, or is removed, a member unit is elected as the new management unit. During this time, traffic is disrupted as the forwarding tables are flushed and relearned.

In S25P, S50N, and S50V models, the stack ports can switch traffic at up to 48Gbps with two 24G stack ports installed in each switch in a ring topology, providing full linerate stacking (not oversubscribed).

In the original S50 model ("S50 Classic"), the stack ports switch traffic between units at a rate of 10 Gbps. Since each unit supports 48 x 1Gbps ports and two optical 10Gbps ports, the switching capacity between units is oversubscribed. It should be mentioned that the ring topology provides some performance gains. For example, a two-unit stack with two cables is able to exploit the full bandwidth of both stacking ports, effectively giving 20 Gbps between the units. Ring topology performance gains in larger stacks (4+) will probably not be as great, since transitional traffic will share bandwidth with traffic destined to, or originating from, a given switch.

Step 9: Save Configuration Changes.

You must save the configuration after making any changes:

 (S50) #copy system:running-config nvram:startup-config
 This operation may take a few minutes.
 Management interfaces will not be available during this time.
 Are you sure you want to save? (y/n) y

The above command writes the configuration to NVRAM on all member units. If you do not save changes and the management unit fails, all changes are lost across the member units.

Step-by-Step Example of Assembling a Stack of S-Series Switches

  1. Management Unit
    1. Power up the first unit; henceforth, this shall be considered the stack management unit.
    2. For clarity, configure stack unit number 1 for this unit (e.g. unit # 1); all physical ports will subsequently be configured via 1/0/1 to 1/0/xx.
    3. Configure the priority of this unit to be management unit by setting it at priority 14.
    4. Assign a management IP address and gateway for the stack.
  2. Backup Management Unit
    1. Power up the second unit (do not attach the stacking cables).
    2. Make sure it has the same software as the management unit.
    3. Configure the stack ID for this unit (e.g. unit # 2); all physical ports will subsequently be configured via 2/0/1 to 2/0/xx.
    4. Configure the priority of this unit to be the backup management unit (e.g. priority 13) .
    5. Power down this unit.
    6. Attach the this unit to the stack via stacking cable, and power it up.
  3. Third Unit
    1. Power up the next unit (do not attach the stacking cables yet).
    2. Make sure it has the same software as the management unit.
    3. Configure the unit number for this unit (e.g. unit # 3); all physical ports will subsequently be configured via 3/0/1 to 3/0/xx.
    4. Configure the management priority of this unit to be low (e.g. priority 1).
    5. Power down the unit.
    6. Attach the unit to the stack via stacking cable, and power it up.
    NOTE: By configuring a priority of 0 it is possible to prevent a unit from becoming a management unit in case of failover. The procedure above may be used for replacing any unit in an existing stack.

Using the show stack-port diag Command

To monitor the health of a stack running SFTOS, use the show stack-port diag command.  

 Force10 S50 #show stack-port diag
 1 - Stack Port A:
 RBYT:5fdd RPKT:53 TBYT:adf13 TPKT:8f2
 RFCS:0 RFRG:0 RJBR:0 RUND:0 ROVR:0
 TAGE:0 TABRT:0
 1 - Stack Port B:
 RBYT:0 RPKT:0 TBYT:b2d03 TPKT:930
 RFCS:0 RFRG:0 RJBR:0 RUND:0 ROVR:0
 TAGE:0 TABRT:0

The meaning of each field in the show stack-port diag command is described below:

  • RBYT – Number of bytes received

  • RPKT – Number of packets received

  • TBYT – Number of bytes transmitted

  • TPKT – Number of packets transmitted

  • RFCS – Number of Frame Check Sequence (FCS) errors received

  • RFRG – Number of undersized packets received with bad FCS

  • RJBR – Number of oversized packets received with bad FCS

  • RUND – Number of undersized packets received with good FCS

  • ROVR – Number of oversized packets received with good FCS

  • TAGE – Aged packets

  • TABRT – Error packets

Information to Collect if You Open a TAC Case

If you would like assistance from Force10 Networks after following the steps above, please use the Create Service Request form on the iSupport page and include the following information if available:

  • Console captures showing the steps taken
  • Output from the show tech-support command to capture the installed hardware and the SFTOS version
  • Network diagrams or other descriptions of the network design, including VLAN configurations and IP address ranges
  How do you rate this document ?
Not Helpful
Very Helpful
Comments / Suggestions