*********************************************************************** * * * This diskette contains IBM* Hot-Plug PCI System Bus Driver v1.00 * * for Novell IntranetWare** I (Novell NetWare** v4.11) * * Device Driver, Support Files, and Supported Hot-Plug PCI drivers * * * *********************************************************************** IBM* Hot-Plug PCI System Bus Driver v1.00 for IntranetWare I (NetWare v4.11) Installation Instructions Diskette 1 - IBM Hot-Plug PCI System Bus Driver v1.00 for IntranetWare I (NetWare v4.11) Device Driver, Support Files, and Supported Hot-Plug PCI drivers. NOTE: The latest version of this diskette can be found at http://www.pc.ibm.com/support on the World Wide Web. NOTE: The software on this diskette must be used to provide hot plug PCI support for Novell IntranetWare (NetWare v4.11) for hot plug capable IBM Netfinity systems. Without the software on this diskette, hot plug PCI operations are not supported. CONTENTS ======== 1.0 Hot-Plug PCI Installation for IntranetWare (NetWare v4.11) 1.1 Overview 1.2 Minimum requirements 1.3 Installation steps 1.4 Required driver support in AUTOEXEC.NCF 2.0 How to perform Hot-Plug PCI Operations 2.1 Required steps in Hot-Plug Operations 2.1.1 Removing an adapter 2.1.2 Replacing an adapter 2.1.3 Adding a new adapter to an empty slot 2.2 Hot-Plug PCI driver support 2.2.1 IBM Netfinity 10/100 Fault Tolerant Adapter 2.2.2 IBM 10/100 EtherJet PCI Adapter 2.2.3 IBM PCI Token-Ring Adapter 2.2.4 IBM ServeRAID II PCI Ultra SCSI Adapter 2.2.5 How to determine if your driver supports Hot-Plug PCI Operations 2.3 Warnings about incorrect usage of Hot-Plug PCI. 3.0 Troubleshooting Hot-Plug PCI Operations 3.1 Basic Troubleshooting 3.1.1 Attention Indicator LED 3.1.2 IBMSBD.NLM messages 3.1.3 Adapter Driver error messages 3.2 Advanced Troubleshooting 3.2.1 How to determine if the problem is with the adapter driver or the hot-plug driver. 3.2.2 Use of switches and log file for IBMSBD driver 3.2.3 Interrupt handling 3.2.4 NCMCON Issues 3.2.5 Known limitations 4.0 Trademarks and Service Marks: 1.0 Hot-Plug PCI Installation for IntranetWare (NetWare v4.11) 1.1 Overview The Hot Plug PCI support for NetWare v4.11 provided by IBM is compliant with many different specifications including full compliance with the NetWare System Bus Driver specification v1.01, PCI v2.1, and the PCI Hot-Plug PCI v1.0 specification. The IBM Hot Plug PCI system support consists of an interlock switch and a set of two LED's for each hot plug PCI slot. One LED will remain on when power is ON to the given slot. The other LED will indicate that attention is required to that slot. Slots that do not have these devices are NOT hot plug slots. All hot plug operations must be controlled through a console provided by Novell. IN NO CASE SHOULD HOT PLUG OPERATIONS BE DONE WITHOUT FIRST REMOVING POWER FROM THE SLOT THROUGH THE NETWARE CONFIGURATION MANAGER CONSOLE (NCMCON). IF A SLOT DOES NOT HAVE AN INTERLOCK, DO NOT REMOVE THE ADAPTER. DOING SO CAN CAUSE SERIOUS DAMAGE TO THE SYSTEM AND ADAPTER. The order of events in any hot-plug operation are: - Load the appropriate drivers (as described in Section 1.3 and 1.4) to provide hot plug PCI support. - Go to the NetWare Configuration Manager Console (NCMCON) - Choose an add or remove operation for the given slot. - When the adapter power is OFF, open the interlock and remove, replace or add an adapter and any necessary cabling. - Close the interlock and return to the NCMCON console. NOTE: The power to a slot cannot be turned ON until the interlock is closed for the given slot. The position of the interlock switch (opened or closed) is not important if no adapter is in the given slot. - In the case of adding a new adapter or replacing an old one, the NCMCON console will prompt you to turn on the newly added adapter. - If selected, the adapter power is turned ON and the adapter is configured. At this point, you will have to go to the system console and manually load the appropriate driver. NOTE: In the case of a network adapter, be sure to BIND the appropriate protocols to the driver after the driver is loaded. NOTE: In the case of a SCSI controller, you may need to issue the console command 'SCAN FOR NEW DEVICES' prior to the drive being recognized by NetWare. 1.2 Minimum requirements The minimum system requirements are: - IBM Netfinity Server with Hot Plug PCI slots. NOTE: The IBMSBD.NLM driver is designed not to load on any system that does not have Hot Plug PCI slots including non-IBM systems. If you wish to verify that your system hot-plug subsystem is working properly, load the IBMSBD.NLM driver with the -DETECT flag. (ie. LOAD IBMSBD -DETECT). A message will be displayed on the console which informs the user as to whether the driver believes the system to be an IBM Hot Plug PCI System or not. - IBM Hot Plug PCI System Bus Driver v1.00 (IBMSBD.NLM) provided by this diskette. - An adapter and Novell NetWare adapter driver that supports the ODI v3.31 specification or NWPA v3.00b specification published by Novell. and either: - Novell IntranetWare I (NetWare v4.11) with Support Pack 4 or later and the LIBUPG.EXE update package from Novell. The LIBUPG.EXE package contains a newer CLIB.NLM that is required by the Hot Plug PCI Installation Program to be able to copy files from the DOS partition. The LIBUPG.EXE file can be obtained from the Novell web site at URL: http://support.novell.com/search/ff_index.htm and entering the filename LIBUPG. or - Novell NetWare v5.0 (This diskette is not required because all the drivers, including IBMSBD.NLM, are located on the NetWare v5.0 CD-ROM. However, Sections 2 and 3 below regarding operations and troubleshooting are still valid) NOTE: Novell is not providing the support modules to allow support for any version of Novell NetWare prior to Novell IntranetWare I (NetWare v4.11). NOTE: To add a new adapter to an empty slot, the server must have the MPS14.PSM and SMP.NLM modules loaded in the STARTUP.NCF to have interrupts properly handled. Failure to have these modules loaded may prevent clients from connecting to the server. 1.3 Installation steps To install the IBM Hot Plug PCI driver and support files on this diskette to the appropriate directories and add the appropriate entries to SYS:\SYSTEM\AUTOEXEC.NCF: 1. Start the NetWare server and go to the NetWare console 2. Type LOAD INSTALL and press Enter. 3. From the installation menu, choose 'Product Options' 4. From the Product Options menu, choose 'Product Not Listed' 5. Place the IBM Hot Plug PCI driver diskette (this diskette) into the floppy drive. 6. Press Enter to accept the default path of A:\ NOTE: The contents of this diskette can also be copied to a subdirectory on the server (in either the DOS or NetWare partitions) and the path in step 6 changed to match the path to the subdirectory. 7. After a brief moment, the Main Menu of the IBM Hot Plug PCI System Bus Driver v1.00 Installation Program is displayed. 8. Choose 'Install NetWare v4.11 Support for IBM Hot Plug PCI System' 9. Type the path where this diskette image is located or accept the default of A:\ and press Enter. NOTE: If the installation program immediately returns to the Main Menu when selecting a path that is not on a NetWare volume, it is because the LIBUPG.EXE package has not been applied. The LIBUPG.EXE package contains a new CLIB.NLM module that is required by the installation program so that file accesses to the DOS partition will be successful. Before the installation program can continue, apply the LIBUPG.EXE package fixes and then restart these installation instructions. The LIBUPG.EXE file can be obtained from the Novell web site at URL: http://support.novell.com/search/ff_index.htm and entering the filename LIBUPG. 10. After the file copy takes place, press Escape to return to the Main Menu. 11. Choose 'Exit' to return to the Install Program. 12. Press Escape two times then 'Yes' to exit the NetWare Installation Program. 13. DOWN and restart the server to have the required modules load. NOTE: If installation is successful, you should see the the NetWare Configuration Manger Console (NCMCON) showing the available Hot Plug PCI slots, their current status, and any adapters that are currently in the Hot Plug PCI slots. 1.4 Required driver support in AUTOEXEC.NCF and diskette contents The IBM Hot Plug System Bus Driver requires five drivers in the AUTOEXEC.NCF file plus several other support files in order to perform hot plug operation. NOTE: These operations are all performed by the IBM Hot Plug PCI System Bus Driver Installation Program and the installation steps in Section 1.3. The five drivers required in the AUTOEXEC.NCF are: (these are order dependent) ODINEB.NLM 13,705 04-27-98 3:20pm (v1.08) IOCONFIG.NLM 2,810 04-30-97 4:56pm (v1.00) NCM.NLM 14,724 10-17-97 9:38am (v1.04) IBMSBD.NLM 98,472 05-15-98 8:55am (v1.00) NCMCON.NLM 24,163 01-26-98 2:20pm (v1.03) In addition, several other support files require replacement in order for Hot Plug PCI Operations to work correctly. The IBMSBD.NLM module requires the following files be replaced: NEB.NLM 19,952 04-30-97 4:39pm (v1.00) NBI.NLM 27,634 03-12-98 12:13pm (v2.16) NCMCON.CFG 8,250 05-15-98 9:24pm For all Ethernet Drivers, the following support files must be replaced: ETHERTSM.NLM 12,000 06-05-98 12:35pm (v3.67a) MSM.NLM 76,381 06-05-98 1:38pm (v3.80a) For all Token-Ring Drivers, the following support files must be replaced: TOKENTSM.NLM 12,264 06-05-98 12:29pm (v3.70a) MSM.NLM 76,381 06-05-98 1:38pm (v3.80a) For all FDDI Drivers, the following support files must be replaced: FDDITSM.NLM 12,500 11-19-97 4:43pm (v3.68) MSM.NLM 76,381 06-05-98 1:38pm (v3.80a) For all SCSI Disk Controllers, the following support files must be replaced: SCSIHD.CDM 24,349 10-18-96 1:48pm (v1.30) SCSICD.CDM 24,635 10-18-96 1:49pm (v1.30) SCSI2TP.CDM 38,047 10-18-96 1:44pm (v1.02q) NWASPI.CDM 12,462 02-25-97 11:46am (v1.00n) NWPALOAD.NLM 3,219 07-02-96 9:34am (v1.30) NWPAMS.NLM 73,849 05-13-98 1:35pm (v2.32g) NWPAIO.NLM 33,895 05-13-98 1:36pm (v2.32g) NPAPATCH.NLM 1,989 06-10-96 3:20pm (v1.02) NWPA.NLM 93,799 05-13-98 1:36pm (v2.32g) The supported drivers shipped on this diskette are: - IBM PCI Token-Ring Adapter IBMTRPO.LAN 79,863 05-06-98 12:00pm (v2.17) IBMTRPO.LDI 56,068 03-23-98 1:00pm RNIC.NLM 64,694 05-04-98 8:19am (v1.08) - IBM EtherJet 10/100 PCI Adapter IBMFE.LAN 48,474 03-23-98 3:25am (v3.25) IBMFE.LDI 8,403 10-02-97 3:19am - IBM ServeRAID II Ultra SCSI Adapter IPSRAID.HAM 23,458 05-18-98 11:29am (v2.81) IPSRAID.DDI 3,509 05-13-98 5:30pm IPSRAID.RED 37,078 09-23-97 9:18am IPSRADM.NLM 115,027 04-16-98 10:45am (v2.80c) - IBM Netfinity 10/100 Fault Tolerant Adapter PCNTNW.LAN 26,525 05-28-98 10:47am (v4.12) PCNTNW.LDI 16,926 05-12-98 10:59pm PCNET40.TXT 28,614 05-15-98 2:35pm - Other files on the diskette are: PINSTALL.NLM IBM Hot Plug PCI System Bus Driver Installation Program IBMSBD.TXT This file. 2.0 How to perform Hot-Plug PCI Operations 2.1 Required steps in Hot-Plug Operations NOTE: Opening an adapter interlock switch will turn off the power to the slot. If a device driver was loaded for that slot, the system will most likely hang, requiring the system to be rebooted. 2.1.1 Removing an adapter 1. Verify that the IBMSBD.NLM and NCMCON.NLM modules are loaded. 2. Go to the NetWare Configuration Manager Console (NCMCON) screen. 3. Select the slot where the adapter you want to remove is located and Press Enter. 4. Select 'Remove Adapter' and press Enter. 5. If a driver is loaded for the adapter, you will get a prompt asking if you want to unload the driver. You must have the driver unloaded for the power to the slot to be removed. 6. If you receive a message stating that the driver could not be unloaded, go to the system console and manually remove the driver. Return to NCMCON after the driver has been unloaded and repeat steps 3 through 5. 7. Verify that the status for the slot reads 'Powered off'. 8. Open the top cover of the system using the two thumb screws at the rear of the system. 9. Verify that the power-on light status for the adapter is OFF. If the light is ON, go back to the console and verify that the given slot is powered OFF. NOTE: NEVER REMOVE AN ADAPTER FROM A SLOT WITH THE SLOT POWER STILL ON. THIS COULD RESULT IN A SYSTEM HANG AND/OR SERIOUS DAMAGE TO THE ADAPTER CARD AND/OR SYSTEM UNIT. 10. Remove the interlock by swiveling the vertical lockdown counterclockwise and lift the interlock switch up. 11. Remove the adapter from the slot. 2.1.2 Replacing an adapter 1. Verify that the IBMSBD.NLM and NCMCON.NLM modules are loaded. 2. Go to the NetWare Configuration Manager Console (NCMCON) screen. 3. Select the slot where the adapter you want to remove is located and Press Enter. 4. Select 'Remove Adapter' and press Enter. 5. If a driver is loaded for the adapter, you will get a prompt asking if you want to unload the driver. You must have the driver unloaded for the power to the slot to be removed. 6. If you receive a message stating that the driver could not be unloaded, go to the system console and manually remove the driver. Return to NCMCON after the driver has been unloaded and repeat steps 3 through 5. NOTE: Beware that unloading the driver in most cases unloaded it for ALL instances of the adapter in the system. See Section 2.2 for any special commands that may be required to remove an individual instance of the driver. In the case where a driver does not have a special command to unload it and NCMCON is not able to unload the driver, a new device driver supporting 'instance unload' must be obtained from the adapter vendor. 'Instance unload' is part of the Novell NetWare ODI v3.31 LAN Specification and the Novell NetWare NWPA v3.00b Disk Specification. 7. Verify that the status for the slot reads 'Powered off'. 8. Open the top cover of the system using the two thumb screws at the rear of the system. 9. Verify that the power-on light status for the adapter is OFF. If the light is ON, go back to the console and verify that the given slot is powered OFF. NOTE: NEVER REMOVE AN ADAPTER FROM A SLOT WITH THE SLOT POWER STILL ON. THIS COULD RESULT IN A SYSTEM HANG AND/OR SERIOUS DAMAGE TO THE ADAPTER CARD AND/OR SYSTEM UNIT. 10. Remove the interlock by swiveling the vertical lockdown counterclockwise and lift the interlock switch up. 11. Remove the adapter from the slot. 12. Place the new adapter in the slot and close the interlock switch. Swivel the vertical lockdown clockwise to lock the interlock in place. 13. Go back to the NCMCON screen. A message should be displayed asking if power should be turned on to the given slot. 14. After selecting to turn on the power to a given slot, you will be prompted to verify that any necessary cables have been attached to the adapter card prior to power being applied. 15. Press Enter at the cable reminder screen. The adapter power will be turned on, the adapter configured, and a message displayed as to the success of adding the adapter. NOTE: If an error occurs during power ON and configuration of the adapter, the power to the slot will be turned back OFF, and an appropriate message displayed on the NCMCON screen and the system console screen. If an adapter does not power up, go to Section 3.0 Troubleshooting Hot Plug PCI Operations. NOTE: In the case of a network adapter, be sure to BIND the appropriate protocols to the driver after the driver is loaded. NOTE: In the case of a SCSI controller, you may need to issue the console command 'SCAN FOR NEW DEVICES' prior to the drive being recognized by NetWare. NOTE: Drivers for newly added adapters do not automatically reload. Please see the instructions for the specifically supported adapters for information on how to reload drivers. 16. The status of the adapter in the NCMCON screen will show READY if a driver is not loaded for the given slot, and ACTIVE when a driver is loaded. 2.1.3 Adding a new adapter to an empty slot NOTE: Adding a new adapter to a slot with an adapter already in the slot is the same as replacing an adapter. The steps to replacing an adapter are given in Section 2.1.2 above. NOTE: To add a new adapter to an empty slot, the server must have the MPS14.PSM and SMP.NLM modules loaded in the STARTUP.NCF to have interrupts properly handled. Failure to have these modules loaded may prevent clients from connecting to the server. 1. Verify that the IBMSBD.NLM and NCMCON.NLM modules are loaded. 2. Go to the NetWare Configuration Manager Console (NCMCON) screen. 3. Verify that the status for the slot reads 'Powered off'. 4. Open the top cover of the system using the two thumb screws at the rear of the system. 5. Verify that the power-on light status for the adapter is OFF. If the light is ON, go back to the console and verify that the given slot is powered OFF. NOTE: NEVER REMOVE AN ADAPTER FROM A SLOT WITH THE SLOT POWER STILL ON. THIS COULD RESULT IN A SYSTEM HANG AND/OR SERIOUS DAMAGE TO THE ADAPTER CARD AND/OR SYSTEM UNIT. 6. Remove the interlock by swiveling the vertical lockdown counterclockwise and lift the interlock switch up. 7. Remove the adapter from the slot. 8. Place the new adapter in the slot and close the interlock switch. Swivel the vertical lockdown clockwise to lock the interlock in place. 9. Go back to the NCMCON screen. A message should be displayed asking if power should be turned on to the given slot. 10. After selecting to turn on the power to a given slot, you will be prompted to verify that any necessary cables have been attached to the adapter card prior to power being applied. 11. Press Enter at the cable reminder screen. The adapter power will be turned on, the adapter configured, and a message displayed as to the success of adding the adapter. NOTE: If an error occurs during power ON and configuration of the adapter, the power to the slot will be turned back OFF, and an appropriate message displayed on the NCMCON screen and the system console screen. If an adapter does not power up, go to Section 3.0 Troubleshooting Hot Plug PCI Operations. NOTE: In the case of a network adapter, be sure to BIND the appropriate protocols to the driver after the driver is loaded. NOTE: In the case of a SCSI controller, you may need to issue the console command 'SCAN FOR NEW DEVICES' prior to the drive being recognized by NetWare. NOTE: Drivers for newly added adapters do not automatically reload. Please see the instructions for the specifically supported adapters for information on how to reload drivers. 12. The status of the adapter in the NCMCON screen will show READY if a driver is not loaded for the given slot, and ACTIVE when a driver is loaded. 2.2 Hot-Plug PCI driver support IBM has tested and certified the following adapters in hot plug PCI systems. Adapters other than the ones listed below may work. See Section 2.2.5 for steps on determining whether an adapter driver is capable of supporting hot plug PCI operations. NOTE: For questions about non-IBM adapters supporting hot plug PCI, please contact the appropriate adapter vendor. Information in Section 2.2.5 can be used when contacting the adapter vendor to determine level of hot plug PCI support provided. 2.2.1 IBM Netfinity 10/100 Fault Tolerant Adapter This adapter is fully supported in hot plug PCI operations. The driver provided with this diskette, PCNTNW.LAN, should be used to provide hot plug PCI support. If the PCNTNW.LAN driver is already loaded, and a second IBM Netfinity 10/100 Fault Tolerant Adapter is loaded, you must use the following command to find the new adapter: LOAD PCNTNW SCAN For information about other parameters and setting up the fault tolerant fail-over features of this adapter, please see the PCNET40.TXT file on this diskette. The latest version of this driver can be found at: http://www.pc.ibm.com/us/searchfiles.html on the World Wide Web. 2.2.2 IBM 10/100 EtherJet PCI Adapter This adapter is fully supported in hot plug PCI operations. The driver provided with this diskette, IBMFE.LAN, should be used to provide hot plug PCI support. The IBMFE.LAN driver will auto-detect newly added adapters when it is reloaded. Once the IBM 10/100 EtherJet PCI Adapter is hot added to the system, use the following command to load the driver for this adapter: LOAD IBMFE No additional information is provided with this diskette for this adapter. The latest version of this driver can be found at: http://www.networking.ibm.com/servers on the World Wide Web. 2.2.3 IBM PCI Token-Ring Adapter This adapter is fully supported in hot plug PCI operations. The driver provided with this diskette, IBMTRPO.LAN, should be used to provide hot plug PCI support. The IBMTRPO.LAN driver will auto-detect newly added adapters when it is reloaded. Once the IBM PCI Token-Ring Adapter is hot added to the system, use the following command to load the driver for this adapter: LOAD IBMTRPO NOTE: If the new adapter is the first adapter in the network, it is necessary to set the datarate for the adapter to the speed that your Token-Ring network operates. The following command will allow for 16Mbps operations: LOAD IBMTRPO datarate=m16 For 4Mbps operation, replace the m16 with m4 in the previous command. NOTE: If you are using the redundant NIC capability provided by the IBMTRPO.LAN v2.14 or later, you will have to re-establish the failover pair after a hot plug PCI operation. For more information about the parameters that should be to re-establish the failover pair, please see the document at the following web site: http://www.networking.ibm.com/trl/trlrnic.html No additional information is provided with this diskette for this adapter. The latest version of this driver can be found at: http://www.networking.ibm.com/servers on the World Wide Web. 2.2.4 IBM ServeRAID II PCI Ultra SCSI Adapter This adapter is fully supported in hot plug PCI operations. The driver provided with this diskette, IPSRAID.HAM, should be used to provide hot plug PCI support. NOTE: The original IBM ServeRAID adapter is not supported for hot plug operations. The IPSRAID.HAM driver will auto-detect newly added adapters when it is reloaded. Once the IBM ServeRAID II Ultra SCSI Adapter is hot added to the system, use the following command to load the driver for this adapter: LOAD IPSRAID NOTE: Loading the IPSRAID adapter may take several minutes to complete. This is because the driver searches all three channels for all fifteen possible SCSI devices. SCSI devices have up to five seconds to timeout. Control will be returned to the console prior to the IPSRAID SCSI search completion so that other server console commands can be issued. The IPSRAID driver requires a special command to unload a single instance of the driver. The command to remove the single instance is: REMOVE STORAGE ADAPTER Ax where x is the adapter number. NOTE: The adapter numbers for the Netfinity 5500 start at A1 for the integrated RAID controller. If any ServeRAID II adapters are in slots 5 and 6, they will be the next in sequence. Any adapters in the hot plug slots will be next. Adapters are sequenced starting at slot 1 and proceeding to slot 4. Example (You have one ServeRAID II adapter in slot 2 and slot 5 on a Netfinity 5500). The ServeRAID II adapters adapters are labeled as follows: A1 is the integrated RAID controller. A2 is the ServeRAID II adapter in slot 5. A3 is the ServeRAID II adapter in slot 2. For information about other parameters and setting up the fault tolerant fail-over features of this adapter, please see the IPSRAID.RED file on this diskette. The latest version of this driver can be found at: http://www.pc.ibm.com/us/searchfiles.html on the World Wide Web. 2.2.5 How to determine if your driver supports Hot-Plug PCI Operations NOTE: This section shows how to handle devices that are not on the IBM approved list of hot plug PCI devices. Some devices may work correctly, but have not been tested and approved by IBM. NOTE: There are no changes to the physical adapter required to support hot plug PCI operations. Many other adapter vendors will be providing drivers that meet the required Novell specifications for hot plug PCI in the future. However, there are times where you may have a driver and will not know whether the driver is hot plug compatible. The following is a list of methods to determine if a given driver supports hot plug PCI operations. 1. Almost any driver will allow a singular instance in the hot plug environment. NOTE: The limitations to this are any adapters with PCI-to-PCI bridges. Because of limitations in how NetWare handles interrupts in NetWare v4.11, any device with a PCI-to-PCI bridge will not be successfully added. These devices will generate a message stating that they are not supported during PCI operations. 2. The device can be powered OFF through the NCMCON screen with a driver loaded or has a command-line option to remove a single instance of the driver. NOTE: A driver that forces all instances of itself to be unloaded is not considered a hot plug PCI driver. 3. Can the driver detect newly added adapters after a hot plug operation? NOTE: Many adapters will not detect new slot numbers because they do a static scan for cards on their first load. If a new card is added, the driver will report that no slots are available. Unloading the adapter (and sometimes the underlying support module, TOKENTSM.NLM, ETHERTSM.NLM, etc) and reloading them will allow the new slots to be seen in most instances. However, this will require unloading the driver for all adapters in the system. 4. Does the driver complain about PCI configuration problems? NOTE: Some older drivers do not allow for PCI configuration resources to be reallocated or changed. These drivers will report that one or more of the PCI resource requirements (IRQs, I/O ports, Memory, or Prefetchable Memory) are not correct/unavailable. 2.3 Warnings about incorrect usage of Hot Plug PCI. Below are several warnings about hot plug PCI operations: 1. Do not remove an adapter from a non hot plug PCI slot. 2. Grounding equipment, including wrist straps, should be used when working inside any system unit to protect against electrostatic discharge (ESD) that could damage system components and adapter cards. 3. Make sure that any adapter card hot added to the system is fully seated before attempting to turn on power. 4. Opening an adapter interlock switch will turn off the power to the slot. If a device driver was loaded for that slot, the system will most likely hang, requiring the system to be rebooted. 5. Do NOT remove the plastic cover over the system components. Removal of the plastic cover exposes system components that may be shorted by adapter brackets during hot plug PCI operations. 6. Processors, memory and integrated systems devices are not hot swappable. 7. Opening the adapter interlock with an adapter powered in the slot or closing the interlock after such an activity may lead to an NMI being reported by the system. If this occurs, the system must be rebooted for the NMI to be cleared. 3.0 Troubleshooting Hot-Plug PCI Operations 3.1 Basic Troubleshooting There are several troubleshooting techniques that can be used to determine why a hot plug PCI operation failed. Two LED's are provided for each hot plug PCI capable slot. One LED blinks to indicate that attention is required. The second LED indicates power state. Messages are generated by the NCMCON screen, IBMSBD.NLM driver, and the various adapter drivers. 3.1.1 Attention Indicator LED The Attention Indicator LED is controlled by NetWare. At present, the Attention Indicator LED is used only by device drivers to indicate that an adapter in a given slot needs attention. Not all device drivers support the attention indicator messaging. The Attention Indicator will only be cleared upon successful replacement of the adapter in the slot or by the device driver clearing the condition that led it to turn on the Attention Indicator LED in the first place. 3.1.2 IBMSBD.NLM messages The IBMSBD.NLM IBM System Bus Driver generates messages to indicate change of state in the hot plug PCI system. If the driver is not loaded, no system messages are generated. The messages that you may see generated by the IBMSBD driver will all be preceeded by the IBMSBD: or IBMSBD Error: tags. The messages that may be seen are: "IBMSBD: New Adapter added. Please use the Novell Configuration Manager console (NCMCON) to configure this new adapter." This indicates that a new adapter was added into a previously empty slot. The empty slot can occur as a result of a previous replace or add hot plug PCI operation. This is an information message directing you to use the console (NCMCON) to configure the newly added adapter. "IBMSBD Error: Allocate NEB.AESTag() failed." This error indicates that the IBMSBD driver was unable to allocate an asynchronous event system (AES) tag. Because events happen asynchronously in the hot plug PCI system, IBMSBD must have an AES handle to function. Unloading other NLM's that have AES tags registered will free up AES resources so that the IBMSBD driver can properly load. "IBMSBD Error: Not enough memory to generate event for queue." This error indicates that memory could not be allocated to place a hot plug PCI event onto the internal resource queue. This normally occurs when the system runs out of available memory. To fix this problem, unload other NLM's or add additional memory to the system. "IBMSBD Error: Add Adapter command failed because of empty slot. Slot is x" This messages occurs when NCMCON is directed to add an adapter to a slot that does not currently have an adapter card in it. This error may also occur if a PCI adapter does not meet the PCI v2.1 specification requirement that the adapter use presence pins. The IBMSBD module uses these presence pins to determine when an adapter has been inserted and removed from the system. "IBMSBD: The IBM Hot-Plug PCI controller is not present." This error occurs when an attempt is made to run the IBMSBD module on a system that does not have a hot plug PCI controller. This error may also occur if the hot plug PCI controller is malfunctioning or not working. The -COMMAND -DEBUG switches can be used on the IBMSBD driver to determine if the hot plug controller is working. See Section 3.2.2 for more information about debug switches for the IBMSBD device driver. All other messages are provided through the Novell Configuration Manager Console (NCMCON) screen. For information about problems in NCMCON such as all slots show 'No' in the Hot Plug field or a message appears asking whether to continue waiting another 10 seconds, see Section 3.2.4. If IBMSBD.NLM fails to load with a message stating that a symbol was not found NBIDetect(), this means that an old level of NBI.NLM is being used Replace the NBI.NLM with the one from this diskette and IBMSBD.NLM will load properly. 3.1.3 Adapter Driver error messages Many non hot plug PCI device drivers will not load correctly with the ODI v3.31 or the NWPA v3.00b specifications. If you have such a device driver, first determine if you adapter vendor has a certified driver available. If no such driver exists, you will need to either use a different adapter or lose at least some of the hot plug functionality. Drivers that do not support the Novell specifications may exhibit such problems as: - Failure to see new adapters until unloaded and reloaded. - Failure to support single instance unload. Single instance unload is the ability to unload the driver for a single adapter even though the driver may be supported many adapters. - Failure to be able to register PCI resources such as interrupt, I/O ports, memory or prefetchable memory ranges. Some drivers that support the Novell specification may exhibit problems such as those above if the support modules (NWPA.NLM, ETHERTSM.NLM, TOKENTSM.NLM, FDDITSM.NLM, and NBI.NLM) are down-level. Verify that all drivers are at least at the level of drivers on this diskette. Since these are Novell support modules, check the Novell web site for later versions that may be on this diskette. The web site is http://support.novell.com. 3.2 Advanced Troubleshooting This section contains information that is more technically advanced than the Basic Troubleshooting section. Information contained in this section includes: - How to determine if a problem loading an adapter device driver is caused by the hot plug PCI driver, IBMSBD.NLM, or the device driver itself - debug switches available with the IBMSBD.NLM driver - how interrupts are assigned - NCMCON issues such as no hot plug slots showed, or a query about waiting another ten seconds - Known limitations of the driver 3.2.1 How to determine if the problem is with the adapter driver or the hot-plug driver. NOTE: There are no changes to the physical adapter required to support hot plug PCI operations. Many other adapter vendors will be providing drivers that meet the required Novell specifications for hot plug PCI in the future. However, there are times where you may have a driver and will not know whether the driver is hot plug compatible. The following is a list of methods to determine if a given driver supports hot plug PCI operations. 1. Almost any driver will allow a singular instance in the hot plug environment. NOTE: The limitations to this are any adapters with PCI-to-PCI bridges. Because of limitations in how NetWare handles interrupts in NetWare v4.11, any device with a PCI-to-PCI bridge will not be successfully added. These devices will generate a message stating that they are not supported during PCI operations. 2. The device can be powered OFF through the NCMCON screen with a driver loaded or has a command-line option to remove a single instance of the driver. NOTE: A driver that forces all instances of itself to be unloaded is not considered a hot plug PCI driver. 3. Can the driver detect newly added adapters after a hot plug operation? NOTE: Many adapters will not detect new slot numbers because they do a static scan for cards on their first load. If a new card is added, the driver will report that no slots are available. Unloading the adapter (and sometimes the underlying support module, TOKENTSM.NLM, ETHERTSM.NLM, etc) and reloading them will allow the new slots to be seen in most instances. However, this will require unloading the driver for all adapters in the system. 4. Does the driver complain about PCI configuration problems? NOTE: Some older drivers do not allow for PCI configuration resources to be reallocated or changed. These drivers will report that one or more of the PCI resource requirements (IRQs, I/O ports, Memory, or Prefetchable Memory) are not correct/unavailable. 5. If new device drivers, support modules, and installation of the latest Support Packs does not fix the problem, then proceed to Section 3.2.2 to debug the IBMSBD.NLM module. 3.2.2 Use of switches and log file for IBMSBD driver There are a number of command line switches that can be used to create an IBMSBD.LOG file in the SYS:\SYSTEM directory. This file can be looked at to verify that the IBMSBD.NLM module is working properly. The following are the command line switches and a description of each: The following commands are NOT case sensitive, NOR do they require the leading hyphen. If multiple commands are given, they should be separated by spaces. (ie. LOAD IBMSBD -DETECT DEBUG -COMMAND) /?,-?, ? These flags are used to print out the following list of switches. -DETECT This parameter will allow the IBMSBD.NLM driver to detect if the hot plug PCI system and controller are detected. The driver will print a message to the system console as to its finding and then terminate. This flag cannot be used with any other flag. -DEBUG This parameter must be listed for any debug actions to be logged to the IBMSBD.LOG file. Various system information including version of NetWare, PCI BIOS discovery, Novell Event Bus (NEB) events, Novell HIN numbers, driver deregistration, memory release traces and the PCI Interrupt Routing Options table are documented. -COMMAND This parameter displays the returns codes from the PCI hot plug controller, return codes to NCMCON, and all PCI hot plug operations and return codes. -RESOURCE This parameter designates that resource tables are to be generated in the IBMSBD.LOG file. This parameter must be used in conjunction with one of the following three parameters to generate anything. -IO This parameter, used in conjunction with the -RESOURCE parameter, will generate tables by system bus showing all I/O resources assigned by BIOS and the IBMSBD.NLM module. This parameter also shows all I/O assignments during the hot plug PCI operation. -MEM This parameter, used in conjunction with the -RESOURCE parameter, will generate tables by system bus showing all PCI memory resources assigned by BIOS and the IBMSBD.NLM module. This parameter also shows all memory assignments during the hot plug PCI operation. -PFMEM This parameter, used in conjunction with the -RESOURCE parameter, will generate tables by system bus showing all PCI prefetchable memory resources assigned by BIOS and the IBMSBD.NLM module. This parameter also shows all prefetchable memory assignments during the hot plug PCI operation. -IRQ This parameter shows all system devices by PCI deviceID and vendorID. The information provided shows the slot that the adapter occupies and the IRQ number (1-15) assigned to offset 0x3c of the PCI Configuration Header for the given adapter. Slot numbers of 0000 indicate that the device is an integrated PCI device. Interrupts for reserved system resources and non-PCI devices are not displayed. All information is reported in the IBMSBD.LOG file in the SYS:\SYSTEM directory. -DEVICE This parameter places information in the IBMSBD.LOG file about PCI devices found in the system. Information about the bus, slot number, device/function ID, classCode, VendorID, and device type are logged. If a problem arises where you must create the IBMSBD.LOG file, the file can be submitted as part of the call to the HelpCenter for further diagnostic help. 3.2.3 Interrupt handling Interrupts are handled in the following manner: - Only interrupts 9, 10, and 11 can be assigned to hot plug PCI devices. These options are not settable. If during BIOS setup, these three interrupts are reserved for ISA Legacy devices, all hot plug PCI operations will fail. - IRQ 15 will never be assigned to hot plug PCI devices. - Interrupts are preserved from the MPS table located in the extended BIOS data area (EBDA) of the system. - For slots that do not have an adapter in them at boot, interrupts are assigned according to the following formula: - If another device that matches the vendorID and deviceID in the PCI Configuration header is found, then the newly added adapter will receive the same interrupt as the other device. - An unused interrupt between 9, 10, and 11 is assigned. - The least used interrupt between interrupts 9, 10, and 11 is assigned. NOTE: There may be instances where a LAN and SCSI controller may be assigned the same interrupt. If this occurs, the LAN adapter may fail to handle interrupts correctly when the SCSI controller accesses the DOS partition and lose connections. If this occurs, it will be necessary to down the server, reboot it, go into System Setup, and manually assign the interrupts so the devices do not share interrupts. 3.2.4 NCMCON issues - If NCMCON.NLM fails to show any slots with 'Yes' in the Hot Plug field, the IBMSBD.NLM module is not loaded. To fix the problem, exit NCMCON, load IBMSBD at the system console, and reload NCMCON. - If you receive a message about Slot Status not being available with a query to wait another 10 seconds, NCM.NLM or one of its support modules, IOCONFIG.NLM or NEB.NLM is not loaded. To fix this problem, exit NCMCON, load the appropriate support modules, load NCM, then NCMCON. It is not necessary for NCM or NCMCON to be loaded for the IBMSBD driver to load properly. - If the adapter names are not displayed correctly in the Adapter Name field of NCMCON, verify that the NCMCON.CFG file is located in the SYS:\SYSTEM subdirectory. - If a fully supported ODI v3.31 LAN device driver cannot be unloaded, verify that ODINEB.NLM is loaded. It can be loaded at the system console after NEB.NLM is loaded. - All SCSI controller drivers must be unloaded from the system console. The following command to remove a single adapter and its volumes from the system: REMOVE STORAGE ADAPTER Ax where x is the adapter number as determined by the scan order of the PCI buses in the system. All integrated devices will be assigned first, followed by adapters in PCI Bus 0, PCI Bus 1, etc. 3.2.5 Known limitations - Bridge adapters or PCI devices containing PCI-to-PCI bridges are NOT supported. - Some LAN adapter drivers may not handle interrupts correctly if their wake-on-lan setting is active after being hot-added. The driver will normally load, but clients will not be able to attach to the server through that adapter. - Multi-Function PCI devices are supported as long as the multi-function capability is not provided with or through a PCI-to-PCI bridge. - Video adapters are not supported in the hot plug PCI environment due to I/O space limitations and restrictions. - Devices that are not PCI v2.1 compliant or that do not implement the PCI presence detection pins are not supported. - Device drivers that are not LAN ODI v3.31 compliant or SCSI NWPA v3.00b compliant are NOT supported. - Non-PCI devices are not hot pluggable. - HWDETECT.NLM, a Novell provided module, may not find the correct driver if more than one driver is capable of loading for the given adapter. - HWDETECT.NLM may give an error about not being able to open the SYS:\SYSTEM\AUTOEXEC.NCF file. Ignore this message by pressing Enter and proceeding through the driver detection. - Because of errors encountered in testing, HWDETECT.NLM is not provided by IBM. If you want the capability of being able to auto-detect the correct driver for newly added adapters, download the latest HWDETECT.NLM from the Novell web site. The URL to find the latest version of HWDETECT.NLM from Novell's web site is: http://support.novell.com/search/ff_index.htm 4.0 Trademarks and Service Marks The following terms, denoted by an asterisk (*) in this README, are trademarks or registered trademarks of the IBM Corporation in the United States or other countries: ------------------------------------------ IBM ------------------------------------------ NetFinity ------------------------------------------ ServeRAID ------------------------------------------ EtherJet ------------------------------------------ The following terms, denoted by a double asterisk (**) in this README, are trademarks of other companies as follows: Trademark: Owner: -------------------------------------------------------------- NetWare Novell, Inc. -------------------------------------------------------------- IntranetWare Novell, Inc. -------------------------------------------------------------- THIS DOCUMENT IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND. IBM DISCLAIMS ALL WARRANTIES, WHETHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF FITNESS FOR A PARTICULAR PURPOSE AND MERCHANTABILITY WITH RESPECT TO THE INFORMATION IN THIS DOCUMENT. BY FURNISHING THIS DOCUMENT, IBM GRANTS NO LICENSES TO ANY PATENTS OR COPYRIGHTS. Note to U.S. Government Users -- Documentation related to restricted rights -- Use, duplication or disclosure is subject to restrictions set forth in GSA ADP Schedule Contract with IBM Corp.