Print | Rate this content

Notice: (Revision) CUSTOMER ATTENTION REQUIRED: HP Integrated Lights-Out and Integrated Lights-Out 2 - Scanning First-Generation iLO or iLO 2 Devices for the Heartbleed Vulnerability Results in iLO Lockup Requiring Power to be PHYSICALLY Removed

SUPPORT COMMUNICATION - CUSTOMER NOTICE

Document ID: c04249852

Version: 3

Notice: (Revision) CUSTOMER ATTENTION REQUIRED: HP Integrated Lights-Out and Integrated Lights-Out 2 - Scanning First-Generation iLO or iLO 2 Devices for the Heartbleed Vulnerability Results in iLO Lockup Requiring Power to be PHYSICALLY Removed
NOTICE: The information in this document, including products and software versions, is current as of the Release Date. This document is subject to change without notice.

Release Date: 2014-05-05

Last Updated: 2014-05-05


DESCRIPTION

Document Version
Release Date
Details
3
05/05/2014
Added Note regarding Integrity Server Blades.
Added "IMPORTANT" statement and recovery steps regarding a potential LAN or SAN outage in a Virtual Connect environment when a server blade is reset, prior to updating to iLO 2 Firmware Version 2.25.
Added links to first-generation iLO Firmware Version 1.96 that resolves potential iLO lockup due to port scanning.
2
04/23/2014
Added links to Integrated Lights-Out 2 (iLO 2) Firmware Version 2.25 that resolves this issue.
1
04/16/2014
Original Document Release.

HP Integrated Lights-Out products (iLO, iLO 2, iLO 3, iLO 4) do not use the OpenSSL library and are NOT exposed to the CVE-2014-0160 vulnerability (now known as "Heartbleed") in the open-source OpenSSL toolkit described in the following Customer Notice:

HP Servers Communication: OpenSSL "Heartbleed" Vulnerability http://h20564.www2.hp.com/portal/site/hpsc/public/kb/docDisplay/?docId=emr_na-c04239413

However, the first-generation iLO and iLO 2 products use the RSA SSL libraries and there is a bug in these libraries that will cause first-generation iLO and iLO 2 devices to enter a live lockup situation when a vulnerability scanner runs to check for the Heartbleed vulnerability. Although the server's operating system will continue to function normally, first-generation iLO and iLO 2 will no longer be responsive over the management network.

Note: Integrated Lights-Out 3 (iLO 3) and Integrated Lights-Out 4 (iLO 4) are NOT affected by this issue.

Note: The following platforms using the Integrity iLO 2 are also affected by Heartbleed vulnerability scanners; however, they will recover to a functional state without intervention after 60 to 90 seconds:

  • HP Integrity BL860c Server Blade
  • HP Integrity BL870c Server Blade
  • HP Integrity rx3600 Server
  • HP Integrity rx6600 Server
  • HP Integrity rx2660 Server

DETAILS

HP recommends that customers DO NOT RUN Heartbleed vulnerability scanners against servers with first-generation iLO and iLO 2.

To prevent this issue from occurring on iLO 2, update to iLO 2 Firmware Version 2.25, available for download at the following ftp locations:

To prevent this issue from occurring on first-generation iLO, update to iLO Firmware Version 1.96, available for download at the following ftp locations:

First-Generation iLO - Online ROM Flash Component for Windows
ftp://ftp.hp.com/pub/softlib2/software1/sc-windows-fw-ilo/p164481422/v97350

First-Generation iLO - Online ROM Flash Component for Linux ftp://ftp.hp.com/pub/softlib2/software1/sc-linux-fw-ilo/p1980791503/v97349

At this time, the only method of recovery for a first-generation iLO or iLO 2 device in a ProLiant ML-series or DL-series server that is in the live lockup situation is to PHYSICALLY remove power from the server for approximately ten seconds to initiate a full restart.

For HP BladeSystem c-Class Server Blades, issue the following command from the Onboard Administrator command line to recover:

reset server <bay #>

IMPORTANT : A potential LAN or SAN outage may occur in a Virtual Connect environment when a server blade is reset using any of the following methods, prior to updating to iLO 2 Firmware Version 2.25:

  • OA CLI reset server <bay #>
  • Physically reseating the server blade
  • Virtual Connect Manager (VCM)
  • OA failover
  • Physically reseating the primary VC Ethernet IO Module
  • VCM-OA NO_COMM restoration

Non-Virtual Connect environments are NOT impacted. This issue impacts the follows versions of VC firmware:

  • VC Firmware Version 4.10 (or earlier): Potential LAN (Ethernet & iSCSI) or SAN (FC & FCoE) outage
  • VC Firmware Version 4.20: Potential FC outage only

Note: The Virtual Connect firmware itself is not vulnerable to CVE-2014-0160 (Heartbleed), however, the installer component in versions 4.10 and 4.20 of Virtual Connect does have the vulnerability, and should be replaced with versions 4.10b or 4.20b, or the latest version of Virtual Connect Support Utility (VCSU). The VCSU vulnerability is only present during the firmware upgrade process.

Recovery Method for HP Blade Servers in a Virtual Connect Environment

The target blade enclosure must be taken offline completely to perform the recovery steps. Therefore, production workloads need to be gracefully terminated prior to performing the following steps.

Quick Recovery Steps

Prerequisite: iLO 2 Firmware Version 2.25

  1. Remove power from the impacted blade enclosure completely
  2. Power the enclosure back on, wait 5 minutes and then log in to Onboard Administrator (OA) and review system log to ensure that no "Management Processor on Blade # appears unresponsive" error messages have occurred, since power was restored.
  3. Update all server blades to iLO Firmware Version 2.25.
  4. Login to the OA and review the System Log. All server blades with a VC server profile should have the message: "Blade # is now configured for Virtual Connect Manager".
  5. Ensure all server blades are powered on and test blade connectivity. If any server blade has connectivity issues, it is recommended to issue a "reset server <bay #>" command to restore the connectivity.

This option quickly recovers unresponsive iLO devices caused by the vulnerability scan, which initially triggered the VCM LAN or SAN disconnection. When running current VC firmware, the system remains susceptible to iLO becoming unresponsive on a server blade for other reasons and triggering the LAN or SAN disconnect when a VCM enclosure discovery process is initiated due to an event from a server blade reset, VCM reset, OA failover, reseating the primary VC Ethernet IO Module or a VCM-OA NO_COMM restoration . These steps should bring the environment back to its stabilized condition prior to iLO becoming unresponsive.

Verification:

  • Check OA status in the User Interface (UI) or System Log for a specific iLO device as shown in the following screen captures:

OK State:

  • Check Status and Management Processor from the Diagnostic Information table as shown below:

 

Check the OA System Log as shown in the following screen capture:

 

iLO 2 shown as unresponsive:

 

Check OA System Log:

 

This issue will be corrected in a future release of Virtual Connect Firmware. This Notice will be updated when additional information becomes available.


Hardware Platforms Affected: HP Integrated Lights-Out (iLO 1), HP Integrated Lights-Out (iLO) Firmware(Standard HP Product), HP Integrated Lights-Out 2 (iLO 2), HP Integrated Lights-Out 2 (iLO 2) Firmware for ProLiant G6 Servers(Standard HP Product), HP Integrity BL60p Server, HP Integrity BL860c Server Blade, HP Integrity BL870c Server Blade, HP ProLiant BL20p G2 Server Blade, HP ProLiant BL20p G3 Server Blade, HP ProLiant BL20p G4 Server Blade, HP ProLiant BL20p Server Blade, HP ProLiant BL25p G2 Server Blade, HP ProLiant BL25p Server Blade, HP ProLiant BL260c G5 Server Blade, HP ProLiant BL280c G6 Server Blade, HP ProLiant BL2x220c G5 Server Blade, HP ProLiant BL2x220c G6 Server Blade, HP ProLiant BL30p Server Blade, HP ProLiant BL35p Server Blade, HP ProLiant BL40p Server series, HP ProLiant BL45p G2 Server series, HP ProLiant BL45p Server series, HP ProLiant BL460c G5 Server Blade, HP ProLiant BL460c G6 Server Blade, HP ProLiant BL460c Server Blade, HP ProLiant BL465c G5 Server Blade, HP ProLiant BL465c G6 Server Blade, HP ProLiant BL465c Server Blade, HP ProLiant BL480c Server Blade, HP ProLiant BL490c G6 Server Blade, HP ProLiant BL495c G5 Server Blade, HP ProLiant BL495c G6 Server Blade, HP ProLiant BL680c G5 Server Blade, HP ProLiant BL685c G5 Server Blade, HP ProLiant BL685c G6 Server Blade, HP ProLiant BL685c Server Blade, HP ProLiant DL320 G3 Server, HP ProLiant DL320 G4 Server, HP ProLiant DL320 G5 Server, HP ProLiant DL320 G5p Server, HP ProLiant DL320 G6 Server, HP ProLiant DL320s Server, HP ProLiant DL320s Storage Server, HP ProLiant DL360 G2 Server, HP ProLiant DL360 G3 Server, HP ProLiant DL360 G4 Server, HP ProLiant DL360 G4p Server, HP ProLiant DL360 G5 Server, HP ProLiant DL360 G6 Server, HP ProLiant DL365 G5 Server, HP ProLiant DL365 Server, HP ProLiant DL370 G6 Server, HP ProLiant DL380 G3 Packaged Cluster with MSA1000, HP ProLiant DL380 G3 Packaged Cluster with MSA500, HP ProLiant DL380 G3 Packaged Cluster with MSA500 Racked, HP ProLiant DL380 G3 Server, HP ProLiant DL380 G4 Data Protection Storage Server, HP ProLiant DL380 G4 Packaged Cluster with MSA1000, HP ProLiant DL380 G4 Packaged Cluster with MSA500 G2, HP ProLiant DL380 G4 Server, HP ProLiant DL380 G4 Storage Server, HP ProLiant DL380 G5 Server, HP ProLiant DL380 G5 Storage Server, HP ProLiant DL380 G6 Server, HP ProLiant DL385 G2 Server, HP ProLiant DL385 G5 Server, HP ProLiant DL385 G5p Server, HP ProLiant DL385 G6 Server, HP ProLiant DL385 Server, HP ProLiant DL560 Server, HP ProLiant DL580 G2 Server, HP ProLiant DL580 G2 Storage Server, HP ProLiant DL580 G3 Server, HP ProLiant DL580 G4 Server, HP ProLiant DL580 G5 Server, HP ProLiant DL585 G2 Server, HP ProLiant DL585 G2 Storage Server, HP ProLiant DL585 G5 Server, HP ProLiant DL585 G6 Server, HP ProLiant DL585 Server, HP ProLiant DL740 Server, HP ProLiant DL785 G5 Server, HP ProLiant DL785 G6 Server, HP ProLiant ML310 G3 Server, HP ProLiant ML310 G3 Storage Server, HP ProLiant ML310 G4 Server, HP ProLiant ML310 G4 Storage Server, HP ProLiant ML310 G5 Server, HP ProLiant ML310 G5p Server, HP ProLiant ML330 G6 Server, HP ProLiant ML350 G4p Server, HP ProLiant ML350 G4p Storage Server, HP ProLiant ML350 G5 Server, HP ProLiant ML350 G5 Storage Server, HP ProLiant ML350 G6 Server, HP ProLiant ML370 G3 Server, HP ProLiant ML370 G4 Server, HP ProLiant ML370 G4 Storage Server, HP ProLiant ML370 G5 Server, HP ProLiant ML370 G6 Server, HP ProLiant ML570 G3 Server, HP ProLiant ML570 G4 Server
Operating Systems Affected: Not Applicable
Software Affected: Not Applicable
Support Communication Cross Reference ID: IA04249852
©Copyright 2014 Hewlett-Packard Development Company, L.P.
Hewlett-Packard Company shall not be liable for technical or editorial errors or omissions contained herein. The information provided is provided "as is" without warranty of any kind. To the extent permitted by law, neither HP or its affiliates, subcontractors or suppliers will be liable for incidental,special or consequential damages including downtime cost; lost profits;damages relating to the procurement of substitute products or services; or damages for loss of data, or software restoration. The information in this document is subject to change without notice. Hewlett-Packard Company and the names of Hewlett-Packard products referenced herein are trademarks of Hewlett-Packard Company in the United States and other countries. Other product and company names mentioned herein may be trademarks of their respective owners.