Massive parallel management with iLOrest on Linux

François Donzé - Technical consultant, HPE

January, 2022

Update: August, 2026

Thanks to HPE having provided the iLOrest Python API, an entirely new avenue has been opened to developers for managing their servers. It allows them to add in their favorite deployment tools (i.e. Ansible or other management tools, like HPE OneView (as described in several articles found here, in the HPE DEV Blog).

But what if you are not a Python geek? What if you prefer managing your servers the old way with bash / grep / awk / jq / curl / wget? In that case, you can loop over the list of your iLO IP addresses and use curl / wget for getting and setting Redfish parameters.

However, this loop approach brings out two fundamental problems: 1) Using curl and wget in management scripts implies the creation of smart crawling functions (as explained in this article), 2) a sequential management with loops is fine for taking care of five or ten servers, but not more. A parallel approach should be considered for an important number of managed servers.

This blog post explains how to manage many servers in parallel using bash scripts without the need of implementing schema crawlers, thus allowing a long-term stability to the scripts.

To achieve this goal, you will combine a parallel task launcher (namely the ClusterShell and/or the Parallel Distributed Shell) with iLOrest (former hprest) the HPE RESTful Interface Tool in a more detailed and slightly different manner than what you can read already in the user guide.

The Parallel Distributed Shell (pdsh) and the ClusterShell (clush) are both part of the Extra Package for Enterprise Linux (EPEL) repository and they give the ability to perform local or remote commands in parallel. Pdsh came first as a replacement of an IBM utility (DSH) with Google being the main contributor. However, Google pulled out from this project in 2013.

The ClusterShell is an interesting alternative to pdsh for many reasons. First the basic syntax and most of the switches, options and arguments are identical to pdsh. Hence, moving from pdsh to clush is easy. Additionally, clush provides more flexibility for selecting the remote nodes you want to execute commands against or on. It is associated with the nodeset(1) command for computing advanced nodeset operations like expand, fold, group, etc. Another advantage of clush over pdsh is that it is written in Python and can be used in Python scripts whenever you are ready to use this language.

For the purposes of this article, I will use ClusterShell only.

For those systems running Red Hat or a similar distribution, you need to mention the two packages containing the code and the eco-system:

sudo dnf install python3-clustershell clustershell

On Ubuntu systems, just mention the clustershell parameter on the command line to install both packages:

sudo apt install clustershell

Here are basic examples to illustrate the launch of several commands in parallel on the local node and then one command on several remote nodes.

If you want to perform a parallel ping toward node1, node2 and node3 type:

clush -R exec -w node[1-3] ping %h

-R exec specifies a local execution of the ping command toward the list of hosts: node[1-3]. %h is substituted with node1, node2 and node3 for each ping command. When you hit the return key, your system forks in parallel the following 3 tasks:

ping node1 
ping node2 
ping node3

Imagine now that you want the root user of node1, node2 and node4 to ping a remote gateway called aremotegateway. With the above syntax you can issue:

clush --nostdin -R exec -w node[1-2,4] ssh --user=root %h "ping aremotegateway"

However, EPEL provides clush compiled with an ssh (default) module, allowing the following simplified syntax:

clush --nostdin -w node[1-2,4] --user=root "ping aremotegateway"

Since the -R switch is not present, the ssh default module is used toward node[1-2,4] and the ping command is executed on this list or remote nodes.

The list of remote nodes following the -w switch can be the output of a nodeset command taking its input from a node set configuration file. The possible locations of the configuration file depends on the groups.conf(5) file. For this article, I populated the default file located in my home directory (~/.local/etc/clustershell/groups.d/local.cfg).

Here is one example showing the power of this clush companion command.

Imagine you define two groups of iLOs in the nodeset configuration file:

ilo-sales: ilo-sales[1-10]
ilo-marketing: ilo-foo,ilo-bar,ilo-test

The ilo-sales group contains ten iLO names and ilo-marketing group contains three. If you want to select both groups and exclude node ilo-test, you can do it with:

nodeset --fold @ilo-sales,@ilo-marketing  --exclude ilo-test

Figure 1: nodeset exclude example

You can now combine the nodeset command with clush to issue a parallel ping toward this list with:

clush --nostdin -R exec -w $(nodeset -f @ilo-sales,@ilo-marketing -x ilo-test) ping -c 2 %h 

As discussed in this blog post, iLOrest offers both out-of-band and in-band management. In the first form, you need to log into a remote iLO with the credentials of an iLO user. The following sequence of commands retrieves the Bios version of a remote server using an out-of-band management:

ilorest login ilo-IP -u ilo-user -p ilo-password
ilorest select computersystem. 
ilorest ls oem/hpe/bios/current # On Gen9, replace with bios/current
ilorest logout

Performing in-band management with iLOrest means that you need to first access the operating system before launching the local ilorest. On iLO 5 and iLO 6 based systems configured with the default Production security state , if you logged in the OS as a privileged user (root or Administrator), you don't need to supply any username/password credentials to access the underlying iLO. If the iLO 5 or iLO 6 of your server is in HighSecurityState or higher (i.e. FIPS, CSNA), you will need to provide iLO credentials to access the underlying iLO.

If your server is an iLO 7 based system (or later), in-band management is handled differently: The lowest security state of those servers corresponds to the HighSecurityState of iLO 5 and 6. Then, the CHIF communication path between the OS and iLO does not exist anymore. The only communication path is through the host interface, also known as vNIC. This communication path requires iLO credentials to be provided to access the underlying iLO, except for some HPE specific applications (i.e. AMS, SUM iSUT and iLOrest) which are allowed to access the underlying iLO without credentials when they have a dedicated application account properly registered in iLO.

The following example shows how to retrieve the Bios version of a remote server using an in-band management from an iLO 5, iLO 6 in Production security state or an iLO 7 with a registered application account or iLOrest.

ssh root@server ilorest login
ssh root@server  ilorest select computersystem.
ssh root@server ilorest ls oem/hpe/bios/current # On Gen9 replace with bios/current
ssh root@server ilorest logout

All of this is awesome except that in both examples (out-of-band and in-band), a cache directory location is not explicitly specified. Hence, you will be kicked out if a concurrent iLOrest session is opened using the default cache directory as shown in the following picture.

The picture below shows the following sequence of events: (1) you first log into ilo-lab1. (2) Then you select the Bios. type (select HpBios. on Gen9 servers) and (3) verify that it has been selected.

(4) In the other terminal session and from the same server, log into ilo-lab2 without selecting any type. This second session clears off the iLOrest cache including the selection we performed in the first terminal. As a result, (5) asking for the resource type selection returns an error.

Figure 2: Demonstrating iLOrest local cache

As a consequence (and to be safe), you should always specify a specific cache directory location for your iLOrest sessions which will avoid such contention.

With a local temporary cache directory, the out-of-band example becomes:

#!/usr/bin/env bash

u="ilo-user"
p="ilo-password"
CACHEDIR=$(mktemp -d "/tmp/ilorest_cache_XXXX")
iLOrest="ilorest --nologo --logdir /var/log --cache-dir=${CACHEDIR}"
$iLOrest login ilo-IP -u $u -p $p
$iLOrest select computersystem. 
$iLOrest ls oem/hpe/bios/current # On Gen9, replace with bios/current
$iLOrest logout
rm -rf ${CACHEDIR}

With a specific cache directory, the in-band example becomes as the following. Loggin as root on the remote server allows you to avoid providing credentials to the ilorest commands against iLO 5 and iLO 6 in Production security state or iLO 7 (and later) with a registered iLOrest application account.

#!/usr/bin/env bash

server="serverIp"
CACHEDIR=$(mktemp --dry-run --directory "/tmp/ilorest_cache_XXXX")
iLOrest="ilorest --nologo --logdir /var/log  --cache-dir=${CACHEDIR}"
ssh root@$server "$iLOrest select computersystem."
ssh root@$server "$iLOrest ls oem/hpe/bios/current"  # On Gen9, replace with bios/current

# Retrieve the SessionCollection from the remote server and display it in JSON format
# In-band sessions URIs start with "systemadministrator" on iLO 5 and iLO 6 and "ilorest*" on iLO 7 and later.
ssh root@$server "$iLOrest ls --select SessionCollection. --json"

# It is always a good practice to logout from the iLO when you are done with your in-band management.
ssh root@$server "$iLOrest logout"

# Cleanup the temporary cache directory on the remote server
ssh root@$server "rm -rf ${CACHEDIR}"

You are now ready to send out-of-band iLOrest commands in parallel toward a set of iLOs sharing common administrator credentials.

This example applies a Bios configuration file to a list of servers. The Bios configuration has been created and customized in an ASCII/json file called BiosConfig.json using the ilorest save --select Bios. command. The list of target servers to configure is represented by a group of iLOs defined in the nodeset configuration file as explained earlier. Note that, for a performance reason, I grouped the type selection (Bios.) with the login process:

#!/usr/bin/env bash

read -p "Enter iLO admin username: " u
read -s -p "Enter iLO admin password (No echo on screen): " p

CLUSH="clush --nostdin -R exec"
iLOrest="ilorest --nologo --cache-dir=cache-%h"
ILO_LIST=$(nodeset -f @myilos)
$CLUSH -w $ILO_LIST $iLOrest login %h -u $u -p $p --selector=Bios. # On gen9, selector is HpBios
$CLUSH -w $ILO_LIST $iLOrest load -f BiosConfig.json
$CLUSH  -w $ILO_LIST $iLOrest logout

# Cleanup the temporary cache directories on the local management station
for n in $(echo $ILO_LIST | tr ',' ' ')  ; do rm -rf cache-${n} ; done

I personally successfully tested this example against 24 servers. I am pretty optimistic that it works as well for a larger number of servers, although a risk of bumping into Redfish session limits exists.

In this paragraph I'll present two use cases of in-band management: the first one sends iLOrest commands in parallel toward a set of remote servers. Those commands are executed on the remote servers by the ilorest application installed in the operating system (Figure 3) of remote servers.

The second use case starts several ilorest commands in parallel, all of them targeting the underlying local iLO (Figure 4).

Offloading iLOrest processing onto remote servers may be required in secure environments where iLOs are physically disconnected from the network.

In this case, in-band management requires an operating environment up and running with sshd and ilorest installed and configured on all the managed nodes. This operating environment could be a customized Live-CD booted via the iLO virtual Drive facility as explained in a previous article.

Figure 3: iLOrest sessions on remote servers

The in-band version of our Bios configuration requires one more step compared to the out-of-band method: the copy of the JSON configuration file onto all the managed nodes. This operation is simple to perform with the clush --copy and clush --dest options dedicated to that purpose. Moreover, this copy will not impact the overall process time because the JSON configuration file to transfer is a small ASCII file:

In the following example, the use of a temporary cache directory is a good practice to avoid cache contention in case other users start their own iLOrest sessions on the same remote servers. In addition, return codes are not handled to keep the example simple. In a production environment, you should always check the return codes of each command to ensure that the operation was successful.

#!/usr/bin/env bash

# Generate a temporary cache directory to be used by iLOrest commands on the remote servers.
CACHEDIR=$(mktemp --dry-run --directory "/tmp/ilorest_cache_XXXX")

# Define the `clush` command to be processed by user `root` to avoid providing credentials to create iLOrest sessions on the remote servers.
# It is possible to communicate with underlying iLO without credentials when connected as root/Administrator
# on iLO 5 and iLO 6 based servers with iLO in `Production` security state.
# On iLO 7 (and later) it is required to have a registered iLOrest application account and be connected as a privileged user (root/Administrator).
CLUSH="clush --nostdin --user=root "

# Specify `--logdir` to avoid the creation of a log folder in an unwanted location on the remote servers.
iLOrest="ilorest --nologo --logdir /var/log --cache-dir=${CACHEDIR}"

# Generate the list of remote servers to manage in parallel.
SRV_LIST=$(nodeset -f "@lab")

# Copy the BiosConfig.json file to all the remote servers
$CLUSH -w $SRV_LIST --copy BiosConfig.json --dest /tmp/. 

# In-band login to the iLO with no credentials and load the BiosConfig.json file.
$CLUSH -w $SRV_LIST "$iLOrest login --selector=Bios." # selector is HpBios. on Gen9 servers
$CLUSH -w $SRV_LIST "$iLOrest load -f /tmp/BiosConfig.json "

# Retrieve the SessionCollection from all the remote servers and display it in JSON format.
# In-band sessions URIs start with "systemadministrator" on iLO 5 and iLO 6 and "ilorest*" on iLO 7 and later.
$CLUSH -w $SRV_LIST "$iLOrest ls --select SessionCollection. --json"

# Logout from the iLO on all the remote servers.
# This is a good practice to avoid saturating the iLO with too many concurrent sessions.
$CLUSH -w $SRV_LIST "$iLOrest logout"

# Remove the temporary cache directory and Bios configuration file on all the remote servers
$CLUSH -w $SRV_LIST "rm -rf ${CACHEDIR} /tmp/BiosConfig.json"

In this second use case, multiple ilorest sessions are started concurrently on the local host that target the underlying iLO (Figure 4). This use case could happen when several system managers are working on the same server and they want to perform in-band management in parallel.

Or, you may be the only one managing a set of servers and, for some reasons, you opened several ssh sessions, each in a different terminal toward the same server and you want to perform in-band management in parallel from those sessions.

Figure 4: iLOrest sessions started concurrently on local host

Although the following example is similar to the ones presented in the Using HPE iLOrest documentation section, it is always good to provide redundant information on important topics like this one.

The key points of this example are the following:

• Start of multiple ilorest sessions in parallel toward the underlying iLO.
• The number of concurrent sessions is limited to 8 (limit recommended by HPE at the time of writing).
• Each session is started with the login iLOrest command.
• An empiric sleep time of a few seconds is added between starting each session to avoid overwhelming the system TPM or the CHIF.
• All commands (including login and logout) are using a specific cache directory to avoid cache contention.
• Each session is performing get and set operations to simulate a real-world management operation.
• Each session is ended with a logout operation to formally close the session in iLO and clean the iLOest local cache.
• A cleanup of the iLOrest sessions is performed at the end of the script in case some sessions are still opened in iLO.

#!/usr/bin/env bash

# IMPORTANT: This script must be used for training purposes only. 
#            It is intended to pin-point the best practices 
#            when performing in-band management of iLO based
#            HPE servers.
#            This script should not be run on production systems.
#            If you don't respect those good practices, you may
#            saturate the iLO number of sessions. 



# This script simulates in-band parallel jobs. A job is composed of the following operations:
#
# * ilorest login   
# * ilorest get and set properties
# * ilorest logout

# Jobs are lauched in parallel. They modify the same `AssetTag` property. 
# The resulting property value is not important at all. This modification
# is just for simulating a real-world management operation.

# SUMMARY of in-band good practices:
#
# * Use a specific cache directory for each and every `ilorest` command in the job,
#   including login and logout.
# * Limit the number of iLOrest opened sessions to 8, regardless the iLO generation.
# * Perform a formal `login` operation to create a session and provide "some time"
#   before starting another iLOrest session/job
# * Always logout when all job commands are performed.

# ==============================================================================

# iLO credentials for session cleanup in case of saturated in-band sessions
read -p "Enter iLO admin username: " user
read -s -p "Enter iLO admin password (No echo on screen): " password

# The following `ilorest` variable definition adds:
# * `--nologo` to limit printing redundant and useless output.
# * `--logdir` to centralize logging messages
#   and avoid the creation a of log folder in un-wanted places.
ilorest="ilorest --nologo --logdir /var/log" 

# Cleanup AssetTag property to avoid reaching property max limit during tests.
$ilorest login  &>/dev/null  # Always good practice to login properly
$ilorest set AssetTag=' ' --select ComputerSystem.  --commit &>/dev/null

# Always logout when job is done in order to disconnect session properly
# in iLO and clean local cache.
$ilorest logout &>/dev/null
echo

run_ilorest_job() {
    j=$1
    echo "Processing Job${j}" 
    
    # Temporary cache directory creation
    CACHEDIR=$(mktemp --directory "/tmp/ilorest_cache_${j}_XXXX")
    
    # Good practice is to login before sending any in-band commands
    $ilorest --cache-dir=${CACHEDIR} login &>/dev/null

    # The goal of the get and set operations below is to simulate real
    # management operations.
    # Retrieving AssetTag property from fresh cache.
    AssetTag=$($ilorest --cache-dir=${CACHEDIR} \
          get AssetTag --select ComputerSystem. --refresh | awk -F= '{print $NF}' | grep -v Done)
    
    # Appending the current Job in the AssetTag property
    if [ $j -eq 1 ] ; then
       AssetTag="J${j}"
    else
       AssetTag+="J${j}"
    fi

    $ilorest --cache-dir=${CACHEDIR} \
          set AssetTag="${AssetTag}" --select ComputerSystem.  --commit #&>/dev/null
    
    # The following logout command is required to avoid the saturation of iLO sessions.
    # That would result in errors (i.e. ilorest error 7), and potentially require an iLO reset to recover.
    echo "Job: $j log out from iLO"
    $ilorest --cache-dir=${CACHEDIR} logout &>/dev/null

    # Cleanup the temporary job's cache directory.
    rm -rf ${CACHEDIR}

    echo "Job $j completed"
    echo
}

# Start jobs in background

# HPE recommends a number of parallel in-band session below or equal to 8 to ensure you
# never reach the maximum in-band session limit. This limitation is valid for
# iLO 5, 6 7 and later.

# By launching more than 8 jobs in parallel, the probability to trigger errors is very high,
# especially if you don't logout properly when jobs are done.
# Feel free to decrease the number of parallel jobs and study the behaviour
# proper logout or not.

i=8
for j in $(seq 1 $i) ; do
    # Start jobs in background
    run_ilorest_job $j &
    # Allow some time before starting the next job.
    # Launching jobs too quickly may generate contention
    # at TPM level on iLO 7 and later generations.
    sleep 3
    echo
done

# Wait for background jobs to finish.
wait

# Cleanup in-band iLOrest sessions if needed.
# This step should not be necessary if you respect 
# the good in-band management practices listed above.

echo "Cleanup of in-band iLOrest sessions"
$ilorest login -u $user -p $password

# iLO generation detection.
iLOGen=$($ilorest detectilo | awk '/Type: / {print $NF}')

# The following case is necesary, as `SessionName` and `UserType` property
# have different values depending on the iLO generation.

case $iLOGen in
    7|8)
        SessionName="ilorest0"
        UserTag="Host Application"
        ;;
   5|6)
       SessionName="systemadministrator0"
       UserTag="RESTHost"
        ;;
   *)
     echo "iLO generation not recognized"
     exit 1
esac

# Retrieve in-band session locations
HostUserList=$($ilorest list --select SessionCollection --json | \
      jq -r '.Members[] | ."@odata.id"'  | grep $SessionName)

# Delete in-band sessions
for u in $HostUserList ; do
    $ilorest select Session. --refresh
    UT=$($ilorest get Oem/Hpe/UserTag --filter "@odata.id=${u}" --json )
    if [[ "$UT" =~ "$UserTag" ]] ; then
        echo "Deleting ${u}"
        $ilorest rawdelete "$u"
    fi
done

$ilorest logout 

In Linux infrastructures, iLOrest combined to the ClusterShell offers easy out-of-band and in-band management scripting possibilities for system administrators who prefer using Bash to pure Python. Of course, there are pros and cons to both methods. Out-of-band iLOrest management does not require a customized operating environment running on the managed servers. In-band management offers the possibility to physically disconnect iLOs to decrease the network footprint.

In Windows environments, system managers should be able to perform similar efficient parallel configuration of their servers using PowerShell.

In addition to presenting out-of-band parallel management, this article also presents in-band parallel management with iLOrest and insists on the importance of starting a limited number of sessions as well as using a specific cache directory for each iLOrest sessions and commands. It also emphasizes the importance of logging out from iLO when the management operation is completed to avoid saturating the iLO with too many concurrent sessions.

And don't forget to check out some of my other blog posts on the HPE Developer portal to learn more about Redfish tips and tricks.