Omnia

Ansible playbook-based tools for deploying Slurm and Kubernetes clusters for High Performance Computing, Machine Learning, Deep Learning, and High-Performance Data Analytics

This project is maintained by dellhpc

Install Omnia

The following sections provide details on installing omnia.yml using CLI.

To install the Omnia control plane and manage workloads on your cluster using the Omnia control plane, see Install the Omnia Control Plane and Monitor Kubernetes and Slurm for more information.

Steps to install Omnia using CLI

  1. Clone the Omnia repository:
    git clone https://github.com/dellhpc/omnia.git 
    

Note: After the Omnia repository is cloned, a folder named omnia is created. Ensure that you do not rename this folder.

  1. Change the directory to omnia: cd omnia

  2. In the omnia_config.yml file, provide the required details (Check the parameter guide for more information).

    Note: Without the login node, Slurm jobs can be scheduled only through the manager node.

  3. Create an inventory file in the omnia folder. Add login node IP address under the [login_node] group, manager node IP address under the [manager] group, compute node IP addresses under the [compute] group, and NFS node IP address under the [nfs_node] group. A template file named INVENTORY is provided in the omnia\docs folder.
    **Note**:  * Omnia checks for red hat subscription being enabled on RedHat nodes as a pre-requisite. Check out [how to enable Red Hat subscription here](/omnia/Installation_Guides/ENABLING_OMNIA_FEATURES.html#red-hat-subscription). Not having Red Hat subscription enabled on the manager node will cause `omnia.yml` to fail. If compute nodes do not have Red Hat subscription enabled, `omnia.yml` will skip the node entirely. * Ensure that all the four groups (login_node, manager, compute, nfs_node) are present in the template, even if the IP addresses are not updated under login_node and nfs_node groups.
    
  4. To install Omnia:
Leap OS CentOS, Rocky, Red Hat
ansible-playbook omnia.yml -i inventory -e 'ansible_python_interpreter=/usr/bin/python3' ansible-playbook omnia.yml -i inventory

Note: Omnia creates a log file which is available at: /var/log/omnia.log.

  1. By default, no skip tags are selected, and both Kubernetes and Slurm will be deployed.

To skip the installation of Kubernetes, enter:
ansible-playbook omnia.yml -i inventory --skip-tags "kubernetes"

To skip the installation of Slurm, enter:
ansible-playbook omnia.yml -i inventory --skip-tags "slurm"

Note: If only Slurm is being installed on the cluster, docker credentials are not required.

Warning: LMOD and LUA are installed with Slurm when running omnia.yml. If LMOD and LUA are required, do not use the skip Slurm tag.

To skip the NFS client setup, enter the following command to skip the k8s_nfs_client_setup role of Kubernetes:
ansible-playbook omnia.yml -i inventory --skip-tags "nfs_client"

The default path of the Ansible configuration file is `/etc/ansible/`. If the file is not present in the default path, then edit the `ansible_config_file_path` variable to update the configuration path.
  1. To provide passwords for mariaDB Database (for Slurm accounting), Kubernetes Pod Network CIDR, and Kubernetes CNI, edit the omnia_config.yml file.

    Note:

    • Supported values for Kubernetes CNI are calico and flannel. The default value of CNI considered by Omnia is calico.
    • The default value of Kubernetes Pod Network CIDR is 10.244.0.0/16. If 10.244.0.0/16 is already in use within your network, select a different Pod Network CIDR. For more information, see https://docs.projectcalico.org/getting-started/kubernetes/quickstart.
    • If you want to view or edit the omnia_config.yml file, run the following command:
      • ansible-vault view omnia_config.yml --vault-password-file .omnia_vault_key – To view the file.
      • ansible-vault edit omnia_config.yml --vault-password-file .omnia_vault_key – To edit the file.
    • It is suggested that you use the ansible-vault view or edit commands and that you do not use the ansible-vault decrypt or encrypt commands. If you have used the ansible-vault decrypt or encrypt commands, provide 644 permission to omnia_config.yml.

Omnia considers slurm as the default username for MariaDB.

Kubernetes roles

The following kubernetes roles are provided by Omnia when omnia.yml file is run:

Slurm roles

The following Slurm roles are provided by Omnia when omnia.yml file is run:

Login node roles

To enable the login node, the login_node_required variable must be set to “true” in the omnia_config.yml file.

Note: If LeapOS is being deployed, login_common and login_server roles will be skipped.
To skip the installation of:

  • The login node: In the omnia_config.yml file, set the login_node_required variable to “false”.
  • The FreeIPA server and client: Use --skip-tags freeipa while executing the omnia.yml file.

Installing JupyterHub and Kubeflow playbooks

If you want to install JupyterHub and Kubeflow playbooks, you have to first install the JupyterHub playbook and then install the Kubeflow playbook.

Commands to install JupyterHub and Kubeflow:

Note: When the Internet connectivity is unstable or slow, it may take more time to pull the images to create the Kubeflow containers. If the time limit is exceeded, the Apply Kubeflow configurations task may fail. To resolve this issue, you must redeploy Kubernetes cluster and reinstall Kubeflow by completing the following steps:

  • Format the OS on manager and compute nodes.
  • In the omnia_config.yml file, change the k8s_cni variable value from calico to flannel.
  • Run the Kubernetes and Kubeflow playbooks.

Add a new compute node to the cluster

To update the INVENTORY file present in omnia directory with the new node IP address under the compute group. Ensure the other nodes which are already a part of the cluster are also present in the compute group along with the new node. Then, run omnia.yml to add the new node to the cluster and update the configurations of the manager node.

Using BeeGFS on Clusters

BeeGFS is a hardware-independent POSIX parallel file system (a.k.a. Software-defined Parallel Storage) developed with a strong focus on performance and designed for ease of use, simple installation, and management. BeeGFS is created on an Available Source development model (source code is publicly available), offering a self-supported Community Edition and a fully supported Enterprise Edition with additional features and functionalities. BeeGFS is designed for all performance-oriented environments including HPC, AI and Deep Learning, Media & Entertainment, Life Sciences, and Oil & Gas (to name a few). BeeGFS Structure

Once all the pre-requisites are met, run omnia.yml to set up BeeGFS.