Create a GPU Container Job in the web interface
Get shell access to whole GPUs by creating a GPU Container Job in the CosmicAC web interface.
Create a GPU Container Job to get shell access to one or more whole GPUs. The form has six sections, and you click Continue to move from one section to the next. For a description of each field in the form, see GPU Container Job configuration.
Prerequisites
Before you start, make sure that you have the following.
- A running CosmicAC deployment. See Set up CosmicAC.
- Access to the CosmicAC web interface.
- The team lead or platform administrator role if you plan to create a team storage volume.
- A deployment that serves volumes if you plan to create or attach a team storage volume. For more information, see Team storage volumes.
Steps
Open the new job form
In the left navigation, click Jobs, and then click New Job.
Select the job type
In the What kind of job? section, select GPU Container, and then click Continue.
Enter the basics
In the Basics section, enter a Job name. Use at least three lowercase letters, numbers, or hyphens.
In the Tags field, add at least one tag. To add a tag, type it, and then press Enter.
Click Continue.
Select the base image and disk
In the Image & access section, select a Base OS image. In the Disk (GB) field, select a preset or enter a value.
Click Continue.
Create a team storage volume (optional)
If you want to attach a team storage volume but your team doesn't have one available, create a volume in the Hardware section.
- Click New volume.
- In the New storage volume dialog, select a location in the Location list.
- In the Name field, enter a name that's unique in your team.
- In the Size field, select a preset or enter a whole number of GiB. The minimum size is 100 GiB.
- Click Create volume. The volume appears in the Team storage volume list with the status Provisioning.
- Wait until the status changes to Available. Provisioning takes a few minutes. CosmicAC shows a Volume ready notification when the volume is ready.
Attach a team storage volume (optional)
To keep the job's data after you delete the job, attach a team storage volume. You can attach a volume only when you create the job.
In the Team storage volume list, select a volume that has the status Available. If you created a volume in the preceding step, CosmicAC selects the volume for you.
CosmicAC sets Location to the volume's location and locks the list. To choose another location, clear the volume. For more information, see Use team storage volumes.
Select the hardware
- If you didn't attach a volume, select a location in the Location list.
- Select a GPU.
- Set the GPU count. Below the count, Available GPUs shows the number of free GPUs in that location.
- In the CUDA / driver list, select a driver.
- Click Continue.
Select the notification events
In the Notifications section, turn off each job lifecycle event that you don't want this job to report. CosmicAC turns on all four events by default. For more information about the events and their delivery, see Set up webhook notifications.
Click Continue.
Review and create the job
In the Review & launch section, check that the section reports Ready to create, and then click Create job. If the section reports issues, click Edit in the row that names the problem. Fix the problem, and then return to the Review & launch section.
CosmicAC opens the Jobs page and shows the Provisioning status dialog.
Open a shell in the job
Wait until the job's status is Active. To open an interactive shell in the job, follow the steps in Connect to a GPU Container Job.
Help and troubleshooting
Job stuck in Creating or Starting
If a job stays in Creating or Starting, check the status of its KubeVirt virtual machine instance (VMI).
-
Find the job's container ID.
- In the left navigation, click Jobs, and then click the job.
- Click the Shell access tab.
The tab shows the Container ID.
-
Find the VMI for the container.
CosmicAC creates one VMI for each container and names it
<container-id>-n0. A multi-node job has one VMI per node.From a machine with
kubectlaccess to your Kubernetes cluster, run the following command.kubectl get vmi -n <namespace>Replace
<namespace>with the namespace configured inK8S_NAMESPACE. -
Check the VMI status.
-
If the VMI is not Running, inspect its events.
kubectl describe vmi <container-id>-n0 -n <namespace> -
If the VMI is Running but the job stays in Creating or Starting, cosmicac-wrk-agent-instance cannot reach cosmicac-wrk-server-k8s-nvidia. These two components connect directly, and some cluster network configurations can block the connection.
To route the connection through a relay, see Set up a relay for CosmicAC.
-