Create a vLLM Managed Inference Job in the web interface

Open the new job form

In the left navigation, click Jobs, and then click New Job.

Select the job type

In the What kind of job? section, select Managed Inference, and then click Continue.

Enter the basics

In the Basics section, enter a Job name. Use at least three lowercase letters, numbers, or hyphens.

In the Tags field, add at least one tag. To add a tag, type it, and then press Enter.

Click Continue.

Select a model

In the Model to serve section, select a vLLM model in the Model list. CosmicAC prefills the Serving configuration from the model's recommended configuration.

Name the endpoint

Under Endpoint, enter an Endpoint name. Use a name that no other Managed Inference Job in the deployment uses, including a failed job that you haven't deleted.

Below the name, Will be reachable at shows the endpoint URL.

Edit the serving configuration (optional)

Under Serving configuration, edit the values that you want to change. For a description of each field, see vLLM Managed Inference Job configuration.

To serve a different runtime image, update the recommended model configuration.

Set the root disk and environment variables

Under Instance resources, in the Root disk (GB) field, select a preset or enter a value.

Under Environment variables, review the prefilled variables. You can make the following changes.

• To add a variable, click Add variable, and then enter a Name and Value.
• To remove a variable, click the X beside the variable.
• To make the model weight download faster, add HF_TOKEN with a Hugging Face read token as the value.
• To serve a gated model, add HF_TOKEN with a token from a Hugging Face account that accepted the model's license. Without the token, the job fails.

For the variable names that CosmicAC reads, see vLLM serving options.

Click Continue.

Select the hardware

1. In the Hardware section, select a location in the Location list.
2. Select a GPU.
3. Set the GPU count. Below the count, Recommended for this model shows the model's GPU count. On one node shows the most free GPUs on any single node, and In this region shows the total free GPUs in the location.
4. In the CUDA / driver list, select a driver.
5. Set Replicas to 1, 2, or 4. Total shows the number of GPUs that the job claims.
6. Click Continue.

For more information about how replicas work, see Replicas.

Select the notification events

In the Notifications section, turn off each job lifecycle event that you don't want this job to report. CosmicAC turns on all four events by default. For more information about the events and their delivery, see Set up webhook notifications.

Click Continue.

Review and create the job

In the Review & launch section, check that the section reports Ready to create, and then click Create job. If the section reports issues, click Edit in the row that names the problem. Fix the problem, and then return to the Review & launch section.

If the form shows a Job creation warning, check the values that the warning names. A job with those values can fail to start. To create the job anyway, select Create anyway with this configuration, and then click Create job. To change which values cause a warning, see Set the job creation warnings.

Open the endpoint

In the Provisioning status dialog, click Open job. When the job's status is Active, click the Endpoint tab to view the endpoint URL. To send a request to the endpoint, follow the steps in Connect to a vLLM Managed Inference endpoint.