Run Llama 3.2 1B on a Cloud GPU: September 2026 Guide

The earlier version of this article incorrectly named a model "Llama 2 1B." Meta's Llama 2 model card describes a family beginning at 7B parameters. This walkthrough uses the published Llama 3.2 1B Instruct model.

Reviewed September 26, 2026. This is a small-model inference example, not a claim that Llama 3.2 is the newest model or the best choice for every task. The code was reviewed against the linked documentation; no GPU inference benchmark was run for this update.

1. Prepare a supported environment

Choose a Linux GPU instance with a working NVIDIA driver and enough disk space for the model, packages and cache. Use a current provider-supported PyTorch image or follow the official PyTorch installation selector for the appropriate CUDA build.

Run:

nvidia-smi
python -c "import torch; print(torch.__version__); print(torch.cuda.is_available())"

The second command should report CUDA availability as True before you use the GPU-only example below. If PyTorch is missing or cannot see CUDA, resolve the image/driver/package setup first.

Create an isolated environment for your chosen stack. Install the supporting libraries alongside the compatible PyTorch installation:

python -m pip install --upgrade transformers accelerate huggingface_hub

After the environment works, record its package versions for reproduction. An unpinned installation command can resolve differently on a later date.

2. Obtain model access

Use the model page to request any required access and review Meta's model license. Authenticate with the Hugging Face CLI using an account that has access:

hf auth login

Use the interactive prompt; do not paste a token into source files or screenshots. The Hugging Face CLI guide describes authentication and account checks.

A 401 or 403 response may indicate missing authentication or model access rather than a GPU problem.

3. Run one bounded inference request

Save this example as run_llama.py:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "meta-llama/Llama-3.2-1B-Instruct"

if not torch.cuda.is_available():
    raise RuntimeError("A CUDA-enabled PyTorch environment is required.")

precision = (
    torch.bfloat16
    if torch.cuda.is_bf16_supported()
    else torch.float16
)

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    dtype=precision,
    device_map={"": 0},
)
model.eval()

conversation = [
    {"role": "user", "content": "Explain a GPU in two short sentences."}
]
batch = tokenizer.apply_chat_template(
    conversation,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
).to("cuda:0")

with torch.inference_mode():
    result = model.generate(
        **batch,
        max_new_tokens=128,
        do_sample=False,
        pad_token_id=tokenizer.eos_token_id,
    )

new_tokens = result[0, batch["input_ids"].shape[-1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

Run it with:

python run_llama.py

The example loads the instruction-tuned checkpoint, formats the conversation with its chat template and prints only the newly generated tokens. The token cap keeps this example bounded. The exact model repository and input format matter; substituting an unrelated model can require changes.

4. Measure before choosing a larger instance

Download and initialization time are different from steady-state generation time. Record them separately. Test your intended prompt lengths, output lengths and concurrency rather than treating one short answer as a production benchmark.

Model weights are only part of GPU memory use. Runtime allocations and the KV cache add overhead, and longer contexts or batches can increase demand. A successful single request does not establish how many concurrent users the instance can serve.

Use the GPU comparison pages to shortlist hardware, then compare measured latency, throughput and full-instance cost. This article does not promise a particular response time or daily serving budget.

5. Treat API serving as a separate step

The example is a local inference script. It does not implement authentication, request limits, queueing, monitoring or multi-user serving. Evaluate a serving engine and test its compatibility with your chosen GPU and model before exposing a network endpoint.

For a cloud trial, keep the model cache on persistent storage when appropriate and check what continues billing after compute stops. Shut down unused compute and review storage charges.

For deployment budgeting, use the provider directory, current prices and our pricing methodology. Affiliate compensation does not determine those price rankings.