Container Images

This page is the deep reference for the container image system: the three modes side by side, the prebuilt-base workflow, the custom-image contract, and the kinetic build-image command.

For a higher-level overview and the recommendation matrix on which mode to pick, start with Execution Modes.

Kinetic supports three container image modes that control how your remote execution environment is built and deployed. Choose the mode that best fits your workflow by setting the container_image parameter in the @kinetic.run() decorator.

Note

Expected timing:

• Bundled, cold (first run / dep change): ~2–5 minutes for the Cloud Build step.

• Bundled, warm (cached image): under a minute to schedule and start the pod.

• Prebuilt: 30–60 seconds for the base image pull (cached after first use on a node), plus the time to uv pip install your requirements.txt.

• Custom: a single image pull, then immediate execution. Cold pulls vary widely with image size and registry latency.

Warning

When not to use custom image mode: if you only need different Python packages, bundled or prebuilt are simpler and cheaper. Reach for custom images when you have non-Python system libraries (CUDA builds, C++ deps), corporate compliance requirements, or you genuinely want to manage the image lifecycle yourself.

Mode	container_image=	Build step	Dependencies installed
Bundled (default)	None or "bundled"	Cloud Build (cached by dep hash)	Baked into the image
Prebuilt	"prebuilt"	None	At pod startup via uv pip install
Custom	"<image-uri>"	None (you manage it)	Whatever is in your image

Bundled Mode (Default)

Bundled mode builds a custom container image via Cloud Build with all your dependencies baked in. The image is tagged by a hash of your dependencies, so unchanged dependencies reuse the cached image.

import kinetic

# Bundled mode — these are equivalent:
@kinetic.run(accelerator="tpu-v6e-8")
def train():
    ...

@kinetic.run(accelerator="tpu-v6e-8", container_image="bundled")
def train():
    ...

Tradeoffs

• Reproducible: The exact environment is frozen in the image.

• First-run cost: The initial build takes ~2-5 minutes. Subsequent runs with unchanged dependencies use the cached image and start within a few seconds.

• Good for: Production workloads, large dependency sets where you want to avoid per-run install overhead, or when you need a fully reproducible environment.

Prebuilt Mode

Prebuilt mode uses a pre-published base image that already contains the accelerator runtime (JAX, CUDA/TPU libraries) and core dependencies. Your project’s requirements.txt or pyproject.toml dependencies are installed at pod startup via uv pip install, so there is no Cloud Build step.

@kinetic.run(accelerator="tpu-v6e-8", container_image="prebuilt")
def train():
    ...

How it works

1. Kinetic resolves the base image from the image repository (see Custom prebuilt images below) using the accelerator category (cpu, gpu, or tpu) and the kinetic package version.

2. Your project dependencies are filtered (JAX packages are removed to avoid conflicts) and uploaded to GCS alongside your code.

3. At pod startup, the runner installs your dependencies with uv pip install before executing your function.

Tradeoffs

• Fast iteration: No build step means jobs start quickly.

• Startup cost: uv pip install runs on every job. For large dependency sets this adds time to each run.

• Good for: Most workflows, especially during development and experimentation.

Custom prebuilt images

By default, Kinetic pulls official base images from Docker Hub (kinetic/base-{category}:{version}). To use your own prebuilt images — for example, with additional system libraries or private packages — build and push them with the kinetic build-image command, then point Kinetic at your repository.

Build and push images:

kinetic build-image --repo us-docker.pkg.dev/my-project/kinetic-base

Then set the repository so Kinetic uses your images:

export KINETIC_BASE_IMAGE_REPO=us-docker.pkg.dev/my-project/kinetic-base

Or pass it directly to the decorator:

@kinetic.run(accelerator="l4", base_image_repo="us-docker.pkg.dev/my-project/kinetic-base")
def train():
    ...

See kinetic build-image for the full command reference.

Custom Image Mode

Provide a full container image URI to use your own image. Kinetic skips all build and dependency steps.

@kinetic.run(
    accelerator="tpu-v6e-8",
    container_image="us-docker.pkg.dev/my-project/kinetic/my-image:v1.0"
)
def train():
    ...

Requirements for custom images

Your custom image must:

1. Include cloudpickle, google-cloud-storage, and a compatible Python environment.

2. Include the necessary dependencies for your function.

3. Be accessible from the GKE nodes (e.g., Artifact Registry in the same GCP project, or a public registry).

When to use custom images

• Complex dependencies: Non-Python system libraries (CUDA builds, C++ libs) that aren’t in the default template.

• Corporate compliance: Base images vetted by your security or platform team.

• Full control: When you want to manage the entire image lifecycle yourself.

kinetic build-image

Build and push prebuilt base images to a Docker Hub or Artifact Registry repository. One image is built per accelerator category (cpu, gpu, tpu) using Cloud Build.

# Interactive mode — guides you through registry selection and setup
kinetic build-image

# Non-interactive with Artifact Registry
kinetic build-image \
  --repo us-docker.pkg.dev/my-project/kinetic-base \
  --project my-project \
  --yes

# Build only GPU and TPU images
kinetic build-image --repo myuser/kinetic --category gpu --category tpu

# Use a custom Dockerfile
kinetic build-image --repo myuser/kinetic --dockerfile ./Dockerfile.custom

# Specific version tag (default: kinetic package version)
kinetic build-image --repo myuser/kinetic --tag v2.0.0

Options

Option	Description
--repo	Image repository (Docker Hub or Artifact Registry). Omit to select interactively.
--category	Accelerator categories to build: cpu, gpu, tpu (default: all). Repeatable.
--tag	Image version tag (default: kinetic package version).
--dockerfile	Path to a custom Dockerfile. Must install uv, cloudpickle, google-cloud-storage, and COPY remote_runner.py to /app/.
--update-credentials	Re-enter Docker Hub credentials even if they already exist in Secret Manager.
--yes, -y	Skip confirmation prompt.
--project	GCP project ID (default: KINETIC_PROJECT).
--cluster	GKE cluster name (default: kinetic-cluster).

Registry support

• Docker Hub: Credentials are stored in GCP Secret Manager and used by Cloud Build during the push. The command prompts for your Docker Hub username and access token on first use.

• Artifact Registry: No additional credentials needed — the build service account authenticates automatically. The command prints the required gcloud setup commands for creating the repository and granting permissions.

Related pages

• Execution Modes — start here for the high-level mode-selection guidance.

• Dependencies — what gets discovered and what gets installed in each mode.

• Configuration — KINETIC_BASE_IMAGE_REPO and other relevant env vars.