GitLab CI/CD Setup

Source: Anshin SRE Team — Infrastructure knowledge base, Rev 1.0, 2026-01-24

This document covers CI/CD pipeline setup for the Proxmox K3s cluster using the GitLab instance at https://gitlab.anshinhealth.net.

Branch Strategy

Branch	CI/CD Behavior	Purpose
`dev`	✅ Build + Deploy	Testing — all new code goes here first
`main`	❌ No CI/CD	Stable archive — merge from dev after testing
Feature branches	❌ No CI/CD	Merge to dev for build/deploy

Prime Directive — No Unauthorized Pushes

Never push to dev without explicit user permission. Only one pipeline should run at a time on the single CI/CD runner. Always ask before pushing.

Infrastructure

GitLab Instance

Item	Value
URL	`https://gitlab.anshinhealth.net`
Groups	`engineering/` (all repos)
Container Registry	`registry.anshinhealth.net:443`

GitLab Runners

ID	Name	Tags	Notes
2	Proxmox K8s Runner	`kubernetes`, `anshin-dev`, `spl-erpnext`	Main K8s runner — all builds and deploys
3	Website Runner	—	Website builds only

Existing Projects

Project	ID	K8s Namespace	Notes
anshin-platform-wsl	21	`anshin-dev-svc`	Multi-service monorepo
payload-mcp	29	`mcp-servers`	HTTP MCP
frappe-mcp	34	`mcp-servers`	HTTP MCP
anshin-mcp	35	`mcp-servers`	HTTP MCP
mcp-gateway	37	`mcp-servers`	MCP Gateway

Pipeline Architecture

┌─────────────────────────────────────────────────────────────────┐
│                     Push to 'dev' branch                         │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  BUILD STAGE (moby/buildkit:v0.12.5-rootless)                   │
│                                                                  │
│  1. Configure registry auth (~/.docker/config.json)             │
│  2. Build image with buildctl-daemonless.sh                     │
│  3. Push to registry.anshinhealth.net:443                       │
│  4. Tag as :$CI_COMMIT_SHORT_SHA and :latest                    │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  DEPLOY STAGE (bitnami/kubectl:latest)                          │
│                                                                  │
│  1. kubectl set image deployment/... (update image tag)         │
│  2. kubectl rollout status (wait up to 5 minutes)               │
└─────────────────────────────────────────────────────────────────┘

Setup Steps for a New Project

Step 1: Enable the K8s Runner

The Proxmox K8s Runner (ID: 2) must be explicitly enabled for each new project.

Via API:

curl -s --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.anshinhealth.net/api/v4/projects/<PROJECT_ID>/runners" \
  --form "runner_id=2"

# Verify runner is assigned
curl -s --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.anshinhealth.net/api/v4/projects/<PROJECT_ID>/runners" | jq '.'

Via UI: Project → Settings → CI/CD → Runners → enable "Proxmox K8s Runner"

Step 2: Configure CI/CD Variables

The following variables must be set at the project level:

Variable	Value	Protected	Masked	Description
`CI_REGISTRY`	`registry.anshinhealth.net:443`	No	No	Docker registry URL
`CI_REGISTRY_USER`	`gitlab-ci-token`	No	No	Registry username
`CI_REGISTRY_PASSWORD`	`<gitlab-token>`	No	Yes	GitLab PAT (api + write_repository scopes)
`KUBECONFIG_CONTENT`	`<base64-kubeconfig>`	Yes	No	Optional — for explicit kubeconfig

# Add CI_REGISTRY
curl -s --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  --header "Content-Type: application/x-www-form-urlencoded" \
  --data-urlencode "key=CI_REGISTRY" \
  --data-urlencode "value=registry.anshinhealth.net:443" \
  "https://gitlab.anshinhealth.net/api/v4/projects/<PROJECT_ID>/variables"

# Add CI_REGISTRY_USER
curl -s --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  --header "Content-Type: application/x-www-form-urlencoded" \
  --data-urlencode "key=CI_REGISTRY_USER" \
  --data-urlencode "value=gitlab-ci-token" \
  "https://gitlab.anshinhealth.net/api/v4/projects/<PROJECT_ID>/variables"

# Add CI_REGISTRY_PASSWORD (masked)
curl -s --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  --header "Content-Type: application/x-www-form-urlencoded" \
  --data-urlencode "key=CI_REGISTRY_PASSWORD" \
  --data-urlencode "value=<your-gitlab-token>" \
  --data-urlencode "masked=true" \
  "https://gitlab.anshinhealth.net/api/v4/projects/<PROJECT_ID>/variables"

# Verify variables are set
curl -s --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.anshinhealth.net/api/v4/projects/<PROJECT_ID>/variables" | jq '.[] | {key}'

Step 3: Create Kubernetes Resources

Before CI/CD can deploy, K8s resources must exist in the cluster:

# Create namespace
kubectl create namespace <your-namespace>

# Apply base manifests
kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml
kubectl apply -f k8s/ingress.yaml

# Verify or create the registry pull secret
kubectl get secrets -n <your-namespace> | grep gitlab-registry

# If missing, create it
kubectl create secret docker-registry gitlab-registry \
  --docker-server=registry.anshinhealth.net:443 \
  --docker-username=gitlab-ci-token \
  --docker-password=<your-gitlab-token> \
  -n <your-namespace>

Step 4: Create `.gitlab-ci.yml`

# GitLab CI/CD Pipeline
# Uses BuildKit rootless mode for K8s compatibility (no docker-in-docker required)

stages:
  - build
  - deploy

variables:
  REGISTRY: registry.anshinhealth.net:443
  IMAGE_TAG: $CI_COMMIT_SHORT_SHA
  K8S_NAMESPACE: <your-namespace>
  GIT_SSL_NO_VERIFY: "true"

# CRITICAL: All jobs must include the kubernetes tag
default:
  tags:
    - kubernetes

# ── BUILD STAGE ──────────────────────────────────────────────────────────────
build:
  stage: build
  image: moby/buildkit:v0.12.5-rootless
  variables:
    BUILDKITD_FLAGS: --oci-worker-no-process-sandbox
  before_script:
    - echo "Building image"
  script:
    - export BUILDKITD_FLAGS="--oci-worker-no-process-sandbox"
    - mkdir -p ~/.docker
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(printf "%s:%s" "$CI_REGISTRY_USER" "$CI_REGISTRY_PASSWORD" | base64 | tr -d '\n')\"}}}" > ~/.docker/config.json
    - |
      buildctl-daemonless.sh build \
        --frontend dockerfile.v0 \
        --local context="$CI_PROJECT_DIR" \
        --local dockerfile="$CI_PROJECT_DIR" \
        --output type=image,name=$CI_REGISTRY_IMAGE:$IMAGE_TAG,push=true
    - |
      buildctl-daemonless.sh build \
        --frontend dockerfile.v0 \
        --local context="$CI_PROJECT_DIR" \
        --local dockerfile="$CI_PROJECT_DIR" \
        --output type=image,name=$CI_REGISTRY_IMAGE:latest,push=true
  only:
    - dev
  retry:
    max: 2
    when:
      - runner_system_failure
      - script_failure

# ── DEPLOY STAGE ─────────────────────────────────────────────────────────────
deploy:
  stage: deploy
  image: bitnami/kubectl:latest
  before_script:
    - kubectl version --client
  script:
    - kubectl set image deployment/<deployment-name> <container-name>=$CI_REGISTRY_IMAGE:$IMAGE_TAG -n $K8S_NAMESPACE
    - kubectl rollout status deployment/<deployment-name> -n $K8S_NAMESPACE --timeout=5m
  needs:
    - build
  only:
    - dev

Step 5: Required Repository Files

Dockerfile (Node.js example)

FROM node:20-alpine AS builder

WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:20-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev && npm cache clean --force
COPY --from=builder /app/dist ./dist

RUN addgroup -g 1001 -S nodejs && adduser -S nodejs -u 1001
USER nodejs

EXPOSE 8000
CMD ["node", "dist/index.js"]

package-lock.json

npm ci requires a lockfile. If missing, generate it:

npm install --package-lock-only

K8s Manifests (`k8s/` directory)

File	Content
`deployment.yaml`	Deployment with `imagePullSecrets: [{name: gitlab-registry}]`
`service.yaml`	ClusterIP service
`ingress.yaml`	Ingress with hostname and TLS secret reference

Step 6: Verify Setup Before First Push

# 1. Check runner is assigned
curl -s --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.anshinhealth.net/api/v4/projects/<PROJECT_ID>/runners" | jq '.'

# 2. Check variables are set
curl -s --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.anshinhealth.net/api/v4/projects/<PROJECT_ID>/variables" | jq '.[] | {key}'

# 3. Test pipeline trigger (shows YAML errors if any)
curl -s --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.anshinhealth.net/api/v4/projects/<PROJECT_ID>/pipeline?ref=dev"

# 4. Verify K8s resources exist
kubectl get deployment -n <your-namespace>
kubectl get svc -n <your-namespace>
kubectl get ingress -n <your-namespace>

Useful Commands

# Check recent pipeline status
curl -s --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.anshinhealth.net/api/v4/projects/<PROJECT_ID>/pipelines?per_page=5" | jq '.'

# Get job logs
curl -s --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.anshinhealth.net/api/v4/projects/<PROJECT_ID>/jobs/<JOB_ID>/trace"

# View pods
kubectl get pods -n <namespace>

# View deployment logs
kubectl logs -f deployment/<name> -n <namespace>

# Restart deployment
kubectl rollout restart deployment/<name> -n <namespace>

Troubleshooting

Pipeline fails immediately with no jobs

Cause: Runner not assigned or YAML syntax error
Fix: Enable runner via API/UI, validate .gitlab-ci.yml syntax

curl -s --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.anshinhealth.net/api/v4/projects/<PROJECT_ID>/pipeline?ref=dev"
# Look for error message in the response JSON

Build fails with "operation not permitted"

Cause: Wrong build image or missing flags
Fix: Use moby/buildkit:v0.12.5-rootless and set BUILDKITD_FLAGS: --oci-worker-no-process-sandbox

Registry authentication failure

Cause: Missing or incorrect CI variables
Fix: Verify CI_REGISTRY, CI_REGISTRY_USER, CI_REGISTRY_PASSWORD are set correctly

`npm ci` fails with "package-lock.json not found"

Cause: Lockfile not committed to repository
Fix: Run npm install --package-lock-only and commit package-lock.json

Deploy fails with "deployment not found"

Cause: K8s resources not yet created
Fix: Run kubectl apply -f k8s/ before the first pipeline run

`before_script` syntax error

Only simple strings are valid in before_script. Multiline | blocks belong in script only:

# ❌ WRONG
before_script:
  - |
    if [ "$VAR" = "value" ]; then
      echo "something"
    fi

# ✅ CORRECT
before_script:
  - echo "Simple string"
script:
  - |
    if [ "$VAR" = "value" ]; then
      echo "something"
    fi

Document Control

Rev	Date	Author	Description
1.0	2026-01-24	Anshin SRE Team	Initial release

Branch Strategy​

Infrastructure​

GitLab Instance​

GitLab Runners​

Existing Projects​

Pipeline Architecture​

Setup Steps for a New Project​

Step 1: Enable the K8s Runner​

Step 2: Configure CI/CD Variables​

Step 3: Create Kubernetes Resources​

Step 4: Create .gitlab-ci.yml​

Step 5: Required Repository Files​

Dockerfile (Node.js example)​

package-lock.json​

K8s Manifests (k8s/ directory)​

Step 6: Verify Setup Before First Push​

Useful Commands​

Troubleshooting​

Pipeline fails immediately with no jobs​

Build fails with "operation not permitted"​

Registry authentication failure​

npm ci fails with "package-lock.json not found"​

Deploy fails with "deployment not found"​

before_script syntax error​

Document Control​