Reclaiming Your Data Sovereignty: Building a Private S3 Vault

Every containerized app I run eventually needs somewhere to store objects — Immich needs a place to write photo thumbnails, Milvus needs an object store for its index snapshots, and my backup jobs need a consistent target that doesn’t care about the underlying filesystem layout. I kept reaching for the same solution: MinIO, running on my TrueNAS box, backed by ZFS.

Using raw file paths or SMB shares for this creates a mess fast. Each app invents its own path conventions, permissions conflict, and you end up with a tangle of bind mounts nobody can audit. MinIO sidesteps all of that by presenting a standard S3 API that every modern app already knows how to speak — while keeping the actual bytes on my hardware, under my ZFS pool.

This isn’t a click-through guide. I’ll explain what’s actually happening at the storage layer so you understand why each configuration choice matters.

What You Need Coming In

Knowledge:

• ZFS fundamentals: how TrueNAS organizes storage pools and datasets
• Container basics: how Kubernetes (k3s) mounts persistent volumes into pods
• S3 identity concepts: access keys, secret keys, bucket policies

Environment:

• Host OS: TrueNAS SCALE 24.04 (Dragonfish) or newer
• Storage: An existing ZFS pool (tank) with adequate redundancy
• App Catalog: The official TrueNAS charts or enterprise catalog synchronized
• Network: Tailscale deployed on the TrueNAS host

MinIO as a Teller Window Over a ZFS Vault

The mental model I use: ZFS is the vault. It handles physical integrity, bit-rot protection via checksums, and RAID-Z redundancy. MinIO is the teller window. It doesn’t know or care how the vault is built — it accepts S3 API requests and hands them off to the filesystem beneath it.

flowchart TD
    subgraph Zero-Trust Network[Tailscale VPN / Zero-Trust Mesh]
        A[Self-Hosted App: Milvus] -->|S3 API: Port 9000| C(MinIO Pod)
        B[Backup Server: Xen/Synology] -->|S3 API: Port 9000| C
        Admin[Admin Browser] -->|Web UI: Port 9001| C
    end

    subgraph TrueNAS SCALE Host
        C -->|Persistent Volume Mount| D{ZFS Dataset}
        D -->|UID: 473 / GID: 473| E[(ZFS Pool: tank)]
    end

    style C fill:#f96,stroke:#333,stroke-width:2px
    style D fill:#69b,stroke:#333,stroke-width:2px

Because MinIO sits on top of ZFS, I don’t need MinIO’s built-in erasure coding or distributed mode. ZFS is already providing redundancy and checksum protection underneath the container. MinIO becomes a thin API gateway — which makes it fast and CPU-efficient.

Step 1: Create the Dataset With the Right Share Type

This is where most deployments go wrong. Create a generic dataset, let the container write as root, and you’ll have permission problems the moment you try to access the data over SMB or move it anywhere.

TrueNAS has a specific dataset share type for application workloads:

# Via TrueNAS CLI or equivalent UI steps
cli -c 'storage dataset create name="tank/Apps/minio_data" share_type="APPS"'

The APPS share type strips legacy SMB ACLs and prepares the dataset for strict POSIX/NFSv4 ownership, which is what Kubernetes pods expect. If you create a dataset with the wrong share type, the MinIO pod will refuse to start or will write data it can’t later read back.

Step 2: TLS — Self-Signed or Let’s Encrypt

MinIO should always run with TLS. Without it, credentials travel in plaintext across your network.

If you’re not using ACME, you can generate a self-signed cert scoped to your Tailscale IP:

cli -c 'cryptosystem certificate create \
    name="minio_internal_cert" \
    type="CERTIFICATE_CREATE_INTERNAL" \
    certificate_authority="minio_ca" \
    san=["100.x.y.z", "truenas.local"]'

One warning: self-signed certs will be rejected by any client with strict CA validation — Proxmox Backup Server, some boto3 scripts with SSL verification enabled. If that’s your environment, use TrueNAS’s built-in ACME authenticator with Cloudflare DNS challenges to get a globally trusted wildcard cert instead.

Step 3: Configure the Deployment

When you install MinIO via the TrueNAS Apps interface, you’re generating a Kubernetes Helm chart. Here’s the logical shape of the configuration, broken into the three chunks that matter:

# Identity and Access Management
minioConfiguration:
  rootUser: "minio_admin"
  rootPassword: "SuperSecretVaultPassword123!"

# Network exposure — mapped to Tailscale interface
serviceConfiguration:
  apiPort: 9000
  consolePort: 9001
  certificate: "minio_internal_cert"

Don’t use the rootUser credentials in your actual applications. Once the container is running, log into the console on port 9001 and generate dedicated access keys for each app — one for Immich, a separate one for Milvus, another for your backup server. If any one of those credentials is compromised, the blast radius is limited to that app’s buckets.

# Stateful storage mapping
storageConfiguration:
  extraHostPathVolumes:
    - hostPath: "/mnt/tank/Apps/minio_data"
      mountPath: "/data"

This last chunk is the important one. By specifying a host path instead of letting TrueNAS create an ephemeral PVC, you’re binding MinIO’s data directory directly to your ZFS dataset. The k3s engine pulls the MinIO image, mounts the dataset to /data, and starts the S3 listener.

What Happens When You Upload an Object

When an app sends a PUT request to MinIO’s S3 API, MinIO translates the object into a standard POSIX file and writes it to /data inside the container. Because /data maps to /mnt/tank/Apps/minio_data, the write immediately reaches ZFS.

ZFS intercepts it, caches it in RAM via the ARC, compresses it (LZ4 or ZSTD), computes a checksum for integrity, and stripes the data across your physical disks via RAID-Z. By the time the S3 API returns success, ZFS has the data safely committed.

Three Things That Will Break Your Deployment

The UID 473 trap. When TrueNAS starts the MinIO pod, it doesn’t run it as root. It runs as internal user minio with UID 473 and GID 473. If you touch the dataset’s ACLs manually — say, chown-ing it to root or your personal account — the container will crash on startup with a permission denied error on /data. Never manually alter the dataset ownership once the app has claimed it.

Certificate rejection. If you used a self-signed cert, clients with strict SSL verification will refuse to connect. The fix is either injecting your CA into the client’s trust store, or switching to a Let’s Encrypt cert via Cloudflare DNS challenges. Don’t try to work around this with verify=False in your application code — that defeats the purpose of TLS entirely.

Exposing port 9000. Don’t port-forward ports 9000 or 9001 on your home router. All apps that talk to MinIO should use the server’s Tailscale IP (100.x.y.z:9000). Your S3 vault is completely invisible to the public internet that way, and you can still reach it from any device in your Tailscale mesh.