DevOps dude, self-hoster, space nerd.

  • 2 Posts
  • 77 Comments
Joined 2 years ago
Cake day: June 12th, 2023
  • rs5th@lemmy.scottlabs.iotoSelfhosted@lemmy.world•What does your infrastructure look like?English
    5·
    2 years ago

    My setup is a mix of on-prem and VPS.

    On-Prem

    • Primary Cluster (24 cores, 192 GB RAM, 36 TB usable storage)
      • Two Dell R610 (12 cores, 96 GB RAM each)
      • vSphere 7.0, ESXi 6.7 (because processors are too old for 7.0)
      • Kubernetes 1.24
        • Single controller VM
        • Two worker VMs
        • OS: Ubuntu 20.04
        • K8s Flavor: Kubeadm
      • Use: Almost everything
      • Storage:
        • Synology 1515 (11 TB usable, RAID 5) - vSphere datastore via NFS
        • Synology 1517 (25 TB usable, RAID 5) - Kubernetes mounts via NFS, media, general NAS stuff
    • Standalone Node (4 cores, 16GB RAM, 250 GB SSD)
      • Lenovo M900 micro-PC
      • OS: Ubuntu 22.04
      • Kubernetes 1.24
      • K8s Flavor: k3s
      • Use: provide critical network services (DNS/DHCP) if any part of the complex cluster goes down, Frigate due to USB Coral TPU plugged in here
    • Networking / Other
      • DNS:
        • Primary: AdGuard Home running on Standalone
        • Internal domain: BIND VM running in Primary Cluster
      • Firewall: Juniper SRX 220H
      • Switch: Juniper EX2200-48
      • WiFi: 3x Unifi In-Wall APs
      • Power:
        • UPS backing compute and storage (10-15 min runtime)
        • UPS backing networking gear (15-20 minute runtime)

    VPS

    • Single Linode (2 cores, 4 GB RAM, 80 GB storage)
      • OS: Ubuntu 22.04
      • Kubernetes 1.24
      • K8s Flavor: k3s
      • Use: UptimeKuma to monitor on-prem infrastructure, services that can’t go down due to home ISP or power issues (like family RocketChat).

    Every service (except Plex) is containerized and running in Kubernetes. Plex will be migrated soon™. Everything in Kubernetes is handled via Infrastructure as Code using FluxCD and GitOps principles. Secrets are stored in git using Mozilla SOPS for encrypt/decrypt. Git repos are currently hosted in GitHub, but I’m considering Gitea, though that might present a bit of a bootstrapping problem if all the infrastructure that hosts Gitea is declared inside Gitea…