其实坑还挺多的,尤其是 immich 还需要使用特定版本的 pgvecto 支持的数据库。加之如果需要硬件加速,还需要额外的配置,这里记录一下防止我忘了。
Kubernetes Device Plugin 与 GPU Device
负载机用的 n5095 核显直通,是个 intel 设备,在用 helm 部署 immich 的情况中,直接 /dev/dri 直通并不是很方便,不过查了一下可以通过 Device Plugin 的形式,更好地管理设备。
# https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/cmd/gpu_plugin/README.md
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
# NFD base
- https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd?ref=v0.34.0
# NodeFeatureRules for GPU detection
- https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd/overlays/node-feature-rules?ref=v0.34.0
# GPU plugin daemonset
- https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/gpu_plugin/overlays/nfd_labeled_nodes?ref=v0.34.0
patches:
- target:
kind: DaemonSet
name: intel-gpu-plugin
patch: |-
- op: replace
path: /spec/template/spec/containers/0/args
value:
- "-enable-monitoring"
- "-v=2"
- "-shared-dev-num=5"
相较于官方的配置,我加入了 -share-dev-num=5, 使 GPU 可以在至多 5 个容器中共享,默认配置只能挂载一个。
PSQL Cluster
immich 用到特定版本的数据库,可以用这种方式部署。
apiVersion: v1
kind: Secret
metadata:
name: immich-postgres-user
type: Opaque
stringData:
username: immich
password: your-strong-password
---
# https://github.com/immich-app/immich-charts/issues/149
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: immich-postgres
spec:
# At the time of writing, immich is only compatible with pgvecto.rs <0.4. Latest postgres image with that version is 16.5.
imageName: ghcr.io/tensorchord/cloudnative-pgvecto.rs:16.5-v0.3.0@sha256:be3f025d79aa1b747817f478e07e71be43236e14d00d8a9eb3914146245035ba
instances: 1
postgresql:
shared_preload_libraries:
- "vectors.so"
managed:
roles:
- name: immich
superuser: true
login: true
bootstrap:
initdb:
database: immich
owner: immich
secret:
name: immich-postgres-user
postInitSQL:
- CREATE EXTENSION IF NOT EXISTS "vectors";
- CREATE EXTENSION IF NOT EXISTS "cube" CASCADE;
- CREATE EXTENSION IF NOT EXISTS "earthdistance" CASCADE;
storage:
size: 4Gi
storageClass: nfs-clientValues 参考
再手动创建一个 PVC 后,就可以进行部署了,注意 tag 的设置,选用合适的加速后端即可:
env:
DB_HOSTNAME: "immich-postgres-rw"
DB_USERNAME: "immich"
DB_DATABASE_NAME: "immich"
DB_PASSWORD: "your-strong-password"
image:
tag: v1.143.1
immich:
metrics:
enabled: false
persistence:
library:
existingClaim: immich-pvc
redis:
enabled: true
server:
resources:
limits:
cpu: 1000m
gpu.intel.com/i915: 1
ingress:
main:
enabled: true
hosts:
- host: immich.core
paths:
- path: "/"
tls: []
machine-learning:
image:
tag: v1.143.1-openvino
resources:
limits:
cpu: 1000m
gpu.intel.com/i915: 1
persistence:
cache:
type: pvc
storageClass: nfs-client