2.3、存储方案配置

# 存储方案配置

## 📖 说明

本文档提供混合云存储方案的完整配置，采用分层设计策略。

## 🎯 存储架构原则

1. **生产数据**：云厂商高可用存储（CBS/云盘）
2. **灰度/开发**：Longhorn分布式存储
3. **备份归档**：公司内网MinIO（利用2x2T容量）
4. **临时数据**：Local-Path（节点本地）

## 📄 完整配置

```yaml
# ========================================
# 混合云存储方案（分层设计）
# ========================================

# 架构原则：
# 1. 生产数据：云厂商高可用存储（CBS/云盘）
# 2. 灰度/开发：Longhorn分布式存储
# 3. 备份归档：公司内网MinIO（利用2x2T容量）
# 4. 临时数据：Local-Path（节点本地）

---
# 方案1：云厂商CSI（生产环境推荐）
---
# 腾讯云CBS示例
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: prod-ssd
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: com.tencent.cloud.csi.cbs
parameters:
  type: CLOUD_HSSD  # 高性能SSD
  zone: ap-guangzhou-3
  diskChargeType: POSTPAID_BY_HOUR
reclaimPolicy: Retain  # 生产数据保留
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer  # 延迟绑定，确保Pod和PVC在同一可用区

---
# 阿里云云盘示例
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: prod-essd
provisioner: diskplugin.csi.alibabacloud.com
parameters:
  type: cloud_essd
  performanceLevel: PL1
  encrypted: "true"
reclaimPolicy: Retain
allowVolumeExpansion: true

---
# 方案2：Longhorn（灰度/开发环境）
---
# 安装Longhorn
# kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.5.3/deploy/longhorn.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gray-longhorn
provisioner: driver.longhorn.io
parameters:
  numberOfReplicas: "2"  # 2副本（仅云侧节点）
  staleReplicaTimeout: "30"
  fromBackup: ""
  fsType: "ext4"
  dataLocality: "best-effort"  # 优先本地访问
  nodeSelector: "topology.kubernetes.io/zone in (cloud-a, cloud-b)"  # 仅云侧
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: Immediate

---
# 开发环境StorageClass（单副本）
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: dev-longhorn
provisioner: driver.longhorn.io
parameters:
  numberOfReplicas: "1"
  nodeSelector: "topology.kubernetes.io/zone=corp"  # 仅公司节点
reclaimPolicy: Delete
allowVolumeExpansion: true

---
# 方案3：MinIO备份存储（公司节点）
---
apiVersion: v1
kind: Namespace
metadata:
  name: backup
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: minio
  namespace: backup
spec:
  serviceName: minio
  replicas: 1
  selector:
    matchLabels:
      app: minio
  template:
    metadata:
      labels:
        app: minio
    spec:
      nodeSelector:
        topology.kubernetes.io/zone: corp
      tolerations:
      - key: location
        operator: Equal
        value: corp
        effect: NoSchedule
      containers:
      - name: minio
        image: minio/minio:RELEASE.2024-01-01T00-00-00Z
        args:
        - server
        - /data
        - --console-address
        - ":9001"
        env:
        - name: MINIO_ROOT_USER
          value: admin
        - name: MINIO_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: minio-secret
              key: password
        ports:
        - containerPort: 9000
          name: api
        - containerPort: 9001
          name: console
        volumeMounts:
        - name: data
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: local-path  # 使用本地存储（2x2T）
      resources:
        requests:
          storage: 1.5Ti  # 留出空间给其他服务
---
apiVersion: v1
kind: Service
metadata:
  name: minio
  namespace: backup
spec:
  type: ClusterIP
  clusterIP: 172.16.72.50  # 固定IP便于配置
  selector:
    app: minio
  ports:
  - name: api
    port: 9000
    targetPort: 9000
  - name: console
    port: 9001
    targetPort: 9001

---
# 方案4：Local-Path（临时数据）
---
# 安装Local-Path Provisioner
# kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.26/deploy/local-path-storage.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-path
provisioner: rancher.io/local-path
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

---
# ========================================
# 关键服务PVC配置示例
# ========================================

# Harbor镜像仓库（使用云厂商高可用存储）
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: harbor-registry-data
  namespace: harbor
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: prod-ssd
  resources:
    requests:
      storage: 500Gi

---
# Prometheus监控数据（使用Longhorn）
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prometheus-data
  namespace: monitoring
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: gray-longhorn
  resources:
    requests:
      storage: 200Gi

---
# Gitea数据（使用云厂商存储）
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gitea-data
  namespace: devops
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: prod-ssd
  resources:
    requests:
      storage: 300Gi

---
# ========================================
# Velero备份配置（使用MinIO后端）
# ========================================
# 安装Velero
# velero install \
#   --provider aws \
#   --plugins velero/velero-plugin-for-aws:v1.8.0 \
#   --bucket k8s-backup \
#   --secret-file ./minio-credentials \
#   --backup-location-config \
#     region=minio,s3ForcePathStyle="true",s3Url=http://minio.backup.svc:9000

---
apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: minio-backup
  namespace: velero
spec:
  provider: aws
  objectStorage:
    bucket: k8s-backup
  config:
    region: minio
    s3ForcePathStyle: "true"
    s3Url: http://minio.backup.svc:9000
    insecureSkipTLSVerify: "true"

---
# 定期备份计划
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: prod-daily-backup
  namespace: velero
spec:
  schedule: "0 2 * * *"  # 每天凌晨2点
  template:
    includedNamespaces:
    - production
    - harbor
    - monitoring
    storageLocation: minio-backup
    ttl: 720h0m0s  # 保留30天

---
# ========================================
# 存储监控告警
# ========================================
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: storage-alerts
  namespace: monitoring
spec:
  groups:
  - name: storage
    interval: 30s
    rules:
    - alert: PVCDiskFull
      expr: (kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes) > 0.85
      for: 5m
      annotations:
        summary: "PVC {{ $labels.persistentvolumeclaim }} 使用率超过85%"
    - alert: LonghornVolumeUnhealthy
      expr: longhorn_volume_robustness != 0
      for: 2m
      annotations:
        summary: "Longhorn卷 {{ $labels.volume }} 不健康"
    - alert: MinIODown
      expr: up{job="minio"} == 0
      for: 3m
      annotations:
        summary: "MinIO备份存储不可用"
```

## 📊 存储类型对照表

| StorageClass | 提供者 | 副本数 | 用途 | 性能 | 成本 |
|-------------|-------|-------|-----|------|-----|
| **prod-ssd** | 云厂商CSI | 云厂商保证 | 生产数据 | 高 | 高 |
| **gray-longhorn** | Longhorn | 2副本 | 灰度/测试 | 中 | 低 |
| **dev-longhorn** | Longhorn | 1副本 | 开发环境 | 中 | 低 |
| **local-path** | Local-Path | 无 | 临时数据 | 高 | 最低 |

## ⚠️ 注意事项

- **生产数据**：必须使用云厂商存储，确保数据安全
- **Longhorn**：仅用于非关键数据，注意配置节点选择器
- **MinIO**：运行在公司节点，注意配置污点容忍
- **备份**：每天自动备份，保留30天
- **监控**：配置存储告警，及时发现问题

## 📁 原始文件

原始YAML配置文件位于：`solutions/storage-solution.yaml`

## 🔗 相关文档

- 架构方案：[1.1、最终架构方案.md](./1.1、最终架构方案.md)
- 部署指南：[1.5、云C部署指南.md](./1.5、云C部署指南.md)

---

**更新时间：** 2025-01-22