本文分享测试阿里推出的gpushare GPU虚拟化方案过程


# 概览

https://developer.aliyun.com/article/690623

gpu share scheduler extender https://github.com/AliyunContainerService/gpushare-scheduler-extender  
gpu share device plugin

# 部署

https://github.com/AliyunContainerService/gpushare-scheduler-extender/blob/master/docs/install.md

## 部署gpu-share scheduler extender

下载scheduler-policy-config，所有master都要做：

```bash
cd /etc/kubernetes/
curl -O https://raw.githubusercontent.com/AliyunContainerService/gpushare-scheduler-extender/master/config/scheduler-policy-config.json
```

注意：127.0.0.1:32766 这个端口只有在部署了gpushare-scheduler的机器才能访问，要改成service的ip和端口。

部署scheduler-extender:

```bash
cd /tmp/
curl -O https://raw.githubusercontent.com/AliyunContainerService/gpushare-scheduler-extender/master/config/gpushare-schd-extender.yaml
kubectl create -f gpushare-schd-extender.yaml
```

## 修改scheduler配置

所有master都要操作

```bash
cp /etc/kubernetes/manifests/kube-scheduler.yaml .
# edit kube-scheduler.yaml
cp kube-scheduler.yaml /etc/kubernetes/manifests/kube-scheduler.yaml
```

## 部署device plugin

```bash
wget https://raw.githubusercontent.com/AliyunContainerService/gpushare-device-plugin/master/device-plugin-rbac.yaml
kubectl create -f device-plugin-rbac.yaml
wget https://raw.githubusercontent.com/AliyunContainerService/gpushare-device-plugin/master/device-plugin-ds.yaml
kubectl create -f device-plugin-ds.yam
```

## 节点打标签

```bash
kubectl label node k8s-container-group-87-89 nodeType-
kubectl label node k8s-container-group-87-89 gpushare=true
```

## kubectl扩展

```bash
cd /usr/bin/
wget https://github.com/AliyunContainerService/gpushare-device-plugin/releases/download/v0.3.0/kubectl-inspect-gpushare
chmod u+x /usr/bin/kubectl-inspect-gpushare
```

# 测试

参考：https://github.com/AliyunContainerService/gpushare-scheduler-extender/blob/master/docs/userguide.md

ali-gpushare试用

本文分享测试TKEStack推出的vcuda GPU虚拟化方案过程



# 概览

vcuda是腾讯TKEStack推出的GPU虚拟化方案，支持细粒度的GPU算力和显存分割，配合kubernetes实现按需分配显存和算力。vcuda项目主要由两个模块组成：[gpu-admission](https://github.com/tkestack/gpu-admission) 和 [gpu-manager](https://github.com/tkestack/gpu-manager). gpu-admission是一个kubernetes的scheduler-extender，负责quota限制和vgpu请求的调度；gpu-manager作为device-plugin运行在节点上，通过内置的vcuda-controller模块，将GPU卡上报为`vcuda-core`和`vcuda-memory`两种设备，实现GPU算力和显存的细粒度（0.01卡和256Mi显存）分配。

# 部署

## gpu-admission

编译

```bash
docker run -ti --rm --name build -v /data/workspace:/data/workspace golang:1.16.4-buster /bin/bash -c "cd /data/workspace/vcuda/src/github.com/tkestack/gpu-admission && export GOPATH=/data/workspace/vcuda && export GOPROXY=https://goproxy.cn,direct && make build"
```

直接运行

```bash
nohup ./gpu-admission --address=0.0.0.0:8848 --v=4 --kubeconfig=admin.conf --log-dir=. 2>&1 &
```

创建`/etc/kubernetes/scheduler-policy-config.json`

```json
{
        "kind" : "Policy",
        "apiVersion" : "v1",
        "predicates" : [
                {"name" : "PodFitsHostPorts"},
                {"name" : "PodFitsResources"},
                {"name" : "NoDiskConflict"},
                {"name" : "MatchNodeSelector"},
                {"name" : "HostName"}
        ],
        "extenders" : [
                {
                        "urlPrefix": "http://${IP}:${PORT}/scheduler",
                        "apiVersion": "v1beta1",
                        "filterVerb": "predicates",
                        "enableHttps": false,
                        "nodeCacheCapable": false
                }
        ],
        "hardPodAffinitySymmetricWeight" : 10,
        "alwaysCheckAllPredicates" : false
}
```

修改kube-scheduler配置`/etc/kubernetes/manifests/kube-scheduler.yaml`（拷贝出来改完拷贝进去），三台master都要配置

```yaml
spec:
  containers:
  - command:
# ...
    - --policy-config-file=/etc/kubernetes/scheduler-policy-config.json
# ...
    volumeMounts:
    - mountPath: /etc/kubernetes/scheduler-policy-config.json
      name: policyconfig
	  readOnly: true
# ...
  volumes:
  - hostPath:
      path: /etc/kubernetes/scheduler-policy-config.json
      type: FileOrCreate
    name: policyconfig
```

## gpu-manager

镜像`thomassong/gpu-manager:1.1.5` `tkestack/gpu-manager:v1.1.5`

在k8s集群中创建gpu-manager sa

```bash
kubectl create sa gpu-manager -n kube-system
kubectl create clusterrolebinding gpu-manager-role --clusterrole=cluster-admin --serviceaccount=kube-system:gpu-manager
```

下载`https://raw.githubusercontent.com/tkestack/gpu-manager/master/gpu-manager.yaml`，修改镜像名为`tkestack/gpu-manager:v1.1.5`

如果节点上的kubelet设置cgroup-driver为systemd，需要给gpu-manager设置同样的参数：

```yaml
spec:
  containers:
    - env:
        - name: EXTRA_FLAGS
          value: "--logtostderr=false --cgroup-driver=systemd"
```

部署：

```bash
kubectl apply -f gpu-manager.yaml
```

给需要运行gpu-manager的节点打标签：

```bash
kubectl label node ${nodeName} nvidia-device-enable=enable
```

启动后，describe node可以看到：

```
tencent.com/vcuda-core:    400
tencent.com/vcuda-memory:  379
```

> 100 tencent.com/vcuda-core for 1 GPU and N tencent.com/vcuda-memory for GPU memory (1 tencent.com/vcuda-memory means 256Mi GPU memory)
you should add tencent.com/vcuda-core-limit: XX in the annotation field of a Pod
the value of tencent.com/vcuda-core is either the multiple of 100 or any value smaller than 100.For example, 100, 200 or 20 is valid value but 150 or 250 is invalid
> 

# 测试
<details>
  <summary>vcuda-testpod.yaml</summary>

```yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    container.apparmor.security.beta.kubernetes.io/<podName>: unconfined
    tencent.com/vcuda-core-limit: "50"
  labels:
    creator: yaorennan
    sshdPort: "22"
  name: <podName>
  namespace: default
spec:
  containers:
  - command:
    - /bin/bash
    - -c
    - /init_container/init_container
    env:
    - name: K8SADAPTER_CONTAINER_PASSWD
      value: xxx
    - name: K8SADAPTER_CONTAINER_SSHD_PORT
      value: "22"
    - name: K8SADAPTER_CONTAINER_SHM_SIZE_MB
      value: "127744"
    - name: K8SADAPTER_DATA_QUOTA
      value: "200"
    - name: K8SADAPTER_ROOT_QUOTA
      value: "50"
    image: reg.xxx.com/cloud-native/nvidia-tensorflow:21.04-tf1.15-py3-devctr
    name: <podName>
    resources:
      limits:
        cpu: "4"
        memory: 32768Mi
        tencent.com/vcuda-core: 50
        tencent.com/vcuda-memory: 16
      requests:
        cpu: "4"
        memory: 32768Mi
        tencent.com/vcuda-core: 50
        tencent.com/vcuda-memory: 16
    securityContext:
      capabilities:
        add:
        - SYS_ADMIN
        - NET_ADMIN
    volumeMounts:
    - mountPath: /init_container
      name: initcontainer
      readOnly: true
    - mountPath: /dockerdata
      name: data
    - mountPath: /sys/fs/cgroup
      name: cgroup
      readOnly: true
  nodeSelector:
    kubernetes.io/hostname: <nodeName>
  hostIPC: false
  restartPolicy: RestartContainer
  volumes:
  - hostPath:
      path: /data/init_container
    name: initcontainer
  - hostPath:
      path: /data/<podName>
    name: data
  - hostPath:
      path: /sys/fs/cgroup
    name: cgroup
```

</details>

运行TensorFlow的benchmark

```bash
python tf_cnn_benchmarks.py --batch_size=2 --num_gpus=1 --model=inception3 --num_batches=10000
```

分配了16个vcuda memory理论上应该能使用`16*256=4096M`，实际显示3612；GPU使用率有波动，总体在50%左右。每次运行都要执行很久的显存整理，增大显存会减少这种情况的发生：

```
2021-09-08 07:11:34.915785: W tensorflow/core/common_runtime/bfc_allocator.cc:419] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.00GiB (rounded to 2147250176).                                                     If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation.                                                                                               Current allocation summary follows.
```

分配32个vcuda memory，理论使用8192，实际使用7708

![](https://image.shiftzz.xyz/i/u/2024/04/22/wdssgc.png)

关于vcuda-memory的切分：  
一个vcuda-memory代表256Mi显存  
RTX3090有24268Mi显存，即‭94.796875‬个vcuda-memory  
经测试，在申请少于100个vcuda-core的情况下，申请94个vcuda-memory，会得到一张卡，运行TensorFlow benchmark，显存最大使用23780Mi；申请95个vcuda-memory，会调度报错：
```
UnexpectedAdmissionError Allocate failed due to rpc error: code = Unknown desc = request memory 25501368320 is larger than 25447170048, which is unexpected
```
申请200个vcuda-core，会得到两张卡，不管vcuda-memory设置了多少，每张卡都能用到23780Mi显存  
所以每张RTX3090只能按94个vcuda-memory来算

# 记录下遇到的坑

nvidia-smi报错

```bash
# nvidia-smi
F0907 09:32:45.524542     440 client.go:78] fail to get response from manager, error rpc error: code = Unknown desc = empty pids
/tmp/cuda-control/src/register.c:87 rpc client exit with 255
```

原因是cgroup-driver是systemd，需要加给gpu-manager加flag
https://github.com/tkestack/gpu-manager/issues/55#issuecomment-764544686

启动报错

```
copy /usr/local/host/local/agenttools/agent/plugins/titan_tools/nvidia-smi to /usr/local/nvidia/bin/
rebuild ldcache
launch gpu manager
E0817 08:08:20.982372   48580 server.go:121] Can not start volume managerImpl, err /usr/local/nvidia/bin/nvidia-smi: EOF
```

把宿主机的`/usr/local/host/local/agenttools/agent/plugins/titan_tools/nvidia-smi`文件（腾讯的监控模块自带的）删掉

```
E0817 08:30:27.632567   83361 server.go:132] Unable to set Type=notify in systemd service file?
```

这是执行systemd-notify时报错了，原因是容器内的`NOTIFY_SOCKET`变量为空，但是这个应该只在使用systemd启动时需要，1.0.4的镜像里直接退出了，用最新的代码编译的1.1.0镜像可以运行

```
E0813 16:37:06.443217   88913 server.go:152] can't create container runtime manager: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
```

containerd默认关了cri，需要打开：

```
# /etc/containerd/config.toml
disabled_plugins = ["cri"] #注释掉
systemctl restart containerd
```

```
I0813 17:42:28.889468   57385 server.go:334] Register to kubelet with endpoint vcore.sock
rpc error: code = Unimplemented desc = unknown service v1beta1.Registration
```

kubelet版本不对

vcuda试用

试用4paradigm-vgpu


# 概览

[4paradigm-k8s-device-plugin](https://github.com/4paradigm/k8s-device-plugin)是第四象限开源的vGPU device plugin实现。

> 在保留官方功能的基础上，实现了对物理GPU进行切分，并对显存和计算单元进行限制，从而模拟出多张小的vGPU卡。在k8s集群中，基于这些切分后的vGPU进行调度，使不同的容器可以安全的共享同一张物理GPU，提高GPU的利用率。
> 

使用场景

1. 显存、计算单元利用率低的情况，如在一张GPU卡上运行10个tf-serving。
2. 需要大量小显卡的情况，如教学场景把一张GPU提供给多个学生使用、云平台提供小GPU实例。
3. 物理显存不足的情况，可以开启虚拟显存，如大batch、大模型的训练。

# 部署

下载 https://raw.githubusercontent.com/4paradigm/k8s-device-plugin/master/nvidia-device-plugin.yml ，根据需要修改`args`里的参数：

- `fail-on-init-error:` 布尔类型, 预设值是true。当这个参数被设置为true时，如果装置插件在初始化过程遇到错误时程序会返回失败，当这个参数被设置为false时，遇到错误它会打印信息并且持续阻塞插件。持续阻塞插件能让装置插件即使部署在没有GPU的节点（也不应该有GPU）也不会抛出错误。这样你在部署装置插件在你的集群时就不需要考虑节点是否有GPU，不会遇到报错的问题。然而，这么做的缺点是如果GPU节点的装置插件因为一些原因执行失败，将不容易察觉。现在预设值为当初始化遇到错误时程序返回失败，这个做法应该被所有全新的部署采纳。
- `device-split-count:` 整数类型，预设值是2。NVIDIA装置的分割数。对于一个总共包含*N*张NVIDIA GPU的Kubernetes集群，如果我们将`device-split-count`参数配置为*K*，这个Kubernetes集群将有*K * N*个可分配的vGPU资源。注意，我们不建议将NVIDIA 1080 ti/NVIDIA 2080 ti `device-split-count`参数配置超过5，将NVIDIA T4配置超过7，将NVIDIA A100配置超过15。
- `device-memory-scaling:` 浮点数类型，预设值是1。NVIDIA装置显存使用比例，可以大于1（启用虚拟显存，实验功能）。对于有*M*显存大小的NVIDIA GPU，如果我们配置`device-memory-scaling`参数为*S*，在部署了我们装置插件的Kubenetes集群中，这张GPU分出的vGPU将总共包含 *S * M*显存。每张vGPU的显存大小也受`device-split-count`参数影响。在先前的例子中，如果`device-split-count`参数配置为*K*，那每一张vGPU最后会取得 *S * M / K* 大小的显存。
- `device-cores-scaling:` 浮点数类型，预设值与`device-split-count`数值相同。NVIDIA装置算力使用比例，可以大于1。如果`device-cores-scaling`参数配置为*S* `device-split-count`参数配置为*K*，那每一张vGPU对应的**一段时间内** sm 利用率平均上限为*S / K*。属于同一张物理GPU上的所有vGPU sm利用率总和不超过1。
- `enable-legacy-preferred:` 布尔类型，预设值是false。对于不支持 PreferredAllocation 的kublet（<1.19）可以设置为true，更好的选择合适的device，开启时，本插件需要有对pod的读取权限，可参看 legacy-preferred-nvidia-device-plugin.yml。对于 kubelet >= 1.9 时，建议关闭。

修改`nodeSelector`，指定要部署到的节点的标签，然后执行部署：

```bash
kubectl apply -f nvidia-device-plugin.yml
```

ds起来后，describe node能看到`nvidia.com/gpu`设备，名字和原生的一样，数量是物理卡乘以`device-split-count`

# 测试

用TensorFlow benchmark测试

- 一虚三，容器内nvidia-smi可以正确显示显存大小，会有112MiB的预留内存被占用了

![](https://image.shiftzz.xyz/i/u/2024/04/22/ouy6cx.png)

- 申请超过物理卡数的vgpu，会报错，文档里也说了分配的vgpu数不能超过物理卡数

![](https://image.shiftzz.xyz/i/u/2024/04/22/ovh3e9.png)

多卡分配测试

分配两个vGPU（1虚5）：

![](https://image.shiftzz.xyz/i/u/2024/04/22/ovqcei.png)

跑单卡程序，在容器内看：

![](https://image.shiftzz.xyz/i/u/2024/04/22/ow6bxh.png)

在宿主机：

![](https://image.shiftzz.xyz/i/u/2024/04/22/oy2x08.png)

跑两个卡的训练，容器内：

![](https://image.shiftzz.xyz/i/u/2024/04/22/oyq85j.png)

宿主机：

![](https://image.shiftzz.xyz/i/u/2024/04/22/ozt1d3.png)

再创建一个两vGPU的容器：

![](https://image.shiftzz.xyz/i/u/2024/04/22/qj6pxf.png)

可以看出来每次创建容器都会尽量分配不同gpu的vgpu，单卡的话是随机分

- 跑inception3 benchmark测试性能

```bash
python tf_cnn_benchmarks.py --batch_size=8 --num_gpus=1 --model=inception3 --num_batches=10000
```

两个一起跑

```
1350    images/sec: 112.2 +/- 0.7 (jitter = 2.1)        7.382
1150    images/sec: 107.2 +/- 0.1 (jitter = 2.0)        7.327
```

单独跑

```
1000    images/sec: 206.9 +/- 0.3 (jitter = 4.0)        7.290
```