kube-scheduler-simulator

a-c-dream 发布于

简介

由于默认的 Kubernetes 调度器是高度可配置的，在很多情况下我们不需要编写任何代码来自定义调度行为。但是，想要了解调度器如何工作或者有更多二次开发需求的人员可能会尝试开发自己的调度器，在本文中，我将介绍如何借助 kube-scheduler-simulator 这个调度器模拟器来构建调度程序开发环境。

安装

1
2
3
$ git clone https://github.com/kubernetes-sigs/kube-scheduler-simulator.git
$ cd kube-scheduler-simulator
$ git checkout c5d91e0d674c721170f6ae08b1c7ef1ba7d18d14

在simulator/Dockerfile中添加ENV GOPROXY=https://goproxy.cn

1
2
3
4
5
6
7
8
9
10
FROM golang:1.16.6 AS build-env

ENV GOOS=linux
ENV GOARCH=amd64
ENV CGO_ENABLED=0
ENV GO111MODULE=on
ENV GOPROXY=https://goproxy.cn

WORKDIR /go/src/simulator-server

修改docker-compose.yml中的镜像

将registry.k8s.io改为k8s.nju.edu.cn

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
version: "3.7"
services:
  simulator-server:
    #    image: registry.k8s.io/scheduler-simulator/simulator-backend:v0.1.1
    image: k8s.nju.edu.cn/scheduler-simulator/simulator-backend:v0.1.1
    container_name: simulator-server
    environment:
      - PORT=1212
      - KUBE_SCHEDULER_SIMULATOR_ETCD_URL=http://simulator-cluster:2379
      - KUBE_APISERVER_URL=http://simulator-cluster:3131
    ports:
      - "1212:1212"
    restart: always
    tty: true
    networks:
      - simulator-internal-network
  simulator-frontend:
    #    image: registry.k8s.io/scheduler-simulator/simulator-frontend:v0.1.1
    image: k8s.nju.edu.cn/scheduler-simulator/simulator-frontend:v0.1.1
    restart: always
    container_name: simulator-frontend
    environment:
      - HOST=0.0.0.0
      - BASE_URL=http://${SIMULATOR_EXTERNAL_IP:-localhost}:1212
      - KUBE_API_SERVER_URL=http://${SIMULATOR_EXTERNAL_IP:-localhost}:3131
    ports:
    - "3000:3000"
    tty: true
  simulator-cluster:
    #    image: registry.k8s.io/kwok/cluster:v0.5.1-k8s.v1.29.0
    image: k8s.nju.edu.cn/kwok/cluster:v0.5.1-k8s.v1.29.0
    container_name: simulator-cluster
    restart: always
    ports:
      - "3131:3131"
    volumes:
      - simulator-etcd-data:/var/lib/etcd
      - ./kwok.yaml:/root/.kwok/kwok.yaml
    environment:
      - KWOK_KUBE_APISERVER_PORT=3131
    networks:
      - simulator-internal-network
networks:
  simulator-internal-network:
    driver: bridge
volumes:
  simulator-etcd-data:

修改docker-compose-local.yml中的镜像

将registry.k8s.io改为k8s.nju.edu.cn

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
version: "3.7"
services:
  simulator-server:
    image: simulator-server
    container_name: simulator-server
    environment:
      - PORT=1212
      - KUBE_SCHEDULER_SIMULATOR_ETCD_URL=http://simulator-cluster:2379
      - KUBE_APISERVER_URL=http://simulator-cluster:3131
    ports:
      - "1212:1212"
    restart: always
    tty: true
    networks:
      - simulator-internal-network
  simulator-frontend:
    image: simulator-frontend
    restart: always
    container_name: simulator-frontend
    environment:
      - HOST=0.0.0.0
      - BASE_URL=http://${SIMULATOR_EXTERNAL_IP:-localhost}:1212
      - KUBE_API_SERVER_URL=http://${SIMULATOR_EXTERNAL_IP:-localhost}:3131
    ports:
    - "3000:3000"
    tty: true
  simulator-cluster:
    #    image: registry.k8s.io/kwok/cluster:v0.5.1-k8s.v1.29.0
    image: k8s.nju.edu.cn/kwok/cluster:v0.5.1-k8s.v1.29.0
    container_name: simulator-cluster
    restart: always
    ports:
      - "3131:3131"
    volumes:
      - simulator-etcd-data:/var/lib/etcd
      - ./kwok.yaml:/root/.kwok/kwok.yaml
    environment:
      - KWOK_KUBE_APISERVER_PORT=3131
    networks:
      - simulator-internal-network
networks:
  simulator-internal-network:
    driver: bridge
volumes:
  simulator-etcd-data:

修改web/Dockerfile添加sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories&& \

1
2
3
4
5
6
FROM node:16-alpine AS deps

RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g'  /etc/apk/repositories&&  \
    apk update && \
    apk upgrade && \
    apk add --no-cache make gcc g++ py-pip

环境变量（！！！！！！！！！！！！！！非常重要！！！！！！！！！！！！！！）

如果出现能正常运行但是不能创建node等（或者创建之后显示NetworkError）就是这个原因

1
export SIMULATOR_EXTERNAL_IP=192.168.192.134

执行make

1
$ sudo make docker_build_and_up

启动和关闭

1
2
3
4
#启动
sudo make docker_up_local
#关闭
sudo make docker_down_local

启动之后就可以通过http://localhost:3000/访问了

image-20240330135354483

目前关闭之后创建的资源就会消失，所以记得及时export

自定义调度器

下面以一个随机调度器为例，说明如何在模拟器中实现自定义调度器。

1
git clone https://github.com/sanposhiho/mini-kube-scheduler.git

  • 将 mini-kube-scheduler/minisched （从分支 initial-random-scheduler）复制到 kube-scheduler-simulator

  • 修改 kube-scheduler-simulator/simulator/scheduler/scheduler.go 文件来使用 minisched

    修改 kube-scheduler-simulator/simulator/scheduler/scheduler.go 文件的内容如下所示，主要看 StartScheduler 函数的修改：

    先back一下scheduler.go

    按照下面的修改

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
     import (
         "context"

    +    "github.com/kubernetes-sigs/kube-scheduler-simulator/minisched"
    +
         "golang.org/x/xerrors"
         v1 "k8s.io/api/core/v1"
         clientset "k8s.io/client-go/kubernetes"
    @@ -14,7 +16,6 @@ import (
         "k8s.io/kubernetes/pkg/scheduler/apis/config"
         "k8s.io/kubernetes/pkg/scheduler/apis/config/scheme"
         "k8s.io/kubernetes/pkg/scheduler/apis/config/v1beta2"
    -    "k8s.io/kubernetes/pkg/scheduler/profile"

         simulatorschedconfig "github.com/kubernetes-sigs/kube-scheduler-simulator/scheduler/config"
         "github.com/kubernetes-sigs/kube-scheduler-simulator/scheduler/plugin"
    @@ -59,7 +60,6 @@ func (s *Service) ResetScheduler() error {
     // StartScheduler starts scheduler.
     func (s *Service) StartScheduler(versionedcfg *v1beta2config.KubeSchedulerConfiguration) error {
         clientSet := s.clientset
    -    restConfig := s.restclientCfg
         ctx, cancel := context.WithCancel(context.Background())

         informerFactory := scheduler.NewInformerFactory(clientSet, 0)
    @@ -71,36 +71,10 @@ func (s *Service) StartScheduler(versionedcfg *v1beta2config.KubeSchedulerConfig

         s.currentSchedulerCfg = versionedcfg.DeepCopy()

    -    cfg, err := convertConfigurationForSimulator(versionedcfg)
    -    if err != nil {
    -        cancel()
    -        return xerrors.Errorf("convert scheduler config to apply: %w", err)
    -    }
    -
    -    registry, err := plugin.NewRegistry(informerFactory, clientSet)
    -    if err != nil {
    -        cancel()
    -        return xerrors.Errorf("plugin registry: %w", err)
    -    }
    -
    -    sched, err := scheduler.New(
    +    sched := minisched.New(
             clientSet,
             informerFactory,
    -        profile.NewRecorderFactory(evtBroadcaster),
    -        ctx.Done(),
    -        scheduler.WithKubeConfig(restConfig),
    -        scheduler.WithProfiles(cfg.Profiles...),
    -        scheduler.WithPercentageOfNodesToScore(cfg.PercentageOfNodesToScore),
    -        scheduler.WithPodMaxBackoffSeconds(cfg.PodMaxBackoffSeconds),
    -        scheduler.WithPodInitialBackoffSeconds(cfg.PodInitialBackoffSeconds),
    -        scheduler.WithExtenders(cfg.Extenders...),
    -        scheduler.WithParallelism(cfg.Parallelism),
    -        scheduler.WithFrameworkOutOfTreeRegistry(registry),
         )
    -    if err != nil {
    -        cancel()
    -        return xerrors.Errorf("create scheduler: %w", err)
    -    }

         informerFactory.Start(ctx.Done())
         informerFactory.WaitForCacheSync(ctx.Done())

    修改完成后重新编译项目：

    1
    $ sudo make docker_build_and_up

失败成功方案

类似的问题，但并没有解决将自定义插件集成到模拟器中的问题 ·问题 #311 ·kubernetes-sigs/kube-scheduler-simulator (github.com)

非常感谢@sanposhiho 的帮助Cannot integrate custom plugins into the emulator · Issue #341 · kubernetes-sigs/kube-scheduler-simulator (github.com)

下面以官网的 nodenumber 为例

注册插件到outOfTreeRegistries

修改kube-scheduler-simulator/simulator/scheduler/config/plugin.go

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
import (
        "golang.org/x/xerrors"
        configv1 "k8s.io/kube-scheduler/config/v1"
        "k8s.io/kubernetes/pkg/scheduler/framework/plugins"
        "k8s.io/kubernetes/pkg/scheduler/framework/runtime"
        //add
         "sigs.k8s.io/kube-scheduler-simulator/simulator/docs/sample/nodenumber"
)

var (
        //add
        outOfTreeRegistries = runtime.Registry{
        // TODO(user): add your plugins registries here.
                nodenumber.Name: nodenumber.New,
        }

        registeredOutOfTreeMultiPointName = []string{
                nodenumber.Name,
        }
)

执行下面的命令重新build并运行

1
2
make docker_build_server
make docker_up_local

打开 后在设置中添加下面的内容

1
2
- name: NodeNumber # added
           weight: 10

image

这样nodeNumber就可以正常运行了

image

相关代码在这里

参考文档

Kube-Scheduler-simulator模拟 K8s 调度器环境_51CTO博客_k8s如何调度docker

k8s官方镜像代理加速_us-west2-docker.pkg.dev-CSDN博客

a-c-dream

a-c-dream

贵在坚持

109
13
0
CATEGORYS
TAGS