k8s-Operator

发表于 分类于
讲解 Kubernetes Operator 模式管理有状态应用，以 Etcd Operator 为例演示部署、CRD 与集群创建流程。

Operator

在 Kubernetes 中，管理“有状态应用”是一个比较复杂的过程，尤其是编写 Pod 模板的时候，总有一种 “在 YAML 文件里编程序”的感觉，让人很不舒服。

而在 Kubernetes 生态中，还有一个相对更加灵活和编程友好的管理“有状态应用”的解决方案，它就是：Operator。

Etcd Operator 示例

⚠️ 注：下方输出为 k8s 1.20 时代的记录，其中多处 API 已随版本演进：rbac.authorization.k8s.io/v1beta1（ClusterRole/ClusterRoleBinding）已在 1.22 移除，请改用 rbac.authorization.k8s.io/v1；extensions/v1beta1 的 Deployment 应改用 apps/v1；apiextensions.k8s.io/v1beta1（CRD）已在 1.22 移除，请改用 apiextensions.k8s.io/v1。（etcd.database.coreos.com/v1beta2 是该 Operator 自定义资源的版本，与 k8s 内置 API 无关，不受影响。）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
[root@k8s01 ~]# git clone https://github.com/coreos/etcd-operator

[root@k8s01 ~]# cd etcd-operator/

[root@k8s01 etcd-operator]# example/rbac/create_role.sh

Creating role with ROLE_NAME=etcd-operator, NAMESPACE=default
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
clusterrole.rbac.authorization.k8s.io/etcd-operator created
Creating role binding with ROLE_NAME=etcd-operator, ROLE_BINDING_NAME=etcd-operator, NAMESPACE=default
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
clusterrolebinding.rbac.authorization.k8s.io/etcd-operator created

[root@k8s01 etcd-operator]# kubectl create -f example/deployment.yaml  # 因为使用的 v1.20, 所以需要修改文件，见最后
error: unable to recognize "example/deployment.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"


[root@k8s01 etcd-operator]# kubectl apply -f  example/deployment.yaml  # 创建 Etcd Operator
deployment.apps/etcd-operator created

[root@k8s01 etcd-operator]# kubectl get pods |grep etcd
etcd-operator-646cbffdb6-rwhl6   1/1     Running   0          6m28s

[root@k8s01 etcd-operator]# kubectl get crd |grep etcd  # 有一个 crd 被创建出来
etcdclusters.etcd.database.coreos.com                 2021-01-05T17:04:06Z


[root@k8s01 etcd-operator]#  kubectl apply -f example/example-etcd-cluster.yaml  # 编写一个 EtcdCluster 的 YAML 文件

[root@k8s01 etcd-operator]# kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
etcd-operator-646cbffdb6-rwhl6    1/1     Running   0          32m
example-etcd-cluster-5hrtljjplq   1/1     Running   0          2m39s
example-etcd-cluster-n5x8pbzfzv   1/1     Running   0          7m20s
example-etcd-cluster-vwz9wcsxcf   1/1     Running   0          5m11s

Operator 的工作原理，实际上是利用了 Kubernetes 的自定义 API 资源（CRD），来描述我们想要部署的“有状态应用”；然后在自定义控制器里，根据自定义 API 对象的变化，来完成具体的部署和运维工作。

Kubernetes 里添加了一个名叫 EtcdCluster 的自定义资源类型。而 Etcd Operator 本身，就是这个自定义资源类型对应的自定义控制器。

部署清单示例

example/deployment.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
apiVersion: apps/v1
kind: Deployment
metadata:
  name: etcd-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      name: etcd-operator
  template:
    metadata:
      labels:
        name: etcd-operator
    spec:
      containers:
      - name: etcd-operator
        image: quay.io/coreos/etcd-operator:v0.9.4
        command:
        - etcd-operator
        # Uncomment to act for resources in all namespaces. More information in doc/user/clusterwide.md
        #- -cluster-wide
        env:
        - name: MY_POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name

etcdclusters.etcd.database.coreos.com

如果有 API 组（Group）是 etcd.database.coreos.com 、API 资源类型（Kind ）是 EtcdCluster 的 YAML 文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
[root@k8s01 etcd-operator]# kubectl describe crd  etcdclusters.etcd.database.coreos.com
Name:         etcdclusters.etcd.database.coreos.com
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  apiextensions.k8s.io/v1
Kind:         CustomResourceDefinition
Metadata:
  Creation Timestamp:  2021-01-05T17:04:06Z
  Generation:          1
  Managed Fields:
    API Version:  apiextensions.k8s.io/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:conversion:
          .:
          f:strategy:
        f:group:
        f:names:
          f:kind:
          f:listKind:
          f:plural:
          f:shortNames:
          f:singular:
        f:preserveUnknownFields:
        f:scope:
        f:version:
        f:versions:
      f:status:
        f:storedVersions:
    Manager:      etcd-operator
    Operation:    Update
    Time:         2021-01-05T17:04:06Z
    API Version:  apiextensions.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:acceptedNames:
          f:kind:
          f:listKind:
          f:plural:
          f:shortNames:
          f:singular:
        f:conditions:
    Manager:         kube-apiserver
    Operation:       Update
    Time:            2021-01-05T17:04:06Z
  Resource Version:  75878
  UID:               2eaa1a5d-e086-413e-873a-bf6d3de062dd
Spec:
  Conversion:
    Strategy:  None
  Group:       etcd.database.coreos.com  # 标识
  Names:
    Kind:       EtcdCluster  # 标识
    List Kind:  EtcdClusterList
    Plural:     etcdclusters
    Short Names:
      etcd
    Singular:               etcdcluster
  Preserve Unknown Fields:  true
  Scope:                    Namespaced
  Versions:
    Name:     v1beta2  # 版本
    Served:   true
    Storage:  true
Status:
  Accepted Names:
    Kind:       EtcdCluster
    List Kind:  EtcdClusterList
    Plural:     etcdclusters
    Short Names:
      etcd
    Singular:  etcdcluster
  Conditions:
    Last Transition Time:  2021-01-05T17:04:06Z
    Message:               spec.preserveUnknownFields: Invalid value: true: must be false
    Reason:                Violations
    Status:                True
    Type:                  NonStructuralSchema
    Last Transition Time:  2021-01-05T17:04:06Z
    Message:               no conflicts found
    Reason:                NoConflicts
    Status:                True
    Type:                  NamesAccepted
    Last Transition Time:  2021-01-05T17:04:06Z
    Message:               the initial names have been accepted
    Reason:                InitialNamesAccepted
    Status:                True
    Type:                  Established
  Stored Versions:
    v1beta2
Events:  <none>

example/example-etcd-cluster.yaml

1
2
3
4
5
6
7
8
9
10
11
12
[root@k8s01 etcd-operator]# cat example/example-etcd-cluster.yaml
apiVersion: "etcd.database.coreos.com/v1beta2"  # 使用的 api ，上文中的 Group/Versions
kind: "EtcdCluster"  #  资源类型，上文定义的 kind
metadata:
  name: "example-etcd-cluster"
  ## Adding this annotation make this cluster managed by clusterwide operators
  ## namespaced operators ignore it
  # annotations:
  #   etcd.database.coreos.com/scope: clusterwide
spec:
  size: 3
  version: "3.2.13"

更多 Operator 示例

一些 operator 示例

1
https://github.com/operator-framework/awesome-operators