etcd与k8s

发表于 | 分类于

获取 Kubernetes 存储在 etcd 中的 keys

1
2
3
4
5
6
7
8
9
10
11
12
[root@test-173 ~]# ETCDCTL_API=3 etcdctl get / --prefix --keys-only  |grep -Ev "^$"
/calico/ipam/v2/assignment/ipv4/block/170.56.144.192-26
/calico/ipam/v2/assignment/ipv4/block/170.56.170.64-26
/calico/ipam/v2/assignment/ipv4/block/170.56.27.0-26
/calico/ipam/v2/assignment/ipv4/block/170.56.73.192-26
/calico/ipam/v2/handle/k8s-pod-network.006edea2a1463ad925976290e5a0fea9d87e3188731566465ec8459f20b54a9d
/calico/ipam/v2/handle/k8s-pod-network.0397054208d1bce46053df4814d1006f14ae0dd98280555aa5b35d1b8b02c06e
/calico/ipam/v2/handle/k8s-pod-network.070dee87e37e685f2f6adac7ea38a54bb2326ac1c5d56bcae3b9b46086ff5c2b
/calico/ipam/v2/handle/k8s-pod-network.0fad3aa32d7c9bfe0392577fc44a9f7ad9482c9914e0a93b1b94363cd1173585
# 略

[root@test-173 ~]# ETCDCTL_API=3 etcdctl --endpoints=https://172.20.40.196:2379,https://172.20.40.107:2379,https://172.20.40.173:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem get / --prefix --keys-only  |grep -Ev "^$"  # 同样结果

查看键值

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
[root@test-107 ~]# ETCDCTL_API=3 etcdctl --endpoints=https://172.20.40.196:2379,https://172.20.40.107:2379,https://172.20.40.173:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem  get /registry/services/endpoints/monitoring/grafana  # 查看键值

/registry/services/endpoints/monitoring/grafana
k8s

v1      Endpoints⚌
⚌
grafana
monitoring"*$c5bca02d-dcde-4638-88fa-d32f1606cd722⚌⚌߃Z&
pp.kubernetes.io/componentgrafanaZ!
app.kubernetes.io/namegrafanaZ,
app.kubernetes.io/part-ofkube-prometheusZ"
app.kubernetes.io/version7.4.3bM
0endpoints.kubernetes.io/last-change-trigger-time2021-04-26T18:01:18+08:00z⚌⚌
kube-controller-managerUpdatev⚌FieldsV1:⚌
⚌{"f:metadata":{"f:annotations":{".":{},"f:endpoints.kubernetes.io/last-change-trigger-time":{}},"f:labels":{".":{},"f:app.kubernetes.io/component":{},"f:app.kubernetes.io/name":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{}}},"f:subsets":{}}⚌
⚌
170.56.73.220_
Pod
172.20.40.173fana-665447c488-245vl"$db4da82f-36ed-49b9-8003-197a61e8ed97*11359764:"
http⚌TCP"


[root@test-107 ~]# ETCDCTL_API=3 etcdctl --endpoints=https://172.20.40.196:2379,https://172.20.40.107:2379,https://172.20.40.173:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem  get /registry/namespaces/default

/registry/namespaces/default
k8s

v1      Namespace⚌
⚌
default"*$fb03c357-9393-11eb-898b-fad65d90ce002⚌⚌⚌Z$
field.cattle.io/projectIdp-zrh2mZ
istio-injectionenabledb⚌
cattle.io/status⚌{"Conditions":[{"Type":"ResourceQuotaInit","Status":"True","Message":"","LastUpdateTime":"2021-04-14T05:51:05Z"},{"Type":"InitialRolesPopulated","Status":"True","Message":"","LastUpdateTime":"2021-04-14T05:51:05Z"}]}b,
field.cattle.io/projectIdc-hxllk:p-zrh2mb1
)lifecycle.cattle.io/create.namespace-authtruer#controller.cattle.io/namespace-authz


kubernetes
Active"

键类型

这些键定义了集群中所有资源的配置和状态：

1
2
3
4
5
6
7
8
9
10
Nodes
Namespaces
ServiceAccounts
Roles and RoleBindings, ClusterRoles / ClusterRoleBindings
ConfigMaps
Secrets
Workloads: Deployments, DaemonSets, Pods, …
Cluster's certificates
The resources within each apiVersion
The events that bring the cluster in the current state

元数据资源在 etcd 中的存储格式由前缀、资源类型、namespace 和具体资源名组成

补

Kubernetes 集群使用了事务 Txn 接口防止并发创建、更新被覆盖等问题。当执行完 BeforeCreate 策略后，API Server 会调用 Storage 模块的 Create 接口写入资源。Storage.Create 接口调用底层存储模块 etcd3，将 user Deployment 资源对象写入 etcd。

Kubernetes 使用 watch 机制获取数据变化的事件，etcd watch 机制提供了流式推送机制，相比于定时轮询减少了高昂的查询开销，方便 API Server 实现数据的监听。服务器端的 store 对象利用 etcd 的 watch 机制，当 watch 机制触发时，数据的变化信息将封装成 event 对象并打包发送出去，客户端则通过不停地监听尝试读取 event chunk。

需要注意的是，Kubernetes 社区提供了通用的Informer 组件，实现了客户端与 API Server 之间的资源和事件同步。Informer 机制的 Reflector 封装了 Watch、List 操作，结合本地 Cache、Indexer，控制器加载完初始状态数据后，接下来的其他操作只需从本地缓存读取，极大降低了 API Server 和 etcd 的压力。