Ingress

发表于 分类于
详解 ingress-nginx 部署与 HTTP/HTTPS 代理、BasicAuth、URL 重写、白名单及 cert-manager 自动证书。

Ingress 简介

k8s 暴露服务 可以通过 LoadBalance service, NodePort service, Ingress。

Ingress 目前只工作在 7层网络

Ingress : 其实就是一组基于 DNS 名称或 URL 路径把请求转发至指定的service资源的规则。(将 nginx 配置抽象出来 成为 ingress 对象，不用修改 配置文件，直接改yaml 文件然后创建/更新就可以了)

Ingress Controller : 其实是一个可以根据 Ingress 对象和被代理后端 Service 的变化，来自动进行更新的 Nginx 负载均衡器。

Ingress Controller 会根据 Ingress 对象定义的内容，生成 一份对应的 nginx 配置文件，并使用这个配置文件 启动一个 nginx 服务。一旦 Ingress 对象被更新， Ingress Controller 就会更新这个配置文件。

外部请求首先到达 Ingress Controller，Ingress Controller 根据 Ingress 的路由规则，查找到对应的 Service ，进而通过 Endpoint 查询到 Pod 的 IP 地址，然后将请求转发给 Pod

使用 ingress-nginx

官网 : https://kubernetes.github.io/ingress-nginx/deploy/

github : https://github.com/kubernetes/ingress-nginx

ingress 规则中 annotations 可以开启 nginx 的部分功能

安装 ingress-control

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
[root@k8s01 ~]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.40.1/deploy/static/provider/baremetal/deploy.yaml
## https://raw.githubusercontent.com/kubernetes/ingress-nginx/refs/heads/main/deploy/static/provider/baremetal/deploy.yaml
##  https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/baremetal/deploy.yaml （nodeport 版） 最新版，按需修改镜像
## 需要修改 deploy.yaml , eg : 镜像 imwl/ingress-nginx-controller:v0.40.1（同步的官方镜像）。
## 修改成 DaemonSet 后，需要删除 strategy字段，添加 hostNetwork: true，与 serviceAccountName: ingress-nginx 同级即可,删除
## 域名解析  ingress-nginx-controller 的 ip 地址， 云服务 loadbalance 的地址，或者把所有node ip都配上

[root@k8s01 ~]# kubectl apply -f deploy.yaml
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
configmap/ingress-nginx-controller created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
service/ingress-nginx-controller-admission created
service/ingress-nginx-controller created
deployment.apps/ingress-nginx-controller created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
serviceaccount/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created

Ingress HTTP 代理访问

test-Ingress01.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: imwl/myapp:v2
        ports:
        - name: http
          containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: myapp02
  namespace: default
spec:
  selector:
    app: myapp
  ports:
  - targetPort: 80
    port: 80

---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: example-ingress
  annotations:  # 可以开启 nginx 的部分功能，无需重启
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: master.k8s.com
    http:
      paths:
      - path: /
        backend:
          serviceName: myapp02
          servicePort: 80

Ingress HTTPS 代理访问

创建证书，以及 cert 存储方式

1
2
3
4
5
6
# CN/SAN 要写成实际要访问的域名，否则浏览器一定报证书不匹配。
# 下面这条签出来的 CN 是 nginxsvc 且没有 SAN，配到 master.k8s01.com 上是不能用的：
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt \
  -subj "/CN=master.k8s01.com/O=nginxsvc" \
  -addext "subjectAltName=DNS:master.k8s01.com"
kubectl create secret tls tls-secret --key tls.key --cert tls.crt

deployment、Service、Ingress Yaml 文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: nginx-test
spec:
  tls:
    - hosts:                      # 字段是 hosts（数组）；写成 host 会被丢弃，
      - master.k8s01.com          # tls 条目退化成"对所有 host 生效的默认证书"
      secretName: tls-secret
  rules:
    - host: master.k8s01.com
      http:
        paths:
        - path: /
          backend:
            serviceName: nginx-svc
            servicePort: 80

Nginx 进行 BasicAuth

1
2
3
yum -y install httpd
htpasswd -c auth foo
kubectl create secret generic basic-auth --from-file=auth
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-with-auth
  annotations:
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: basic-auth
    nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - foo'
spec:
  rules:
    - host: foo2.bar.com
      http:
        paths:
        - path: /
          backend:
            serviceName: nginx-svc
            servicePort: 80

Nginx 进行重写

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-test
  annotations:
    # rewrite-target 的值应该是目标 URI 路径，它会被渲染成 rewrite ... break;
    # 填带 scheme 的完整 URL 时 nginx 会转成 302 发回客户端，那就不是"重写"而是重定向了。
    # 要做外跳请用 nginx.ingress.kubernetes.io/permanent-redirect
    nginx.ingress.kubernetes.io/rewrite-target: /hostname.htm
spec:
  rules:
    - host: foo10.bar.com
      http:
        paths:
        - path: /
          backend:
            serviceName: nginx-svc
            servicePort: 80

k8s>1.19 设置

示例

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-myserviceb
spec:
  rules:
  - host: myserviceb.foo.org
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myserviceb
            port:
              name: https-dashboard

开启白名单

  1. 第一次需要修改 ingress-nginx-controller 的配置
1
2
3
4
# kubectl edit cm -n ingress-nginx ingress-nginx-controller   # 这个 cm 在 ingress-nginx 命名空间（安装时创建的），不在 kube-system
#data:
  allow-backend-server-header: "true"
  allow-snippet-annotations: "true"
  1. ingress 添加注解
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
annotations:
  ....
  nginx.ingress.kubernetes.io/server-snippet: |
    set $allow_access 0;
    if ($remote_addr ~* "<PUBLIC_IP>|<PUBLIC_IP>") {
      set $allow_access 1;
    }
    if ($uri ~* "webhook") {
      set $allow_access 1;
    }
    if ($allow_access = 0) {
      set $ingress_block 1;
      return 403;
    }
  # nginx 的 if 只接受单个条件，没有 && 和 ||。
  # 写 if ($status = 403 && ...) 会让 nginx -t 报 invalid condition，
  # ingress-nginx 拒绝加载新配置，白名单直接失效，还会卡住这个 controller 后续所有配置更新。
  # 所以把判断折进上面已经算好的那个变量，只留单条件：
  nginx.ingress.kubernetes.io/configuration-snippet: |
    more_set_headers "X-Maintenance-Mode: $ingress_block";

3, 通过 curl 命令验证生效：
非白名单ip

1
2
3
4
5
6
7
curl -I -X GET http://service-domain/healthcheck
# 会有 403 以及 X-Maintenance-Mode: true
HTTP/1.1 403 Forbidden
X-Maintenance-Mode: true

curl -I -X GET http://service-domain/xxx/webhook/xxx
# 不会是 403 也不会 添加 X-Maintenance-Mode: true

使用 cert-manager

cert-manager.yaml ，不需要修改

1
wget https://github.com/cert-manager/cert-manager/releases/download/v1.16.2/cert-manager.yaml

校验方式

HTTP-01 的校验方式的优点是: 配置简单通用，不管使用哪个 DNS 提供商都可以使用相同的配置方法；缺点是：需要依赖 Ingress，如果你的服务不是用 Ingress 暴露流量的就不适用，而且不支持泛域名证书。

DNS-01 的校验方式的优点是没有 HTTP-01 校验方式缺点，不依赖 Ingress，也支持泛域名；缺点就是不同 DNS 提供商的配置方式不一样，而且 DNS 提供商有很多，cert-manager 的 Issuer 不可能每个都去支持，不过有一些可以通过部署实现了 cert-manager 的 Webhook 的服务来扩展 Issuer 进行支持，比如 DNSPod 和 阿里 DNS，详细 Webhook 列表请参考: https://cert-manager.io/docs/configuration/acme/dns01/#webhook

dns01 方式

使用 ClusterIssuer，dns01, cloudflare 更多方式参考官网。dns01 的方式

其中 api-token 的生成 https://dash.cloudflare.com/profile/api-tokens

示例

cloudflare-api-tokens
编辑完成后会显示 token

cluster-issuer.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: cert-manager # 这里配置为安装cert-manager资源的命名空间
type: Opaque
stringData:
  api-token: 'xxxxxxxx' # 这里的值为上图创建的API Token值

---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: cloudflare-acme-cluster-issuer
spec:
  acme:
    email: '[email protected]'
    # 配置证书目录，演练环境使用Staging环境，注意区分
    server: https://acme-v02.api.letsencrypt.org/directory
    # server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: acme-issuer-account-key
    solvers:
    - dns01:
        cloudflare:
          apiTokenSecretRef:
            name: cloudflare-api-token-secret # 引用当前文档中创建的Secret名称
            key: api-token

配置 dns

cloudflare-dns-k8s

ingress 规则
  1. 默认设置
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: nginx
        ports:
        - name: http
          containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: myapp
  namespace: default
spec:
  selector:
    app: myapp
  ports:
  - targetPort: 80
    port: 80

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-myapp
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    cert-manager.io/cluster-issuer: cloudflare-acme-cluster-issuer
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - k8s.grafana.eu.org
    secretName: my-tls-secret  # Cert-Manager 自动创建
  rules:
  - host: k8s.grafana.eu.org
    http:
      paths:
      - path: /nginx
        pathType: Prefix
        backend:
          service:
            name: myapp
            port:
              number: 80
  1. 使用子域名
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-grafana
  namespace: monitor
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    cert-manager.io/cluster-issuer: cloudflare-acme-cluster-issuer
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - grafana.k8s.grafana.eu.org
    secretName: grafana-tls-secret  # Cert-Manager 自动创建
  rules:
  - host: grafana.k8s.grafana.eu.org
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: prometheus-grafana
            port:
              number: 80
  1. 使用路径(同一域名下，复用 k8s.grafana.eu.org的证书)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana
  namespace: monitor
  annotations:
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - k8s.grafana.eu.org
    secretName: my-tls-secret  # Cert-Manager 自动创建
  rules:
  - host: k8s.grafana.eu.org
    http:
      paths:
      - path: /grafana
        pathType: Prefix
        backend:
          service:
            name: prometheus-grafana
            port:
              number: 80
  1. 代理 https
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-dashboard
  namespace: kube-system
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    # Add https backend protocol support for GKE
    service.alpha.kubernetes.io/app-protocols: '{"https":"HTTPS"}'
    nginx.ingress.kubernetes.io/rewrite-target: /
    cert-manager.io/cluster-issuer: cloudflare-acme-cluster-issuer
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - dashboard.k8s.grafana.eu.org
    secretName: dashboard-tls-secret  # Cert-Manager 自动创建
  rules:
  - host: dashboard.k8s.grafana.eu.org
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kubernetes-dashboard
            port:
              number: 443
  1. 使用通配符, tls 引用 secretName 只能使用同一命名空间下的. 建议需要的命名空间都申请一遍
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-cert-k8s-grafana-eu-org
  namespace: monitor
spec:
  secretName: wildcard-cert-k8s-grafana-eu-org-tls  # 生成的证书将保存在此 Secret 中
  issuerRef:
    name: cloudflare-acme-cluster-issuer
    kind: ClusterIssuer    # 如果使用 Issuer 则改为 Issuer
  commonName: "*.k8s.grafana.eu.org"  # 通配符域名
  dnsNames:
  - "*.k8s.grafana.eu.org"
  - k8s.grafana.eu.org  # 如果需要顶级域名

ingress 使用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: altermanager
  namespace: monitor
  annotations:
    cert-manager.io/cluster-issuer: cloudflare-acme-cluster-issuer
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - "*.k8s.grafana.eu.org"
    secretName: wildcard-cert-k8s-grafana-eu-org-tls
  rules:
  - host: altermanager.k8s.grafana.eu.org
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: prometheus-kube-prometheus-alertmanager
            port:
              number: 9093

http01 方式

HTTP-01 的校验方式的优点是: 配置简单通用，不管使用哪个 DNS 提供商都可以使用相同的配置方法；缺点是：需要依赖 Ingress，如果你的服务不是用 Ingress 暴露流量的就不适用，而且不支持泛域名证书

首先需要 dns 解析到 ip 。 当前已添加A记录 *.k8s.grafana.eu.org xxx.xxx.xxx.xxx

使用 letsencrypt-http01

1
2
3
4
5
6
7
8
9
10
11
12
13
14
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-http01
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-http01
    solvers:
    - http01:
        ingress:
          class: nginx

然后在 ingress 中引用 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: prometheus-ingress
  namespace: monitor
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-http01"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - prometheus.k8s.grafana.eu.org
    secretName: prometheus-tls-secret
  rules:
  - host: prometheus.k8s.grafana.eu.org
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: prometheus-kube-prometheus-prometheus
            port:
              number: 9090

执行后会生成一个临时的 ingress 用于验证，完成后则可以访问 https://prometheus.k8s.grafana.eu.org

1
2
3
imwl@ubuntu:~$ kubectl get  ingress -A
monitor       cm-acme-http-solver-5d7v6   <none>   prometheus.k8s.grafana.eu.org                 80        9s
monitor       prometheus-ingress             nginx    prometheus.k8s.grafana.eu.org   10.68.56.59   80, 443   3d

cm-acme-http-solver-5d7v6 验证完成后会自动删除

或者手动创建

需要先有 ingress 使用 prometheus.k8s.grafana.eu.org

1
2
3
4
5
6
7
8
9
10
11
12
13
14
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: prometheus-k8s-grafana-eu-org
  namespace: monitor
spec:
  secretName: prometheus-tls-secret # 生成的证书将保存在此 Secret 中
  issuerRef:
    name: letsencrypt-http01
    kind: ClusterIssuer    # 如果使用 Issuer 则改为 Issuer
  commonName: "prometheus.k8s.grafana.eu.org"
  dnsNames:
  - prometheus.k8s.grafana.eu.org
  1. 使用多域名证书（SAN Certificate）
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: growthbook-growthbook
  namespace: default
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-http01"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
      - growthbook-admin.k8s.grafana.eu.org
      - growthbook-api.k8s.grafana.eu.org
      secretName: tls-growthbook.k8s.grafana.eu.org
  rules:
    - host: growthbook-admin.k8s.grafana.eu.org
      http:
        paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: growthbook-growthbook
                port:
                  name: app
    - host: growthbook-api.k8s.grafana.eu.org
      http:
        paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: growthbook-growthbook
                port:
                  name: api