Edit This Page

在联邦中设置放置策略

已过时

强烈建议不要使用联邦 v1 版本，联邦 v1 版本从未达到 GA 状态，且不再处于积极开发阶段。文档仅作为历史参考。

有关更多信息，请参阅预期的替代品 Kubernetes 联邦 v2 版本。

此页面显示如何使用外部策略引擎对联邦资源强制执行基于策略的放置决策。

准备开始

您需要一个正在运行的 Kubernetes 集群(它被引用为主机集群)。有关您的平台的安装说明，请参阅入门指南。

Deploying 联邦并配置外部策略引擎

可以使用 kubefed init 部署联邦控制平面。

Deploying 联邦控制平面之后，必须在联邦 API 服务器中配置一个准入控制器，该控制器强制执行从外部策略引擎接收到的放置决策。

kubectl create -f scheduling-policy-admission.yaml

下图是准入控制器的 ConfigMap 示例：

federation/scheduling-policy-admission.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: admission
  namespace: federation-system
data:
  config.yml: |
    apiVersion: apiserver.k8s.io/v1alpha1
    kind: AdmissionConfiguration
    plugins:
    - name: SchedulingPolicy
      path: /etc/kubernetes/admission/scheduling-policy-config.yml
  scheduling-policy-config.yml: |
    kubeconfig: /etc/kubernetes/admission/opa-kubeconfig
  opa-kubeconfig: |
    clusters:
      - name: opa-api
        cluster:
          server: http://opa.federation-system.svc.cluster.local:8181/v0/data/kubernetes/placement
    users:
      - name: scheduling-policy
        user:
          token: deadbeefsecret
    contexts:
      - name: default
        context:
          cluster: opa-api
          user: scheduling-policy
    current-context: default

ConfigMap 包含三个文件：

  • config.yml 指定 调度策略 准入控制器配置文件的位置。
  • scheduling-policy-config.yml 指定与外部策略引擎联系所需的 kubeconfig 文件的位置。 该文件还可以包含一个 retryBackoff 值，该值以毫秒为单位控制初始重试 backoff 延迟。
  • opa-kubeconfig 是一个标准的 kubeconfig，包含联系外部策略引擎所需的 URL 和凭证。

编辑联邦 API 服务器部署以启用 SchedulingPolicy 准入控制器。

kubectl -n federation-system edit deployment federation-apiserver

更新 Federation API 服务器命令行参数以启用准入控制器， 并将 ConfigMap 挂载到容器中。如果存在现有的 -enable-admissionplugins 参数，则追加 SchedulingPolicy 而不是添加另一行。

--enable-admission-plugins=SchedulingPolicy
--admission-control-config-file=/etc/kubernetes/admission/config.yml

将以下卷添加到联邦 API 服务器 pod：

- name: admission-config
  configMap:
    name: admission

添加以下卷挂载联邦 API 服务器的 apiserver 容器：

volumeMounts:
- name: admission-config
  mountPath: /etc/kubernetes/admission

Deploying 外部策略引擎

Open Policy Agent (OPA) 是一个开源的通用策略引擎， 您可以使用它在联邦控制平面中执行基于策略的放置决策。

在主机群集中创建服务以联系外部策略引擎：

kubectl create -f policy-engine-service.yaml

下面显示的是 OPA 的示例服务。

federation/policy-engine-service.yaml 
kind: Service
apiVersion: v1
metadata:
  name: opa
  namespace: federation-system
spec:
  selector:
    app: opa
  ports:
  - name: http
    protocol: TCP
    port: 8181
    targetPort: 8181

使用联邦控制平面在主机群集中创建部署：

kubectl create -f policy-engine-deployment.yaml

下面显示的是 OPA 的部署示例。

federation/policy-engine-deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: opa
  name: opa
  namespace: federation-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: opa
  template:
    metadata:
      labels:
        app: opa
      name: opa
    spec:
      containers:
        - name: opa
          image: openpolicyagent/opa:0.4.10
          args:
          - "run"
          - "--server"
        - name: kube-mgmt
          image: openpolicyagent/kube-mgmt:0.2
          args:
          - "-kubeconfig=/srv/kubernetes/kubeconfig"
          - "-cluster=federation/v1beta1/clusters"
          volumeMounts:
           - name: federation-kubeconfig
             mountPath: /srv/kubernetes
             readOnly: true
      volumes:
      - name: federation-kubeconfig
        secret:
          secretName: federation-controller-manager-kubeconfig

通过 ConfigMaps 配置放置策略

外部策略引擎将发现在 Federation API 服务器的 kube-federation-scheduling-policy 命名空间中创建的放置策略。

如果命名空间尚不存在，请创建它：

kubectl --context=federation create namespace kube-federation-scheduling-policy

配置一个示例策略来测试外部策略引擎:

# OPA supports a high-level declarative language named Rego for authoring and
# enforcing policies. For more information on Rego, visit
# http://openpolicyagent.org.

# Rego policies are namespaced by the "package" directive.
package kubernetes.placement

# Imports provide aliases for data inside the policy engine. In this case, the
# policy simply refers to "clusters" below.
import data.kubernetes.clusters

# The "annotations" rule generates a JSON object containing the key
# "federation.kubernetes.io/replica-set-preferences" mapped to <preferences>.
# The preferences values is generated dynamically by OPA when it evaluates the
# rule.
#
# The SchedulingPolicy Admission Controller running inside the Federation API
# server will merge these annotations into incoming Federated resources. By
# setting replica-set-preferences, we can control the placement of Federated
# ReplicaSets.
#
# Rules are defined to generate JSON values (booleans, strings, objects, etc.)
# When OPA evaluates a rule, it generates a value IF all of the expressions in
# the body evaluate successfully. All rules can be understood intuitively as
# <head> if <body> where <body> is true if <expr-1> AND <expr-2> AND ...
# <expr-N> is true (for some set of data.)
annotations["federation.kubernetes.io/replica-set-preferences"] = preferences {
    input.kind = "ReplicaSet"
    value = {"clusters": cluster_map, "rebalance": true}
    json.marshal(value, preferences)
}

# This "annotations" rule generates a value for the "federation.alpha.kubernetes.io/cluster-selector"
# annotation.
#
# In English, the policy asserts that resources in the "production" namespace
# that are not annotated with "criticality=low" MUST be placed on clusters
# labelled with "on-premises=true".
annotations["federation.alpha.kubernetes.io/cluster-selector"] = selector {
    input.metadata.namespace = "production"
    not input.metadata.annotations.criticality = "low"
    json.marshal([{
        "operator": "=",
        "key": "on-premises",
        "values": "[true]",
    }], selector)
}

# Generates a set of cluster names that satisfy the incoming Federated
# ReplicaSet's requirements. In this case, just PCI compliance.
replica_set_clusters[cluster_name] {
    clusters[cluster_name]
    not insufficient_pci[cluster_name]
}

# Generates a set of clusters that must not be used for Federated ReplicaSets
# that request PCI compliance.
insufficient_pci[cluster_name] {
    clusters[cluster_name]
    input.metadata.annotations["requires-pci"] = "true"
    not pci_clusters[cluster_name]
}

# Generates a set of clusters that are PCI certified. In this case, we assume
# clusters are annotated to indicate if they have passed PCI compliance audits.
pci_clusters[cluster_name] {
    clusters[cluster_name].metadata.annotations["pci-certified"] = "true"
}

# Helper rule to generate a mapping of desired clusters to weights. In this
# case, weights are static.
cluster_map[cluster_name] = {"weight": 1} {
    replica_set_clusters[cluster_name]
}

下面显示的是创建示例策略的命令：

kubectl --context=federation -n kube-federation-scheduling-policy create configmap scheduling-policy --from-file=policy.rego

这个示例策略说明了一些关键思想：

  • 位置策略可以引用联邦资源中的任何字段。
  • 放置策略可以利用外部上下文(例如，集群元数据)来做出决策。
  • 管理策略可以集中管理。
  • 策略可以定义简单的接口(例如 requirements -pci 注解)，以避免在清单中重复逻辑。

测试放置政策

注释其中一个集群以表明它是经过 PCI 认证的。

kubectl --context=federation annotate clusters cluster-name-1 pci-certified=true

部署联邦副本来测试放置策略。

federation/replicaset-example-policy.yaml 
apiVersion: apps/v1
kind: ReplicaSet
metadata:
  labels:
    app: nginx-pci
  name: nginx-pci
  annotations:
    requires-pci: "true"
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx-pci
  template:
    metadata:
      labels:
        app: nginx-pci
    spec:
      containers:
      - image: nginx
        name: nginx-pci

下面显示的命令用于部署与策略匹配的副本集。

kubectl --context=federation create -f replicaset-example-policy.yaml

检查副本集以确认已应用适当的注解：

kubectl --context=federation get rs nginx-pci -o jsonpath='{.metadata.annotations}'