This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Concepts

1: PediaCluster
2: Public Configuration of Cluster Sync Resources(ClusterSyncResources)
3: Collection Resource
4: Cluster Auto Import Policy

1 - PediaCluster

Clusterpedia uses the PediaCluster resource to represent a cluster that needs to synchronize and retrieve resources.

Clusterpedia needs to be very friendly to other multi-cloud platforms that may use Cluster resource to represent managed clusters, to avoid conflicts Clusterpedia uses PediaCluster.

$ kubectl get pediacluster
NAME        APISERVER                        VERSION            STATUS
demo1       https://10.6.101.100:6443        v1.22.3-aliyun.1   Healthy
demo2       https://10.6.101.100:6443        v1.21.0            Healthy

apiVersion: cluster.clusterpedia.io/v1alpha2
kind: PediaCluster
metadata:
  name: demo1
spec:
  apiserver: https://10.6.101.100:6443
  kubeconfig:
  caData:
  tokenData:
  certData:
  keyData:
  syncResources: []
  syncAllCustomResources: false
  syncResourcesRefName: ""

PediaCluster has two uses:

Configure authentication information for the cluster
Configure resources for synchronization

Configuring cluster authentication information can be found in Import Clusters

There are three fields to configure the resources to be synchronized.

spec.syncResources configures the resources that need to be synchronized for this cluster
spec.syncAllCustomResources synchronizes all custom resources
spec.syncResourcesRefName references Public Configuration of Cluster Sync Resources

For details on configuring synchronization resources, see Synchronize Cluster Resources

2 - Public Configuration of Cluster Sync Resources(ClusterSyncResources)

The Clusterpedia provides the public configuration of cluster sync resources —— ClusterSyncResources

kubectl get clustersyncresources

The spec.syncResources field of ClusterSyncResources is configured in the same way as PediaCluster’s spec.syncResources, see Synchronize Cluster Resources

apiVersion: cluster.clusterpedia.io/v1alpha2
kind: ClusterSyncResources
metadata:
  name: global-base
spec:
  syncResources:
    - group: ""
      resources:
        - pods
    - group: "apps"
      resources:
        - "*"

The spec.syncAllCustomResource field will be supported in the future to support setting up the synchronization of all custom resources

Any PediaCluster can refer to the same ClusterSyncResources via spec.syncResourcesRefName field.

apiVersion: cluster.clusterpedia.io/v1alpha2
kind: PediaCluster
metadata:
  name: demo1
spec:
  syncResourcesRefName: "global-base"

When we modify ClusterSyncResources, all resource types syncronized within the PediaCluster that reference it will be modified accordingly.

If PediaCluster has both spec.syncResourcesRefName and spec.syncResources set, then the concatenation of the two will be used.

apiVersion: cluster.clusterpedia.io/v1alpha2
kind: PediaCluster
metadata:
  name: demo1
spec:
  syncResourcesRefName: "global-base"
  syncResources:
    - group: ""
      resources:
        - pods
        - configmaps

In the above example, clusterpedia synchronizes the pods and configmaps resources, and all resources under the apps group in the demo1 cluster.

3 - Collection Resource

In order to query multiple types of resources at once, Clusterpedia provides a new resource: Collection Resource.

Collection Resource is composed of different types of resources, and these resources can be retrieved and paged in a uniform way through the Collection Resource.

What Collection Resources are supported by the Clusterpedia depends on the Storage Layer. For example, the Default Storage Layer temporarily supports the any, workloads and kuberesources.

kubectl get collectionresources

# Output:
NAME            RESOURCES
any             *
workloads       deployments.apps,daemonsets.apps,statefulsets.apps
kuberesources   .*,*.admission.k8s.io,*.admissionregistration.k8s.io,*.apiextensions.k8s.io,*.apps,*.authentication.k8s.io,*.authorization.k8s.io,*.autoscaling,*.batch,*.certificates.k8s.io,*.coordination.k8s.io,*.discovery.k8s.io,*.events.k8s.io,*.extensions,*.flowcontrol.apiserver.k8s.io,*.imagepolicy.k8s.io,*.internal.apiserver.k8s.io,*.networking.k8s.io,*.node.k8s.io,*.policy,*.rbac.authorization.k8s.io,*.scheduling.k8s.io,*.storage.k8s.io

any means any resources, the use need to pass the groups or resources he wants to combine when using it, for details see Use Any CollectionResource

kuberesources contains all of kube’s built-in resources, and we can use kuberesources to filter and search all of theme in a uniform api.

View the supported Collection Resource in a yaml file

kubectl get collectionresources -o yaml

# Output:
apiVersion: v1
items:
- apiVersion: clusterpedia.io/v1beta1
  kind: CollectionResource
  metadata:
    creationTimestamp: null
    name: any
  resourceTypes: []
- apiVersion: clusterpedia.io/v1beta1
  kind: CollectionResource
  metadata:
    creationTimestamp: null
    name: workloads
  resourceTypes:
  - group: apps
    resource: deployments
    version: v1
  - group: apps
    resource: daemonsets
    version: v1
  - group: apps
    resource: statefulsets
    version: v1
- apiVersion: clusterpedia.io/v1beta1
  kind: CollectionResource
  metadata:
    creationTimestamp: null
    name: kuberesources
  resourceTypes:
  - group: ""
  - group: admission.k8s.io
  - group: admissionregistration.k8s.io
  - group: apiextensions.k8s.io
  - group: apps
  - group: authentication.k8s.io
  - group: authorization.k8s.io
  - group: autoscaling
  - group: batch
  - group: certificates.k8s.io
  - group: coordination.k8s.io
  - group: discovery.k8s.io
  - group: events.k8s.io
  - group: extensions
  - group: flowcontrol.apiserver.k8s.io
  - group: imagepolicy.k8s.io
  - group: internal.apiserver.k8s.io
  - group: networking.k8s.io
  - group: node.k8s.io
  - group: policy
  - group: rbac.authorization.k8s.io
  - group: scheduling.k8s.io
  - group: storage.k8s.io
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

It is found that workloads includes three resources: deployments, daemonsets, and statefulsets.

And kuberesources contains all of kube’s built-in resources.

For details about Collection Resource, see Search for Collection Resource

Custom Collection Resource

Clusterpedia plans to provide two ways to let users combine the types of resources they want to query at will.

Any CollectionResource —— use any collectionresource
CustomCollectionResource —— custom collection resource

After 0.4, Clusterpedia provides any collectionresource to allow users to combine defferent types of resources by passing groups and resources parameters.

However, it should be noted that any collectionresource cannot be retrieved using kubectl, see Using Any CollectionResource

$ kubectl get collectionresources any
Error from server (BadRequest): url query - `groups` or `resources` is required

Custom Collection Resource allows users to create or update a Collection Resource via kubectl apply collectionresource <collectionresource name>, and users can configure the resource type of the Collection Resource at will.

apiVersion: clusterpedia.io/v1beta1
kind: CollectionResource
metadata:
  name: workloads
resourceTypes:
- group: apps
  resource: deployments
- group: apps
  resource: daemonsets
- group: apps
  resource: statefulsets
- group: batch
  resource: cronjobs

Custom Collection Resource are not currently supported

4 - Cluster Auto Import Policy

The custom resource ClusterImportPolicy defines how a certain type of resource should be converted into a PediaCluster, so that clusterpedia can automatically create, update and delete PediaCluster based on a certain resource.

First we need to define the type of resource (that you want to convert) to watch to in the ClusterImportPolicy resource , we will call the resource being watched to the Source resource.

When a Source resource is created or deleted, the ClusterImportPolicy Controller creates the corresponding PediaClusterLifecycle resource.

The custom resource PediaClusterLifecycle creates, updates and deletes PediaCluster based on the specific Source resource.

When creating and updating a PediaCluster, the Source resource may store the cluster’s authentication information (eg. CA, Token, etc.) in other resources, which are collectively referred to as Reference resources.

The reconciliation of ClusterImportPolicy and PediaClusterLifecycle is done by the ClusterImportPolicy Controller and PediaClusterLifecycle Controller within the Clusterpedia Controller Manager

ClusterImportPolicy and PediaClusterLifecycle resource structures may be updated more frequently, so to avoid unnecessary impact on the cluster.clusterpedia.io group, they are placed in the policy.clusterpedia.io group. It may be migrated to cluster.clusterpedia.io in the future 1.0 release.

ClusterImportPolicy

An example of a complete ClusterImportPolicy resource is as follows:

apiVersion: policy.clusterpedia.io/v1alpha1
kind: ClusterImportPolicy
metadata:
  name: mcp
spec:
  source:
    group: "cluster.example.io"
    resource: clusters
    selectorTemplate: ""
  references:
    - group: ""
      resource: secrets
      namespaceTemplate: "{{ .source.spec.authSecretRef.namespace }}"
      nameTemplate: "{{ .source.spec.authSecretRef.name }}"
      key: authSecret
  nameTemplate: "mcp-{{ .source.metadata.name }}"
  template: |
    spec:
      apiserver: "{{ .source.spec.apiEndpoint }}"
      caData: "{{ .references.authSecret.data.ca }}"
      tokenData: "{{ .references.authSecret.data.token }}"
      syncResources:
        - group: ""
          resources:
          - "pods"
        - group: "apps"
          resources:
          - "*"    
  creationCondition: |
    {{ if ne .source.spec.apiEndpoint "" }}
      {{ range .source.status.conditions }}
        {{ if eq .type "Ready" }}
          {{ if eq .status "True" }} true {{ end }}
        {{ end }}
      {{ end }}
    {{ end }}

There are these sections:

spec.source and spec.references define Source resource type and Reference resources
spec.nameTemplate defines the name of the generated PediaClusterLifecycle and PediaCluster, which can be rendered according to the Source resource.
spec.template and spec.creationCondition define the resource conversion policy.

Templates within references, template, and creationCondition all support the 70+ template functions provided by sprig, Sprig Function Documentation

Source and References Resource

The first thing we need to define in the ClusterImportPolicy resource is the Source resource type and the Reference resources.

apiVersion: policy.clusterpedia.io/v1alpha1
kind: ClusterImportPolicy
metadata:
  name: multi-cluster-polatform
spec:
  source:
    group: "example.io"
    resource: clusters
    versions: []
    selectorTemplate: |
            {{ if eq .source.metadata.namespace "default" }} true {{ end }}
  references:
    - group: ""
      resource: secrets
      versions: []
      namespaceTemplate: "{{ .source.spec.secretRef.namespace }}"
      nameTemplate: "{{ .source.spec.secretRef.name }}"
      key: secret

The Source resource specifies the resource group and resource name via spec.source.group and spec.source.resource. We can also use spec.source.versions to restrict Source resource versions, by default there is no restriction on Source resource versions

A Source resource can only have one ClusterImportPolicy resource responsible for converting that resource

You can also filter the Source by the spec.source.selectorTemplate field.

apiVersion: policy.clusterpedia.io/v1alpha1
kind: ClusterImportPolicy
metadata:
  name: kubevela
spec:
  source:
    group: ""
    resource: secrets
    selectorTemplate: |
      {{ if eq .source.metadata.namespace "vela-system" }}
        {{ if .source.metadata.labels }}
          {{ eq (index .source.metadata.labels "cluster.core.oam.dev/cluster-credential-type") "X509Certificate" }}
        {{ end }}
      {{ end }}

source resources can be used for other template fields via {{ .source.<field> }}

The resources involved in the conversion process are defined in spec.references and we need to specify the type of the resource and the specific reference resource via namespace and name templates. We can also restrict the version of the Reference resource in the same way as the Source resource.

In addition to this, we need to set a key for the reference resource, which will be used by subsequent template fields as {{ .references.<key> }}

*The later items in *spec.references can also refer to the earlier ones via .references.<key>

spec:
  references:
  - group: a.example.io
    resource: aresource
    namespaceTemplate: "{{ .source.spec.aNamespace }}"
    nameTemplate: "{{ .source.spec.aName }}"
    key: refA
  - group: b.example.io
    resource: bresource
    namespaceTemplate: "{{ .references.refA.spec.bNamespace }}"
    nameTemplate: "{{ .references.refA.spec.bName }}"
    key: refB

PediaClusterLifecycle

When a Source is created, the ClusterImportPolicy Controller creates the corresponding PediaClusterLifecycle resource based on the ClusterImportPolicy resource.

The name of the PediaClusterLifecycle resource is set via the spec.nameTemplate field of the ClusterImportPolicy.

apiVersion: policy.clusterpedia.io/v1alpha1
kind: ClusterImportPolicy
metadata:
  name: multi-cluster-platform
spec:
  nameTemplate: "mcp-{{ .source.metadata.namespace }}-{{ .source.metadata.name }}"

The nameTemplate can only render templates based on Source resources, and the field is usually set based on whether it is a Cluster Scoped resource.

nameTemplate: "<prefix>-{{ .source.metadata.namespace}}-{{ .source.metadata.name }}"
nameTemplate: "<prefix>-{{ .source.metadata.name }}"

**nameTemplate is usually prefixed with a multi-cloud platform or other meaningful name, but can of course be set without a prefix. **

The PediaClusterLifecycle sets the specific source resource and references resource and the conversion policy

apiVersion: policy.clusterpedia.io/v1alpha1
kind: PediaClusterLifecycle
metadata:
  name: <prefix>-example
spec:
  source:
    group: example.io
    version: v1beta1
    resource: clusters
    namespace: ""
    name: example
  references:
    - group: ""
      resource: secrets
      version: v1
      namespaceTemplate: "{{ .source.spec.secretRef.namespace }}"
      nameTemplate: "{{ .source.spec.secretRef.name }}"
      key: secret

The spec.source of the PediaClusterLifecycle sets a specific Source resource, including the specific version, namespace and name of the resource.

spec.references contains the specific resource version compared to ClusterImportPolicy, the other fields are the same as the References definition within ClusterImportPolicy.

The namespace and name of the references resource will be resolved when converting the resource.

PediaClusterLifecycle 和 PediaCluster

The name of PediaClusterLifecycle corresponds to PediaCluster, and PediaClusterLifecycle creates and updates PediaCluster with the same name according to the conversion policy.

PediaCluster Convertion Policy

We focus on the following aspects when defining the conversion policy:

the template used to create or update the PediaCluster
When to trigger the creation of a `PediaCluster

In ClusterImportPolicy, we use spec.template and spec.creationCondition to define them

apiVersion: policy.clusterpedia.io/v1alpha1
kind: ClusterImportPolicy
metadata:
  name: mcp
spec:
... other fields
  template: |
    spec:
      apiserver: "{{ .source.spec.apiEndpoint }}"
      caData: "{{ .references.authSecret.data.ca }}"
      tokenData: "{{ .references.tokenData.data.token }}"
      syncResources:
        - group: ""
          resources:
          - "pods"
        - group: "apps"
          resources:
          - "*"    
  creationCondition: |
    {{ if ne .source.spec.apiEndpoint "" }}
      {{ range .source.status.conditions }}
        {{ if eq .type "Ready" }}
          {{ if eq .status "True" }} true {{ end }}
        {{ end }}
      {{ end }}
    {{ end }}

Both of these fields are template fields that need to be rendered based on Source resource and References resources.

When the Source resource is created, the ClusterImportPolicy Controller will create the corresponding PediaClusterLifecycle based on the ClusterImportPolicy, and the PediaClusterLifecycle will will also contain the conversion policy.

Of course, if the policy in ClusterImportPolicy is modified, it will be synchronized to all PediaClusterLifecycle resources it belongs to.

apiVersion: policy.clusterpedia.io/v1alpha1
kind: ClusterImportPolicy
metadata:
  name: mcp-example
spec:
... other fields
  template: |
    spec:
      apiserver: "{{ .source.spec.apiEndpoint }}"
      caData: "{{ .references.authSecret.data.ca }}"
      tokenData: "{{ .references.tokenData.data.token }}"
      syncResources:
        - group: ""
          resources:
          - "pods"
        - group: "apps"
          resources:
          - "*"    
  creationCondition: |
    {{ if ne .source.spec.apiEndpoint "" }}
      {{ range .source.status.conditions }}
        {{ if eq .type "Ready" }}
          {{ if eq .status "True" }} true {{ end }}
        {{ end }}
      {{ end }}
    {{ end }}

PediaClusterLifecycleis responsible for creating and updating specificPediaClustersbased onspec.creationConditionandspec.template`.

Creation Condition

Sometimes we don’t create a PediaCluster immediately after a Source resource is created, but rather we need to wait until some fields or some state of the Source resource is ready before creating the PediaCluster.

spec.creationCondition uses the template syntax to determine if the creation condition is met, and when the template is rendered with a value of True (case insensitive) then PediaCluster is created according to `spec.

If the PediaCluster already exists, spec.creationCondition will not affect updates to the PediaCluster.

PediaCluster Template

spec.template defines a PediaCluster resource template that renders specific resources based on Source resource and References resources when creating or updating a PediaCluster.

The PediaCluster template can be separated into three parts.

Metadata: labels and annotations
Cluster endpoint and authentication fields: spec.apiserver, spec.caData, spec.tokenData, spec.certData, spec.keyData and spec.kubeconfig
Resource sync fields: spec.syncResources, spec.syncAllCustomResources, spec.syncResourcesRefName

The metadata and resource sync fields of the PediaCluster resource are only available when the PediaCluser is created, and only the cluster endpoint and authentication fields are updated when the PediaCluster resource is updated

PediaCluster Deletion Condition

If a PediaCluster is created by PediaClusterLifecycle, then PediaClusterLifecycle will be set to the owner of that PediaCluster resource.

apiVersion: cluster.clusterpedia.io/v1alpha2
kind: PediaCluster
metadata:
  name: mcp-example
  ownerReferences:
  - apiVersion: policy.clusterpedia.io/v1alpha1
    kind: PediaClusterLifecycle
    name: mcp-example
    uid: f932483a-b1c5-4894-a524-00f78ea34a9f

When a Source resource is deleted, the PediaClusterLifecycle is deleted at the same time and the PediaCluster is deleted automatically.

If PediaCluster already existed before PediaClusterLifecycle, then Source will not automatically delete PediaCluster when it is deleted.

DeletionCondition will be added in the future to allow users to force or preempt the deletion of PediaCluster.