The configuration pages below describe optional features that you can add to your EKS Anywhere provider’s clusterspec file. See each provider’s installation section for details on which optional features are supported.
This is the multi-page printable view of this section. Click here to print.
Optional Configuration
- 1: etcd
- 2: Encrypting Confidential Data at Rest
- 3: Operating system
- 4: Container Networking Interface
- 5: IAM Roles for Service Accounts configuration
- 6: IAM Authentication
- 7: OIDC
- 8: Proxy
- 9: MachineHealthCheck
- 10: Registry Mirror
- 11: Autoscaling configuration
- 12: Skipping validations configuration
- 13: GitOps
- 14: Package controller
1 - etcd
Unstacked etcd topology (recommended)
Provider support details
vSphere | Bare Metal | Nutanix | CloudStack | Snow | |
---|---|---|---|---|---|
Supported? | ✓ | ✓ | ✓ | ✓ |
There are two types of etcd topologies for configuring a Kubernetes cluster:
- Stacked: The etcd members and control plane components are colocated (run on the same node/machines)
- Unstacked/External: With the unstacked or external etcd topology, etcd members have dedicated machines and are not colocated with control plane components
The unstacked etcd topology is recommended for a HA cluster for the following reasons:
- External etcd topology decouples the control plane components and etcd member. For example, if a control plane-only node fails, or if there is a memory leak in a component like kube-apiserver, it won’t directly impact an etcd member.
- Etcd is resource intensive, so it is safer to have dedicated nodes for etcd, since it could use more disk space or higher bandwidth. Having a separate etcd cluster for these reasons could ensure a more resilient HA setup.
EKS Anywhere supports both topologies. In order to configure a cluster with the unstacked/external etcd topology, you need to configure your cluster by updating the configuration file before creating the cluster. This is a generic template with detailed descriptions below for reference:
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
name: my-cluster-name
spec:
clusterNetwork:
pods:
cidrBlocks:
- 192.168.0.0/16
services:
cidrBlocks:
- 10.96.0.0/12
cniConfig:
cilium: {}
controlPlaneConfiguration:
count: 1
endpoint:
host: ""
machineGroupRef:
kind: VSphereMachineConfig
name: my-cluster-name-cp
datacenterRef:
kind: VSphereDatacenterConfig
name: my-cluster-name
# etcd configuration
externalEtcdConfiguration:
count: 3
machineGroupRef:
kind: VSphereMachineConfig
name: my-cluster-name-etcd
kubernetesVersion: "1.27"
workerNodeGroupConfigurations:
- count: 1
machineGroupRef:
kind: VSphereMachineConfig
name: my-cluster-name
name: md-0
externalEtcdConfiguration (under Cluster)
External etcd configuration for your Kubernetes cluster.
count (required)
This determines the number of etcd members in the cluster. The recommended number is 3.
machineGroupRef (required)
Refers to the Kubernetes object with provider specific configuration for your nodes.
2 - Encrypting Confidential Data at Rest
You can configure EKS Anywhere clusters to encrypt confidential API resource data, such as secrets
, at-rest in etcd using a KMS encryption provider.
EKS Anywhere supports a hybrid model for configuring etcd encryption where cluster admins are responsible for deploying and maintaining
the KMS provider on the cluster and EKS Anywhere handles configuring kube-apiserver
with the KMS properties.
Because of this model, etcd encryption can only be enabled on cluster upgrades after the KMS provider has been deployed on the cluster.
Note
Currently, etcd encryption is only supported for Nutanix, CloudStack and vSphere. Support for other providers will be added in a future release.Before you begin
Before enabling etcd encryption, make sure you have done the following:
- Learn how encrypting confidential data works in Kubernetes by reading Encrypting Secret Data at Rest.
- Learn how Kubernetes uses KMS encryption providers to encrypt data by reading Using a KMS Provider for data encryption.
- Learn how you can use AWS KMS keys to encrypt data using AWS Encryption Provider.
- Create an EKS Anywhere cluster since encryption can only be enabled on already created clusters.
- Deploy a KMS provider on the cluster that the
kube-apiserver
can use to encrypt/decrypt secrets.
AWS Encryption Provider is the recommended KMS provider for EKS Anywhere.
Example etcd encryption configuration
The following cluster spec enables etcd encryption configuration:
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
name: my-cluster
namespace: default
spec:
...
etcdEncryption:
- providers:
- kms:
cachesize: 1000
name: example-kms-config
socketListenAddress: unix:///var/run/kmsplugin/socket.sock
timeout: 3s
resources:
- secrets
Description of etcd encryption fields
etcdEncryption
Key used to specify etcd encryption configuration for a cluster. This field is only supported on cluster upgrades.
-
providers
Key used to specify which encryption provider to use. Currently, only one provider can be configured.
-
kms
Key used to configure KMS encryption provider.
-
name
Key used to set the name of the KMS plugin. This cannot be changed once set.
-
endpoint
Key used to specify the listen address of the gRPC server (KMS plugin). The endpoint is a UNIX domain socket.
-
cachesize
Number of data encryption keys (DEKs) to be cached in the clear. When cached, DEKs can be used without another call to the KMS; whereas DEKs that are not cached require a call to the KMS to unwrap. If
cachesize
isn’t specified, a default of1000
is used. -
timeout
How long should kube-apiserver wait for kms-plugin to respond before returning an error. If a timeout isn’t specified, a default timeout of
3s
is used.
-
-
-
resources
Key used to specify a list of resources that should be encrypted using the corresponding encryption provider. These can be native Kubernetes resources such as
secrets
andconfigmaps
or custom resource definitions such asclusters.anywhere.eks.amazonaws.com
.
Example AWS Encryption Provider DaemonSet
Here’s a sample AWS encryption provider daemonset configuration.
Note
This example doesn’t include any configuration for AWS credentials to call the KMS APIs.You can configure this daemonset with static creds for an IAM user or use IRSA for a more secure way to authenticate to AWS.
Expand
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
app: aws-encryption-provider
name: aws-encryption-provider
namespace: kube-system
spec:
selector:
matchLabels:
app: aws-encryption-provider
template:
metadata:
labels:
app: aws-encryption-provider
spec:
containers:
- image: <AWS_ENCRYPTION_PROVIDER_IMAGE> # Specify the AWS KMS encryption provider image
name: aws-encryption-provider
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
command:
- /aws-encryption-provider
- --key=<KEY_ARN> # Specify the arn of KMS key to be used for encryption/decryption
- --region=<AWS_REGION> # Specify the region in which the KMS key exists
- --listen=<KMS_SOCKET_LISTEN_ADDRESS> # Specify a socket listen address for the KMS provider. Example: /var/run/kmsplugin/socket.sock
ports:
- containerPort: 8080
protocol: TCP
livenessProbe:
httpGet:
path: /healthz
port: 8080
volumeMounts:
- mountPath: /var/run/kmsplugin
name: var-run-kmsplugin
- mountPath: /root/.aws
name: aws-credentials
tolerations:
- key: "node-role.kubernetes.io/master"
effect: "NoSchedule"
- key: "node-role.kubernetes.io/control-plane"
effect: "NoSchedule"
volumes:
- hostPath:
path: /var/run/kmsplugin
type: DirectoryOrCreate
name: var-run-kmsplugin
- hostPath:
path: /etc/kubernetes/aws
type: DirectoryOrCreate
name: aws-credentials
3 - Operating system
Host OS Configuration
You can configure certain host OS settings through EKS Anywhere.
Provider support details
vSphere | Bare Metal | Nutanix | CloudStack | Snow | |
---|---|---|---|---|---|
Supported? | ✓ | ✓ |
Note
Settings underbottlerocketConfiguration
are only supported for osFamily: bottlerocket
The following cluster spec shows an example of how to configure host OS settings:
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: VSphereMachineConfig # Replace "VSphereMachineConfig" with "TinkerbellMachineConfig" for Tinkerbell clusters
metadata:
name: machine-config
spec:
...
hostOSConfiguration:
ntpConfiguration:
servers:
- time-a.ntp.local
- time-b.ntp.local
certBundles:
- name: "bundle_1"
data: |
-----BEGIN CERTIFICATE-----
MIIF1DCCA...
...
es6RXmsCj...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
bottlerocketConfiguration:
kubernetes:
allowedUnsafeSysctls:
- "net.core.somaxconn"
- "net.ipv4.ip_local_port_range"
clusterDNSIPs:
- 10.96.0.10
maxPods: 100
kernel:
sysctlSettings:
net.core.wmem_max: "8388608"
net.core.rmem_max: "8388608"
...
boot:
bootKernelParameters:
slub_debug:
- "options,slabs"
...
Host OS Configuration Spec Details
hostOSConfiguration
Top level object used for host OS configurations.
-
ntpConfiguration
Key used for configuring NTP servers on your EKS Anywhere cluster nodes.
-
Servers is a list of NTP servers that should be configured on EKS Anywhere cluster nodes.servers
-
-
certBundles
Key used for configuring custom trusted CA certs on your EKS Anywhere cluster nodes. Multiple cert bundles can be configured.
Note
This setting is _only valid_ for Botlerocket OS.
-
name
Name of the cert bundle that should be configured on EKS Anywhere cluster nodes. This must be a unique name for each entry
-
data
Data of the cert bundle that should be configured on EKS Anywhere cluster nodes. This takes in a PEM formatted cert bundle and can contain more than one CA cert per entry.
-
-
bottlerocketConfiguration
Key used for configuring Bottlerocket-specific settings on EKS Anywhere cluster nodes. These settings are only valid for Bottlerocket.
-
kubernetes
Key used for configuring Bottlerocket Kubernetes settings.
-
allowedUnsafeSysctls
List of unsafe sysctls that should be enabled on the node.
-
clusterDNSIPs
List of IPs of DNS service(s) running in the kubernetes cluster.
-
maxPods
Maximum number of pods that can be scheduled on each node.
-
-
kernel
Key used for configuring Bottlerocket Kernel settings.
-
Map of kernel sysctl settings that should be enabled on the node.sysctlSettings
-
-
boot
Key used for configuring Bottlerocket Boot settings.
Note
This setting has not been validated for baremetal.
-
Map of Boot Kernel parameters Bottlerocket should configure.bootKernelParameters
-
-
4 - Container Networking Interface
Specifying CNI Plugin in EKS Anywhere cluster spec
Provider support details
vSphere | Bare Metal | Nutanix | CloudStack | Snow | |
---|---|---|---|---|---|
Supported? | ✓ | ✓ | ✓ | ✓ | ✓ |
EKS Anywhere currently supports two CNI plugins: Cilium and Kindnet. Only one of them can be selected
for a cluster, and the plugin cannot be changed once the cluster is created.
Up until the 0.7.x releases, the plugin had to be specified using the cni
field on cluster spec.
Starting with release 0.8, the plugin should be specified using the new cniConfig
field as follows:
-
For selecting Cilium as the CNI plugin:
apiVersion: anywhere.eks.amazonaws.com/v1alpha1 kind: Cluster metadata: name: my-cluster-name spec: clusterNetwork: pods: cidrBlocks: - 192.168.0.0/16 services: cidrBlocks: - 10.96.0.0/12 cniConfig: cilium: {}
EKS Anywhere selects this as the default plugin when generating a cluster config.
-
Or for selecting Kindnetd as the CNI plugin:
apiVersion: anywhere.eks.amazonaws.com/v1alpha1 kind: Cluster metadata: name: my-cluster-name spec: clusterNetwork: pods: cidrBlocks: - 192.168.0.0/16 services: cidrBlocks: - 10.96.0.0/12 cniConfig: kindnetd: {}
NOTE: EKS Anywhere allows specifying only 1 plugin for a cluster and does not allow switching the plugins after the cluster is created.
Policy Configuration options for Cilium plugin
Cilium accepts policy enforcement modes from the users to determine the allowed traffic between pods.
The allowed values for this mode are: default
, always
and never
.
Please refer the official Cilium documentation
for more details on how each mode affects
the communication within the cluster and choose a mode accordingly.
You can choose to not set this field so that cilium will be launched with the default
mode.
Starting release 0.8, Cilium’s policy enforcement mode can be set through the cluster spec
as follows:
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
name: my-cluster-name
spec:
clusterNetwork:
pods:
cidrBlocks:
- 192.168.0.0/16
services:
cidrBlocks:
- 10.96.0.0/12
cniConfig:
cilium:
policyEnforcementMode: "always"
Please note that if the always
mode is selected, all communication between pods is blocked unless
NetworkPolicy objects allowing communication are created.
In order to ensure that the cluster gets created successfully, EKS Anywhere will create the required
NetworkPolicy objects for all its core components. But it is up to the user to create the NetworkPolicy
objects needed for the user workloads once the cluster is created.
Network policies created by EKS Anywhere for “always” mode
As mentioned above, if Cilium is configured with policyEnforcementMode
set to always
,
EKS Anywhere creates NetworkPolicy objects to enable communication between
its core components. EKS Anywhere will create NetworkPolicy resources in the following namespaces allowing all ingress/egress traffic by default:
- kube-system
- eksa-system
- All core Cluster API namespaces:
- capi-system
- capi-kubeadm-bootstrap-system
- capi-kubeadm-control-plane-system
- etcdadm-bootstrap-provider-system
- etcdadm-controller-system
- cert-manager
- Infrastructure provider’s namespace (for instance, capd-system OR capv-system)
- If Gitops is enabled, then the gitops namespace (flux-system by default)
This is the NetworkPolicy that will be created in these namespaces for the cluster:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-all-ingress-egress
namespace: test
spec:
podSelector: {}
ingress:
- {}
egress:
- {}
policyTypes:
- Ingress
- Egress
Switching the Cilium policy enforcement mode
The policy enforcement mode for Cilium can be changed as a part of cluster upgrade through the cli upgrade command.
-
Switching to
always
mode: When switching fromdefault
/never
toalways
mode, EKS Anywhere will create the required NetworkPolicy objects for its core components (listed above). This will ensure that the cluster gets upgraded successfully. But it is up to the user to create the NetworkPolicy objects required for the user workloads. -
Switching from
always
mode: When switching fromalways
todefault
mode, EKS Anywhere will not delete any of the existing NetworkPolicy objects, including the ones required for EKS Anywhere components (listed above). The user must delete NetworkPolicy objects as needed.
EgressMasqueradeInterfaces option for Cilium plugin
Cilium accepts the EgressMasqueradeInterfaces
option from users to limit which interfaces masquerading is performed on.
The allowed values for this mode are an interface name
such as eth0
or an interface prefix
such as eth+
.
Please refer to the official Cilium documentation
for more details on how this option affects masquerading traffic.
By default, masquerading will be performed on all traffic leaving on a non-Cilium network device. This only has an effect on traffic egressing from a node to an external destination not part of the cluster and does not affect routing decisions.
This field can be set as follows:
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
name: my-cluster-name
spec:
clusterNetwork:
pods:
cidrBlocks:
- 192.168.0.0/16
services:
cidrBlocks:
- 10.96.0.0/12
cniConfig:
cilium:
egressMasqueradeInterfaces: "eth0"
RoutingMode option for Cilium plugin
By default all traffic is sent by Cilium over Geneve tunneling on the network. The routingMode
option allows users to switch to native routing
instead.
This field can be set as follows:
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
name: my-cluster-name
spec:
clusterNetwork:
pods:
cidrBlocks:
- 192.168.0.0/16
services:
cidrBlocks:
- 10.96.0.0/12
cniConfig:
cilium:
routingMode: "direct"
Use a custom CNI
EKS Anywhere can be configured to skip EKS Anywhere’s default Cilium CNI upgrades via the skipUpgrade
field.
skipUpgrade
can be true
or false
. When not set, it defaults to false
.
When creating a new cluster with skipUpgrade
enabled, EKS Anywhere Cilium will be installed as it
is required to successfully provision an EKS Anywhere cluster.
When the cluster successfully provisions, EKS Anywhere Cilium may be uninstalled and replaced with
a different CNI.
Subsequent upgrades to the cluster will not attempt to upgrade or re-install EKS Anywhere Cilium.
Once enabled, skipUpgrade
cannot be disabled.
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
name: my-cluster-name
spec:
clusterNetwork:
pods:
cidrBlocks:
- 192.168.0.0/16
services:
cidrBlocks:
- 10.96.0.0/12
cniConfig:
cilium:
skipUpgrade: true
The Cilium CLI
can be used to uninstall EKS Anywhere Cilium
via cilium uninstall
.
See the replacing Cilium task
for a walkthrough on how to successfully replace EKS Anywhere Cilium.
Warning
When uninstalling EKS Anywhere Cilium, nodes will become unhealthy. If nodes are left without a CNI for longer than 5m the nodes will begin rolling. To maintain a healthy cluster, operators should immediately install a CNI after uninstalling EKS Anywhere Cilium.Warning
Clusters created using the Full Lifecycle Controller prior to v0.15 that have removed the EKS Anywhere Cilium CNI must manually populate their cluster.anywhere.eks.amazonaws.com
object with the following annotation to ensure EKS Anywhere does not attempt to re-install EKS Anywhere Cilium.
anywhere.eks.amazonaws.com/eksa-cilium: ""
Node IPs configuration option
Starting with release v0.10, the node-cidr-mask-size
flag
for Kubernetes controller manager (kube-controller-manager) is configurable via the EKS anywhere cluster spec. The clusterNetwork.nodes
being an optional field,
is not generated in the EKS Anywhere spec using generate clusterconfig
command. This block for nodes
will need to be manually added to the cluster spec under the
clusterNetwork
section:
clusterNetwork:
pods:
cidrBlocks:
- 192.168.0.0/16
services:
cidrBlocks:
- 10.96.0.0/12
cniConfig:
cilium: {}
nodes:
cidrMaskSize: 24
If the user does not specify the clusterNetwork.nodes
field in the cluster yaml spec, the value for this flag defaults to 24 for IPv4.
Please note that this mask size needs to be greater than the pods CIDR mask size. In the above spec, the pod CIDR mask size is 16
and the node CIDR mask size is 24
. This ensures the cluster 256 blocks of /24 networks. For example, node1 will get
192.168.0.0/24, node2 will get 192.168.1.0/24, node3 will get 192.168.2.0/24 and so on.
To support more than 256 nodes, the cluster CIDR block needs to be large, and the node CIDR mask size needs to be small, to support that many IPs. For instance, to support 1024 nodes, a user can do any of the following things
- Set the pods cidr blocks to
192.168.0.0/16
and node cidr mask size to 26 - Set the pods cidr blocks to
192.168.0.0/15
and node cidr mask size to 25
Please note that the node-cidr-mask-size
needs to be large enough to accommodate the number of pods you want to run on each node.
A size of 24 will give enough IP addresses for about 250 pods per node, however a size of 26 will only give you about 60 IPs.
This is an immutable field, and the value can’t be updated once the cluster has been created.
5 - IAM Roles for Service Accounts configuration
IAM Role for Service Account on EKS Anywhere clusters with self-hosted signing keys
IAM Roles for Service Account (IRSA) enables applications running in clusters to authenticate with AWS services using IAM roles. The current solution for leveraging this in EKS Anywhere involves creating your own OIDC provider for the cluster, and hosting your cluster’s public service account signing key. The public keys along with the OIDC discovery document should be hosted somewhere that AWS STS can discover it.
The steps below are based on the guide for configuring IRSA for DIY Kubernetes, with modifications specific to EKS Anywhere’s cluster provisioning workflow. The main modification is the process of generating the keys.json document. As per the original guide, the user has to create the service account signing keys, and then use that to create the keys.json document prior to cluster creation. This order is reversed for EKS Anywhere clusters, so you will create the cluster first, and then retrieve the service account signing key generated by the cluster, and use it to create the keys.json document. The sections below show how to do this in detail.
Create an OIDC provider and make its discovery document publicly accessible
You must use a single OIDC provider per EKS Anywhere cluster, which is the best practice to prevent a token from one cluster being used with another cluster. These steps describe the process of using a S3 bucket to host the OIDC discovery.json
and keys.json
documents.
-
Create an S3 bucket to host the public signing keys and OIDC discovery document for your cluster . Make a note of the
$HOSTNAME
and$ISSUER_HOSTPATH
. -
Create the OIDC discovery document as follows:
cat <<EOF > discovery.json { "issuer": "https://$ISSUER_HOSTPATH", "jwks_uri": "https://$ISSUER_HOSTPATH/keys.json", "authorization_endpoint": "urn:kubernetes:programmatic_authorization", "response_types_supported": [ "id_token" ], "subject_types_supported": [ "public" ], "id_token_signing_alg_values_supported": [ "RS256" ], "claims_supported": [ "sub", "iss" ] } EOF
-
Upload the
discovery.json
file to the S3 bucket:aws s3 cp ./discovery.json s3://$S3_BUCKET/.well-known/openid-configuration
-
Create an OIDC provider for your cluster. Set the Provider URL to
https://$ISSUER_HOSTPATH
and Audience tosts.amazonaws.com
. -
Make a note of the
Provider
field of OIDC provider after it is created. -
Assign an IAM role to the OIDC provider.
-
Navigate to the AWS IAM Console.
-
Click on the OIDC provider.
-
Click Assign role.
-
Select Create a new role.
-
Select Web identity as the trusted entity.
-
In the Web identity section:
- If your Identity provider is not auto selected, select it.
- Select
sts.amazonaws.com
as the Audience.
-
Click Next.
-
Configure your desired Permissions poilicies.
-
Below is a sample trust policy of IAM role for your pods. Replace
ACCOUNT_ID
,ISSUER_HOSTPATH
,NAMESPACE
andSERVICE_ACCOUNT
. Example: Scoped to a service account{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::ACCOUNT_ID:oidc-provider/ISSUER_HOSTPATH" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "ISSUER_HOSTPATH:sub": "system:serviceaccount:NAMESPACE:SERVICE_ACCOUNT" }, } } ] }
-
Create the IAM Role and make a note of the Role name.
-
After the cluster is created you can grant service accounts access to the role by modifying the trust relationship. See the How to use trust policies with IAM Roles for more information on trust policies. Refer to Configure the trust relationship for the OIDC provider’s IAM Role for a working example.
-
Create (or upgrade) the EKS Anywhere cluster
When creating (or upgrading) the EKS Anywhere cluster, you need to configure the kube-apiserver’s service-account-issuer
flag so it can issue and mount projected service account tokens in pods. For this, use the value obtained in the first section for $ISSUER_HOSTPATH
as the service-account-issuer
. Configure the kube-apiserver by setting this value through the EKS Anywhere cluster spec:
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
name: my-cluster-name
spec:
podIamConfig:
serviceAccountIssuer: https://$ISSUER_HOSTPATH
Set the remaining fields in cluster spec as required and create the cluster.
Generate keys.json and make it publicly accessible
-
The cluster provisioning workflow generates a pair of service account signing keys. Retrieve the public signing key from the cluster and create a
keys.json
document with the content.git clone https://github.com/aws/amazon-eks-pod-identity-webhook cd amazon-eks-pod-identity-webhook kubectl get secret ${CLUSTER_NAME}-sa -n eksa-system -o jsonpath={.data.tls\\.crt} | base64 --decode > ${CLUSTER_NAME}-sa.pub go run ./hack/self-hosted/main.go -key ${CLUSTER_NAME}-sa.pub | jq '.keys += [.keys[0]] | .keys[1].kid = ""' > keys.json
-
Upload the
keys.json
document to the S3 bucket.aws s3 cp ./keys.json s3://$S3_BUCKET/keys.json
-
Use a bucket policy to grant public read access to the
discovery.json
andkeys.json
documents:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": [
"arn:aws:s3:::$S3_BUCKET/.well-known/openid-configuration",
"arn:aws:s3:::$S3_BUCKET/keys.json"
]
}
]
}
Deploy pod identity webhook
The Amazon Pod Identity Webhook
configures pods with the necessary environment variables and tokens (via file mounts) to interact with AWS services. The webhook will configure any pod associated with a service account that has an eks-amazonaws.com/role-arn
annotation.
-
Clone amazon-eks-pod-identity-webhook .
-
Set the $KUBECONFIG environment variable to the path of the EKS Anywhere cluster.
-
Apply the manifests for the
amazon-eks-pod-identity-webhook
. The image used here will be pulled from docker.io. Optionally, the image can be imported into (or proxied through) your private registry. Change theIMAGE
argument here to your private registry if needed.make cluster-up IMAGE=amazon/amazon-eks-pod-identity-webhook:latest
-
Create a service account with an
eks.amazonaws.com/role-arn
annotation set to the IAM Role created for the OIDC provider.apiVersion: v1 kind: ServiceAccount metadata: name: my-serviceaccount namespace: default annotations: # set this with value of OIDC_IAM_ROLE eks.amazonaws.com/role-arn: "arn:aws:iam::ACCOUNT_ID:role/s3-reader" # optional: Defaults to "sts.amazonaws.com" if not set eks.amazonaws.com/audience: "sts.amazonaws.com" # optional: When set to "true", adds AWS_STS_REGIONAL_ENDPOINTS env var # to containers eks.amazonaws.com/sts-regional-endpoints: "true" # optional: Defaults to 86400 for expirationSeconds if not set # Note: This value can be overwritten if specified in the pod # annotation as shown in the next step. eks.amazonaws.com/token-expiration: "86400"
-
Finally, apply the
my-service-account.yaml
file to create your service account.kubectl apply -f my-service-account.yaml
-
You can validate IRSA by following IRSA setup and test . Ensure the awscli pod is deployed in the same namespace of ServiceAccount
pod-identity-webhook
.
Configure the trust relationship for the OIDC provider’s IAM Role
In order to grant certain service accounts access to the desired AWS resources, edit the trust relationship for the OIDC provider’s IAM Role (OIDC_IAM_ROLE
) created in the first section, and add in the desired service accounts.
-
Choose the role in the console to open it for editing.
-
Choose the Trust relationships tab, and then choose Edit trust relationship.
-
Find the line that looks similar to the following:
"$ISSUER_HOSTPATH:aud": "sts.amazonaws.com"
-
Add another condition after that line which looks like the following line. Replace
KUBERNETES_SERVICE_ACCOUNT_NAMESPACE
andKUBERNETES_SERVICE_ACCOUNT_NAME
with the name of your Kubernetes service account and the Kubernetes namespace that the account exists in."$ISSUER_HOSTPATH:sub": "system:serviceaccount:KUBERNETES_SERVICE_ACCOUNT_NAMESPACE:KUBERNETES_SERVICE_ACCOUNT_NAME"
The allow list example below applies
my-serviceaccount
service account to thedefault
namespace and all service accounts to theobservability
namespace for theus-west-2
region. Remember to replaceAccount_ID
andS3_BUCKET
with the required values.{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::$Account_ID:oidc-provider/s3.us-west-2.amazonaws.com/$S3_BUCKET" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringLike": { "s3.us-west-2.amazonaws.com/$S3_BUCKET:aud": "sts.amazonaws.com", "s3.us-west-2.amazonaws.com/$S3_BUCKET:sub": [ "system:serviceaccount:default:my-serviceaccount", "system:serviceaccount:amazon-cloudwatch:*" ] } } } ] }
-
Refer this doc for different ways of configuring one or multiple service accounts through the condition operators in the trust relationship.
-
Choose Update Trust Policy to finish.
6 - IAM Authentication
AWS IAM Authenticator support (optional)
Provider support details
vSphere | Bare Metal | Nutanix | CloudStack | Snow | |
---|---|---|---|---|---|
Supported? | ✓ | ✓ | ✓ | ✓ | ✓ |
EKS Anywhere can create clusters that support AWS IAM Authenticator-based api server authentication. In order to add IAM Authenticator support, you need to configure your cluster by updating the configuration file before creating the cluster. This is a generic template with detailed descriptions below for reference:
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
name: my-cluster-name
spec:
...
# IAM Authenticator support
identityProviderRefs:
- kind: AWSIamConfig
name: aws-iam-auth-config
---
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: AWSIamConfig
metadata:
name: aws-iam-auth-config
spec:
awsRegion: ""
backendMode:
- ""
mapRoles:
- roleARN: arn:aws:iam::XXXXXXXXXXXX:role/myRole
username: myKubernetesUsername
groups:
- ""
mapUsers:
- userARN: arn:aws:iam::XXXXXXXXXXXX:user/myUser
username: myKubernetesUsername
groups:
- ""
partition: ""
identityProviderRefs (Under Cluster)
List of identity providers you want configured for the Cluster.
This would include a reference to the AWSIamConfig
object with the configuration below.
awsRegion (required)
- Description: awsRegion can be any region in the aws partition that the IAM roles exist in.
- Type: string
backendMode (required)
- Description: backendMode configures the IAM authenticator server’s backend mode (i.e. where to source mappings from). We support EKSConfigMap and CRD modes supported by AWS IAM Authenticator, for more details refer to backendMode
- Type: string
mapRoles, mapUsers (recommended for EKSConfigMap
backend)
-
Description: When using
EKSConfigMap
backendMode
, we recommend providing eithermapRoles
ormapUsers
to set the IAM role mappings at the time of creation. This input is added to an EKS style ConfigMap. For more details refer to EKS IAM -
Type: list object
roleARN, userARN (required)
- Description: IAM ARN to authenticate to the cluster.
roleARN
specifies an IAM role anduserARN
specifies an IAM user. - Type: string
username (required)
- Description: The Kubernetes username the IAM ARN is mapped to in the cluster. The ARN gets mapped to the Kubernetes cluster permissions associated with the username.
- Type: string
groups
- Description: List of kubernetes user groups that the mapped IAM ARN is given permissions to.
- Type: list string
- Description: IAM ARN to authenticate to the cluster.
partition
- Description: This field is used to set the aws partition that the IAM roles are present in. Default value is
aws
. - Type: string
7 - OIDC
OIDC support (optional)
EKS Anywhere can create clusters that support api server OIDC authentication.
Provider support details
vSphere | Bare Metal | Nutanix | CloudStack | Snow | |
---|---|---|---|---|---|
Supported? | ✓ | ✓ | ✓ | ✓ | ✓ |
In order to add OIDC support, you need to configure your cluster by updating the configuration file to include the details below. The OIDC configuration can be added at cluster creation time, or introduced via a cluster upgrade in VMware and CloudStack.
This is a generic template with detailed descriptions below for reference:
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
name: my-cluster-name
spec:
...
# OIDC support
identityProviderRefs:
- kind: OIDCConfig
name: my-cluster-name
---
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: OIDCConfig
metadata:
name: my-cluster-name
spec:
clientId: ""
groupsClaim: ""
groupsPrefix: ""
issuerUrl: "https://x"
requiredClaims:
- claim: ""
value: ""
usernameClaim: ""
usernamePrefix: ""
identityProviderRefs (Under Cluster)
List of identity providers you want configured for the Cluster.
This would include a reference to the OIDCConfig
object with the configuration below.
clientId (required)
- Description: ClientId defines the client ID for the OpenID Connect client
- Type: string
groupsClaim (optional)
- Description: GroupsClaim defines the name of a custom OpenID Connect claim for specifying user groups
- Type: string
groupsPrefix (optional)
- Description: GroupsPrefix defines a string to be prefixed to all groups to prevent conflicts with other authentication strategies
- Type: string
issuerUrl (required)
- Description: IssuerUrl defines the URL of the OpenID issuer, only HTTPS scheme will be accepted
- Type: string
requiredClaims (optional)
List of RequiredClaim objects listed below. Only one is supported at this time.
requiredClaims[0] (optional)
- Description: RequiredClaim defines a key=value pair that describes a required claim in the ID Token
- claim
- type: string
- value
- type: string
- claim
- Type: object
usernameClaim (optional)
- Description: UsernameClaim defines the OpenID claim to use as the user name. Note that claims other than the default (‘sub’) is not guaranteed to be unique and immutable
- Type: string
usernamePrefix (optional)
- Description: UsernamePrefix defines a string to be prefixed to all usernames. If not provided, username claims other than ‘email’ are prefixed by the issuer URL to avoid clashes. To skip any prefixing, provide the value ‘-’.
- Type: string
8 - Proxy
Proxy support (optional)
Provider support details
vSphere | Bare Metal | Nutanix | CloudStack | Snow | |
---|---|---|---|---|---|
Supported? | ✓ | ✓ | ✓ | ✓ | ✓ |
You can configure EKS Anywhere to use a proxy to connect to the Internet. This is the generic template with proxy configuration for your reference:
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
name: my-cluster-name
spec:
...
proxyConfiguration:
httpProxy: http-proxy-ip:port
httpsProxy: https-proxy-ip:port
noProxy:
- list of no proxy endpoints
Configuring Docker daemon
EKS Anywhere will proxy for you given the above configuration file. However, to successfully use EKS Anywhere you will also need to ensure your Docker daemon is configured to use the proxy.
This generally means updating your daemon to launch with the HTTPS_PROXY, HTTP_PROXY, and NO_PROXY environment variables.
For an example of how to do this with systemd, please see Docker’s documentation here .
Configuring EKS Anywhere proxy without config file
For commands using a cluster config file, EKS Anywhere will derive its proxy config from the cluster configuration file.
However, for commands that do not utilize a cluster config file, you can set the following environment variables:
export HTTPS_PROXY=https-proxy-ip:port
export HTTP_PROXY=http-proxy-ip:port
export NO_PROXY=no-proxy-domain.com,another-domain.com,localhost
Proxy Configuration Spec Details
proxyConfiguration (required)
- Description: top level key; required to use proxy.
- Type: object
httpProxy (required)
- Description: HTTP proxy to use to connect to the internet; must be in the format IP:port
- Type: string
- Example:
httpProxy: 192.168.0.1:3218
httpsProxy (required)
- Description: HTTPS proxy to use to connect to the internet; must be in the format IP:port
- Type: string
- Example:
httpsProxy: 192.168.0.1:3218
noProxy (optional)
- Description: list of endpoints that should not be routed through the proxy; can be an IP, CIDR block, or a domain name
- Type: list of strings
- Example
noProxy:
- localhost
- 192.168.0.1
- 192.168.0.0/16
- .example.com
Note
- For Bottlerocket OS, it is required to add the local subnet CIDR range in the
noProxy
list. - For Bare Metal provider, it is required to host hook images locally which should be accessible by admin machines as well as all the nodes without using proxy configuration. Please refer to the documentation for getting hook images here
.
9 - MachineHealthCheck
MachineHealthCheck Support
Provider support details
vSphere | Bare Metal | Nutanix | CloudStack | Snow | |
---|---|---|---|---|---|
Supported? | ✓ | ✓ | ✓ | ✓ | ✓ |
You can configure EKS Anywhere to specify timeouts and maxUnhealthy
values for machine health checks.
A MachineHealthCheck (MHC) is a resource in Cluster API which allows users to define conditions under which Machines within a Cluster should be considered unhealthy. A MachineHealthCheck is defined on a management cluster and scoped to a particular workload cluster.
Note: Even though the MachineHealthCheck configuration in the EKS-A spec is optional, MachineHealthChecks are still installed for all clusters using the default values mentioned below.
EKS Anywhere allows users to have granular control over MachineHealthChecks in their cluster configuration, with default values (derived from Cluster API) being applied if the MHC is not configured in the spec. The top-level machineHealthCheck
field governs the global MachineHealthCheck settings for all Machines (control-plane and worker). These global settings can be overridden through the nested machineHealthCheck
field in the control plane configuration and each worker node configuration. If the nested MHC fields are not configured, then the top-level settings are applied to the respective Machines.
The following cluster spec shows an example of how to configure health check timeouts and maxUnhealthy
:
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
name: my-cluster-name
spec:
...
machineHealthCheck: # Top-level MachineHealthCheck configuration
maxUnhealthy: "60%"
nodeStartupTimeout: "10m0s"
unhealthyMachineTimeout: "5m0s"
...
controlPlaneConfiguration: # MachineHealthCheck configuration for Control plane
machineHealthCheck:
maxUnhealthy: 100%
nodeStartupTimeout: "15m0s"
unhealthyMachineTimeout: 10m
...
workerNodeGroupConfigurations:
- count: 1
name: md-0
machineHealthCheck: # MachineHealthCheck configuration for Worker Node Group 0
maxUnhealthy: 100%
nodeStartupTimeout: "10m0s"
unhealthyMachineTimeout: 20m
- count: 1
name: md-1
machineHealthCheck: # MachineHealthCheck configuration for Worker Node Group 1
maxUnhealthy: 100%
nodeStartupTimeout: "10m0s"
unhealthyMachineTimeout: 20m
...
MachineHealthCheck Spec Details
machineHealthCheck (optional)
- Description: top-level key; required to configure global MachineHealthCheck timeouts and
maxUnhealthy
. - Type: object
machineHealthCheck.maxUnhealthy (optional)
- Description: determines the maximum permissible number or percentage of unhealthy Machines in a cluster before further remediation is prevented. This ensures that MachineHealthChecks only remediate Machines when the cluster is healthy.
- Default:
100%
for control plane machines,40%
for worker nodes (Cluster API defaults). - Type: integer (count) or string (percentage)
machineHealthCheck.nodeStartupTimeout (optional)
- Description: determines how long a MachineHealthCheck should wait for a Node to join the cluster, before considering a Machine unhealthy.
- Default:
20m0s
for Tinkerbell provider,10m0s
for all other providers. - Minimum Value (If configured):
30s
- Type: string
machineHealthCheck.unhealthyMachineTimeout (optional)
- Description: determines how long the unhealthy Node conditions (e.g.,
Ready=False
,Ready=Unknown
) should be matched for, before considering a Machine unhealthy. - Default:
5m0s
- Type: string
controlPlaneConfiguration.machineHealthCheck (optional)
- Description: Control plane level configuration for MachineHealthCheck timeouts and
maxUnhealthy
values. - Type: object
controlPlaneConfiguration.machineHealthCheck.maxUnhealthy (optional)
- Description: determines the maximum permissible number or percentage of unhealthy control plane Machines in a cluster before further remediation is prevented. This ensures that MachineHealthChecks only remediate Machines when the cluster is healthy.
- Default: Top-level MHC
maxUnhealthy
if set or100%
otherwise. - Type: integer (count) or string (percentage)
controlPlaneConfiguration.machineHealthCheck.nodeStartupTimeout (optional)
- Description: determines how long a MachineHealthCheck should wait for a control plane Node to join the cluster, before considering the Machine unhealthy.
- Default: Top-level MHC
nodeStartupTimeout
if set or20m0s
for Tinkerbell provider,10m0s
for all other providers otherwise. - Minimum Value (if configured):
30s
- Type: string
controlPlaneConfiguration.machineHealthCheck.unhealthyMachineTimeout (optional)
- Description: determines how long the unhealthy conditions (e.g.,
Ready=False
,Ready=Unknown
) should be matched for a control plane Node, before considering the Machine unhealthy. - Default: Top-level MHC
nodeStartupTimeout
if set or5m0s
otherwise. - Type: string
workerNodeGroupConfigurations.machineHealthCheck (optional)
- Description: Worker node level configuration for MachineHealthCheck timeouts and
maxUnhealthy
values. - Type: object
workerNodeGroupConfigurations.machineHealthCheck.maxUnhealthy (optional)
- Description: determines the maximum permissible number or percentage of unhealthy worker Machines in a cluster before further remediation is prevented. This ensures that MachineHealthChecks only remediate Machines when the cluster is healthy.
- Default: Top-level MHC
maxUnhealthy
if set or40%
otherwise. - Type: integer (count) or string (percentage)
workerNodeGroupConfigurations.machineHealthCheck.nodeStartupTimeout (optional)
- Description: determines how long a MachineHealthCheck should wait for a worker Node to join the cluster, before considering the Machine unhealthy.
- Default: Top-level MHC
nodeStartupTimeout
if set or20m0s
for Tinkerbell provider,10m0s
for all other providers otherwise. - Minimum Value (if configured):
30s
- Type: string
workerNodeGroupConfigurations.machineHealthCheck.unhealthyMachineTimeout (optional)
- Description: determines how long the unhealthy conditions (e.g.,
Ready=False
,Ready=Unknown
) should be matched for a worker Node, before considering the Machine unhealthy. - Default: Top-level MHC
nodeStartupTimeout
if set or5m0s
otherwise. - Type: string
10 - Registry Mirror
Registry Mirror Support (optional)
Provider support details
vSphere | Bare Metal | Nutanix | CloudStack | Snow | |
---|---|---|---|---|---|
Supported? | ✓ | ✓ | ✓ | ✓ | ✓ |
You can configure EKS Anywhere to use a local registry mirror for its dependencies. When a registry mirror is configured in the EKS Anywhere cluster specification, EKS Anywhere will use it instead of defaulting to Amazon ECR for its dependencies. For details on how to configure your local registry mirror for EKS Anywhere, see the Configure local registry mirror section.
See the airgapped documentation page for instructions on downloading and importing EKS Anywhere dependencies to a local registry mirror.
Registry Mirror Cluster Spec
The following cluster spec shows an example of how to configure registry mirror:
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
name: my-cluster-name
spec:
...
registryMirrorConfiguration:
endpoint: <private registry IP or hostname>
port: <private registry port>
ociNamespaces:
- registry: <upstream registry IP or hostname>
namespace: <namespace in private registry>
...
caCertContent: |
-----BEGIN CERTIFICATE-----
MIIF1DCCA...
...
es6RXmsCj...
-----END CERTIFICATE-----
Registry Mirror Cluster Spec Details
registryMirrorConfiguration (optional)
- Description: top level key; required to use a private registry.
- Type: object
endpoint (required)
- Description: IP address or hostname of the private registry for pulling images
- Type: string
- Example:
endpoint: 192.168.0.1
port (optional)
- Description: port for the private registry. This is an optional field. If a port
is not specified, the default HTTPS port
443
is used - Type: string
- Example:
port: 443
ociNamespaces (optional)
-
Description: when you need to mirror multiple registries, you can map each upstream registry to the “namespace” of its mirror. The namespace is appended with the endpoint,
<endpoint>/<namespace>
to setup the mirror for the registry specified. Note while usingociNamespaces
, you need to specify all the registries that need to be mirrored. This includes an entry for thepublic.ecr.aws
registry to pull EKS Anywhere images from. -
Type: array
-
Example:
ociNamespaces: - registry: "public.ecr.aws" namespace: "" - registry: "783794618700.dkr.ecr.us-west-2.amazonaws.com" namespace: "curated-packages"
Warning
Currently onlypublic.ecr.aws
registry is supported for mirroring with Bottlerocket OS.
caCertContent (optional)
-
Description: certificate Authority (CA) Certificate for the private registry . When using self-signed certificates it is necessary to pass this parameter in the cluster spec. This must be the individual public CA cert used to sign the registry certificate. This will be added to the cluster nodes so that they are able to pull images from the private registry.
It is also possible to configure CACertContent by exporting an environment variable:
export EKSA_REGISTRY_MIRROR_CA="/path/to/certificate-file"
-
Type: string
-
Example:
CACertContent: | -----BEGIN CERTIFICATE----- MIIF1DCCA... ... es6RXmsCj... -----END CERTIFICATE-----
authenticate (optional)
- Description: optional field to authenticate with a private registry. When using private registries that
require authentication, it is necessary to set this parameter to
true
in the cluster spec. - Type: boolean
When this value is set to true, the following environment variables need to be set:
export REGISTRY_USERNAME=<username>
export REGISTRY_PASSWORD=<password>
insecureSkipVerify (optional)
- Description: optional field to skip the registry certificate verification. Only use this solution for isolated testing or in a tightly controlled, air-gapped environment. Currently only supported for Ubuntu and RHEL OS.
- Type: boolean
Configure local registry mirror
Project configuration
The following projects must be created in your registry before importing the EKS Anywhere images:
bottlerocket
eks-anywhere
eks-distro
isovalent
cilium-chart
For example, if a registry is available at private-registry.local
, then the following projects must be created.
https://private-registry.local/bottlerocket
https://private-registry.local/eks-anywhere
https://private-registry.local/eks-distro
https://private-registry.local/isovalent
https://private-registry.local/cilium-chart
Admin machine configuration
You must configure the Admin machine with the information it needs to communicate with your registry.
Add the registry’s CA certificate to the list of CA certificates on the Admin machine if your registry uses self-signed certificates.
- For Linux
, you can place your certificate here:
/etc/docker/certs.d/<private-registry-endpoint>/ca.crt
- For Mac , you can follow this guide to add the certificate to your keychain: https://docs.docker.com/desktop/mac/#add-tls-certificates
If your registry uses authentication, the following environment variables must be set on the Admin machine before running the eksctl anywhere import images
command.
export REGISTRY_USERNAME=<username>
export REGISTRY_PASSWORD=<password>
11 - Autoscaling configuration
EKS Anywhere supports autoscaling worker node groups using the Kubernetes Cluster Autoscaler . The Kubernetes Cluster Autoscaler Curated Package is an image and helm chart installed via the Curated Packages Controller
The helm chart utilizes the Cluster Autoscaler clusterapi
mode
to scale resources.
Configure an EKS Anywhere worker node group to be picked up by a Cluster Autoscaler deployment by adding autoscalingConfiguration
block to the workerNodeGroupConfiguration
.
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
name: my-cluster-name
spec:
workerNodeGroupConfigurations:
- name: md-0
autoscalingConfiguration:
minCount: 1
maxCount: 5
machineGroupRef:
kind: VSphereMachineConfig
name: worker-machine-a
- name: md-1
autoscalingConfiguration:
minCount: 1
maxCount: 3
machineGroupRef:
kind: VSphereMachineConfig
name: worker-machine-b
Note that if count
is specified for the worker node group, it’s value will be ignored during cluster creation as well as cluster upgrade. If only one of minCount
or maxCount
is specified, then the other will have a default value of 0
and count
will have a default value of minCount
.
EKS Anywhere automatically applies the following annotations to your MachineDeployment
objects for worker node groups with autoscaling enabled. The Cluster Autoscaler component uses these annotations to identify which node groups to autoscale. If a node group is not autoscaling as expected, check for these annotations on the MachineDeployment
to troubleshoot.
cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: <minCount>
cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: <maxCount>
12 - Skipping validations configuration
EKS Anywhere runs a set of validations while performing cluster operations. Some of these validations can be chosen to be skipped.
One such validation EKS Anywhere runs is a check for whether cluster’s control plane ip is in use or not.
- If a cluster is being created using the EKS Anywhere cli, this validation can be skipped by using the
--skip-ip-check
flag or by setting the below annotation on theCluster
object. - If a workload cluster is being created using tools like
kubectl
orGitOps
, the validation can only be skipped by adding the below annotation.
Configure an EKS Anywhere cluster to skip the validation for checking the uniqueness of the control plane IP by using the anywhere.eks.amazonaws.com/skip-ip-check
annotation and setting the value to true
like shown below.
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
annotations:
anywhere.eks.amazonaws.com/skip-ip-check: "true"
name: my-cluster-name
spec:
.
.
.
Note that this annotation is also automatically set if you use the --skip-ip-check
flag while running the EKS Anywhere create cluster command.
13 - GitOps
GitOps Support (Optional)
Provider support details
vSphere | Bare Metal | Nutanix | CloudStack | Snow | |
---|---|---|---|---|---|
Supported? | ✓ | ✓ | ✓ | ✓ | ✓ |
EKS Anywhere can create clusters that supports GitOps configuration management with Flux.
In order to add GitOps support, you need to configure your cluster by specifying the configuration file with gitOpsRef
field when creating or upgrading the cluster.
We currently support two types of configurations: FluxConfig
and GitOpsConfig
.
Flux Configuration
The flux configuration spec has three optional fields, regardless of the chosen git provider.
Flux Configuration Spec Details
systemNamespace (optional)
- Description: Namespace in which to install the gitops components in your cluster. Defaults to
flux-system
- Type: string
clusterConfigPath (optional)
- Description: The path relative to the root of the git repository where EKS Anywhere will store the cluster configuration files. Defaults to the cluster name
- Type: string
branch (optional)
- Description: The branch to use when committing the configuration. Defaults to
main
- Type: string
EKS Anywhere currently supports two git providers for FluxConfig: Github and Git.
Github provider
Please note that for the Flux config to work successfully with the Github provider, the environment variable EKSA_GITHUB_TOKEN
needs to be set with a valid GitHub PAT
.
This is a generic template with detailed descriptions below for reference:
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
name: my-cluster-name
namespace: default
spec:
...
#GitOps Support
gitOpsRef:
name: my-github-flux-provider
kind: FluxConfig
---
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: FluxConfig
metadata:
name: my-github-flux-provider
namespace: default
spec:
systemNamespace: "my-alternative-flux-system-namespace"
clusterConfigPath: "path-to-my-clusters-config"
branch: "main"
github:
personal: true
repository: myClusterGitopsRepo
owner: myGithubUsername
---
github Configuration Spec Details
repository (required)
- Description: The name of the repository where EKS Anywhere will store your cluster configuration, and sync it to the cluster. If the repository exists, we will clone it from the git provider; if it does not exist, we will create it for you.
- Type: string
owner (required)
- Description: The owner of the Github repository; either a Github username or Github organization name. The Personal Access Token used must belong to the owner if this is a personal repository, or have permissions over the organization if this is not a personal repository.
- Type: string
personal (optional)
- Description: Is the repository a personal or organization repository?
If personal, this value is
true
; otherwise,false
. If using an organizational repository (e.g.personal
isfalse
) theowner
field will be used as theorganization
when authenticating to github.com - Default: true
- Type: boolean
Git provider
Before you create a cluster using the Git provider, you will need to set and export the EKSA_GIT_KNOWN_HOSTS
and EKSA_GIT_PRIVATE_KEY
environment variables.
EKSA_GIT_KNOWN_HOSTS
EKS Anywhere uses the provided known hosts file to verify the identity of the git provider when connecting to it with SSH.
The EKSA_GIT_KNOWN_HOSTS
environment variable should be a path to a known hosts file containing entries for the git server to which you’ll be connecting.
For example, if you wanted to provide a known hosts file which allows you to connect to and verify the identity of github.com using a private key based on the key algorithm ecdsa, you can use the OpenSSH utility ssh-keyscan
to obtain the known host entry used by github.com for the ecdsa
key type.
EKS Anywhere supports ecdsa
, rsa
, and ed25519
key types, which can be specified via the sshKeyAlgorithm
field of the git provider config.
ssh-keyscan -t ecdsa github.com >> my_eksa_known_hosts
This will produce a file which contains known-hosts entries for the ecdsa
key type supported by github.com, mapping the host to the key-type and public key.
github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=
EKS Anywhere will use the content of the file at the path EKSA_GIT_KNOWN_HOSTS
to verify the identity of the remote git server, and the provided known hosts file must contain an entry for the remote host and key type.
EKSA_GIT_PRIVATE_KEY
The EKSA_GIT_PRIVATE_KEY
environment variable should be a path to the private key file associated with a valid SSH public key registered with your Git provider.
This key must have permission to both read from and write to your repository.
The key can use the key algorithms rsa
, ecdsa
, and ed25519
.
This key file must have restricted file permissions, allowing only the owner to read and write, such as octal permissions 600
.
If your private key file is passphrase protected, you must also set EKSA_GIT_SSH_KEY_PASSPHRASE
with that value.
This is a generic template with detailed descriptions below for reference:
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
name: my-cluster-name
namespace: default
spec:
...
#GitOps Support
gitOpsRef:
name: my-git-flux-provider
kind: FluxConfig
---
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: FluxConfig
metadata:
name: my-git-flux-provider
namespace: default
spec:
systemNamespace: "my-alternative-flux-system-namespace"
clusterConfigPath: "path-to-my-clusters-config"
branch: "main"
git:
repositoryUrl: ssh://git@github.com/myAccount/myClusterGitopsRepo.git
sshKeyAlgorithm: ecdsa
---
git Configuration Spec Details
repositoryUrl (required)
- Description: The URL of an existing repository where EKS Anywhere will store your cluster configuration and sync it to the cluster. For private repositories, the SSH URL will be of the format
ssh://git@provider.com/$REPO_OWNER/$REPO_NAME.git
- Type: string
- Value: A common
repositoryUrl
value can be of the formatssh://git@provider.com/$REPO_OWNER/$REPO_NAME.git
. This may differ from the default SSH URL given by your provider. Consider these differences between github and CodeCommit URLs:- The github.com user interface provides an SSH URL containing a
:
before the repository owner, rather than a/
. Make sure to replace this:
with a/
, if present. - The CodeCommit SSH URL must include SSH-KEY-ID in format
ssh://<SSH-Key-ID>@git-codecommit.<region>.amazonaws.com/v1/repos/<repository>
.
- The github.com user interface provides an SSH URL containing a
sshKeyAlgorithm (optional)
- Description: The SSH key algorithm of the private key specified via
EKSA_PRIVATE_KEY_FILE
. Defaults toecdsa
- Type: string
Supported SSH key algorithm types are ecdsa
, rsa
, and ed25519
.
Be sure that this SSH key algorithm matches the private key file provided by EKSA_GIT_PRIVATE_KEY_FILE
and that the known hosts entry for the key type is present in EKSA_GIT_KNOWN_HOSTS
.
GitOps Configuration
Warning
GitOps Config will be deprecated in v0.11.0 in lieu of using the Flux Config described above.Please note that for the GitOps config to work successfully the environment variable EKSA_GITHUB_TOKEN
needs to be set with a valid GitHub PAT
. This is a generic template with detailed descriptions below for reference:
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
name: my-cluster-name
namespace: default
spec:
...
#GitOps Support
gitOpsRef:
name: my-gitops
kind: GitOpsConfig
---
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: GitOpsConfig
metadata:
name: my-gitops
namespace: default
spec:
flux:
github:
personal: true
repository: myClusterGitopsRepo
owner: myGithubUsername
fluxSystemNamespace: ""
clusterConfigPath: ""
GitOps Configuration Spec Details
flux (required)
- Description: our supported gitops provider is
flux
. This is the only supported value. - Type: object
Flux Configuration Spec Details
github (required)
- Description:
github
is the only currently supported git provider. This defines your github configuration to be used by EKS Anywhere and flux. - Type: object
github Configuration Spec Details
repository (required)
- Description: The name of the repository where EKS Anywhere will store your cluster configuration, and sync it to the cluster. If the repository exists, we will clone it from the git provider; if it does not exist, we will create it for you.
- Type: string
owner (required)
- Description: The owner of the Github repository; either a Github username or Github organization name.
The Personal Access Token used must belong to the
owner
if this is apersonal
repository, or have permissions over the organization if this is not apersonal
repository. - Type: string
personal (optional)
- Description: Is the repository a personal or organization repository?
If personal, this value is
true
; otherwise,false
. If using an organizational repository (e.g.personal
isfalse
) theowner
field will be used as theorganization
when authenticating to github.com - Default:
true
- Type: boolean
clusterConfigPath (optional)
- Description: The path relative to the root of the git repository where EKS Anywhere will store the cluster configuration files.
- Default:
clusters/$MANAGEMENT_CLUSTER_NAME
- Type: string
fluxSystemNamespace (optional)
- Description: Namespace in which to install the gitops components in your cluster.
- Default:
flux-system
. - Type: string
branch (optional)
- Description: The branch to use when committing the configuration.
- Default:
main
- Type: string
14 - Package controller
Package Controller Configuration (optional)
You can specify EKS Anywhere package controller configurations. For more on the curated packages and the package controller, visit the package management documentation.
The following cluster spec shows an example of how to configure package controller:
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
name: my-cluster-name
spec:
...
packages:
disable: false
controller:
resources:
requests:
cpu: 100m
memory: 50Mi
limits:
cpu: 750m
memory: 450Mi
Package Controller Configuration Spec Details
packages (optional)
- Description: Top level key; required controller configuration.
- Type: object
packages.disable (optional)
- Description: Disable the package controller.
- Type: bool
- Example:
disable: true
packages.controller (optional)
- Description: Disable the package controller.
- Type: object
packages.controller.resources (optional)
- Description: Resources for the package controller.
- Type: object
packages.controller.resources.limits (optional)
- Description: Resource limits.
- Type: object
packages.controller.resources.limits.cpu (optional)
- Description: CPU limit.
- Type: string
packages.controller.resources.limits.memory (optional)
- Description: Memory limit.
- Type: string
packages.controller.resources.requests (optional)
- Description: Requested resources.
- Type: object
packages.controller.resources.requests.cpu (optional)
- Description: Requested cpu.
- Type: string
packages.controller.resources.limits.memory (optional)
- Description: Requested memory.
- Type: string
packages.cronjob (optional)
- Description: Disable the package controller.
- Type: object
packages.cronjob.disable (optional)
- Description: Disable the cron job.
- Type: bool
- Example:
disable: true
packages.cronjob.resources (optional)
- Description: Resources for the package controller.
- Type: object
packages.cronjob.resources.limits (optional)
- Description: Resource limits.
- Type: object
packages.cronjob.resources.limits.cpu (optional)
- Description: CPU limit.
- Type: string
packages.cronjob.resources.limits.memory (optional)
- Description: Memory limit.
- Type: string
packages.cronjob.resources.requests (optional)
- Description: Requested resources.
- Type: object
packages.cronjob.resources.requests.cpu (optional)
- Description: Requested cpu.
- Type: string
packages.cronjob.resources.limits.memory (optional)
- Description: Requested memory.
- Type: string