E2E test cases for opr-paas

Below you will find a list of features that we want to test using end-to-end testing. For each feature, the setup and the assessments are listed.

Paas

What we test: CRUD for Paas

Scenarios:

1. A Paas is created that already exists.
   
   Given that a specified Paas already exists,
   when someone configures a new Paas with the same name,
   then the Paas related namespace must not be created
   and the operator must return an error.

2. A minimal Paas is created.
   
   Given that the Paas does not exist,
   when someone configures the minimal Paas
   then the Paas namespace must be created
   and the status of the Paas contains no errors.

3. A Paas is renamed.
   
   Given that a specified Paas exists,
   when the Paas is renamed in the configuration,
   then the related Paas namespace must be renamed as well.

4. A Paas is deleted.
   
   Given that a specified Paas exists,
   when the Paas is deleted,
   then the namespace belonging to the Paas is also removed.

Post scenarios: reset environment to clean slate.

PaasNs

What we test: CRUD for PaasNs

Scenarios:

1. A PaasNs is created for a Paas that does not exist.
   
   Given that a specified Paas does not exist,
   when someone configures a PaasNs to be created under specified Paas,
   then the PaasNs related namespace must not be created
   and the status of said PaasNs must contain the correct error.

2. A minimal Paas that is referenced in the PaasNs is created.
   
   Given that the PaasNs does not exist,
   and that the minimal Paas does not exist,
   when someone configures the minimal Paas
   and someone configures the PaasNs referencing the minimal Paas
   then the PaasNS namespace must be created
   and the PaasNS namespace has the quota_label label with the value of ... FIXME
   and the status of the PaasNs contains no errors.

3. A PaasNs is renamed.
   
   Given that a specified PaasNs exists,
   when the PaasNs is renamed in the configuration,
   then the related PaasNs namespace must be renamed as well.

4. A PaasNs is deleted.
   
   Given that a specified PaasNs exists,
   when the PaasNs is deleted,
   then the namespace belonging to the PaasNs is also removed.

Post scenarios: reset environment to clean slate.

Namespaces

1. A minimal Paas configuration without namespaces results in one namespace.
   
   Given a minimal Paas configuration without namespaces,
   when someone adds said configuration to the system,
   then a single namespace should have been created,
   and this namespace should be named the same as the Paas,
   and no PaasNs's or there namespaces are linked to th Paas.

2. Adding two namespaces to the Paas's spec.
   
   Given a minimal Paas without any namespaces exists,
   when the Paas is updated by adding 2 namespaces to the spec,
   then one namespace with the same name as the Paas must exist,
   and two PaasNs's must exist in the namespace of the Paas,
   and these PaasNs's each have a namespace,
   and these PaasNs namespaces must be named according to their spec.namespaces entries, prefixed by the Paas namespace name

3. Removing the namespaces from a Paas.
   
   Given a minimal Paas with two namespaces exists,
   when the Paas configuration is updated to remove the namespaces,
   then the PaasNs's should have be removed from the Paas namespace,
   and the Paas namespace was removed.

Post scenarios: reset environment to clean slate.

ClusterResourceQuotas

What we test: CRQ CRUD

Note

The spec.quota does not fall under not cluster wide quotas, hence a separate set of test scenarios.

1. Ensure the correct CRQ is created for a Paas.
   
   Given a minimal Paas exists,
   when someone adds a quota to spec.quota for the Paas configuration,
   then a CRQ with the name of the Paas must be created,
   and clusterquotagroup= followed by the Paas name should have been applied as label selector on the CRQ,
   and the size of the created CRQ equals the size as specified in the spec.quota.

2. The spec.quota for a Paas is updated.
   
   Given a minimal Paas exists,
   and a valid CRQ exists for this Paas,
   when someone updates the spec.quota section for the specified Paas configuration,
   then the CRQ should be updated,
   and the size of the updated CRQ equals the size as specified in the spec.quota.

3. Removing the Paas should remove the associated CRQ.
   
   Given a minimal Paas and its associated CRQ exist,
   when the Paas is removed,
   then the associated CRW with the name of the Paas should have removed as well.

Post scenarios: reset environment to clean slate.

Cluster wide quotas

What we test: cluster wide quota CRUD

Scenarios:

TODO

Post scenarios: reset environment to clean slate.

Groups => Users

What we test: managing users and group memberships through Paas configuration.

Scenarios:

1. Creating a group with a single user without a specified role.
   
   Given a minimal Paas with a single namespace,
   and a group with a single user without a specified role,
   when that Paas is created,
   then a Group was created with the correct name,
   and the user is a member of said group,
   and the correct labels were placed on the group,
   and the Owner Reference for the Group points to the correct Paas,
   and there are were no changes to the Whitelist,
   and the rolebinding on the namespace points to the group to the default role,
   and the rolebinding was not applied to the default Paas namespace (name of the Paas)

2. Updating the Paas, adding a group with a role other than default.
   
   Given an existing, minimal Paas with a single namespace,
   when a group is added to said Paas,
   and said group has a specific role, other than default (see test_config),
   and a different user is a member of said group than in scenario 1,
   then a Group was created with the correct name,
   and the user is a member of said group,
   and the correct labels were placed on the group,
   and the Owner Reference for the Group points to the correct Paas,
   and there are were no changes to the Whitelist,
   and the rolebinding on the namespace points to the group to the default role,
   and the rolebinding was not applied to the default Paas namespace (name of the Paas)

3. Removing the Paas.
   
   Given an existing Paas with a single group,
   when said Paas is removed,
   

Note

There are known issues on groups, updating / removing does not go perfectly. Goal here is to test the baseline in the current situation.

Determine what that baseline currently is.

Post scenarios: reset environment to clean slate.

Groups => Query

What we test: managing users and group memberships through an LDAP query specified in the Paas configuration.

Scenarios:

1. Minimal Paas with one namespace and a Group with a Query but no Role.
   
   Given no existing Paas,
   when a minimal Paas is created with a single namespace,
   and a Group with a Query, but without a Role,
   then a Group with the correct name should have been created,
   and there should be no users in said Group,
   and the correct labels were added onto the Group,
   and the Owner Reference for the Group points to the correct Paas,
   and the query was added to the whitelist,
   and the rolebinding on the namespace points to the group, to the specified role,
   and the rolebinding was not applied to the default Paas namespace (name of the Paas).

2. Updating the Paas, adding a group with a role other than default.
   
   Given an existing, minimal Paas with a single namespace,
   when another query is added to said Paas (compared to step scenario 1),
   and said group has a specific role, other than default (see test_config),
   then a Group was created with the correct name,
   and there are no users in said group,
   and the correct labels were placed on the group, (no ldap things)
   and the Owner Reference for the Group points to the correct Paas,
   and there were changes made to the Whitelist,
   and the rolebinding on the namespace points to the group to the specified role,
   and the rolebinding was not applied to the default Paas namespace (name of the Paas)

3. Removing the Paas.
   
   Given an existing Paas with a single group,
   when said Paas is removed,
   

Note

There are known issues on groups, updating / removing does not go perfectly. Goal here is to test the baseline in the current situation.

Determine what that baseline currently is.

Post scenarios: reset environment to clean slate.

Note

TODO, there a known issue regarding groups. Good regression test to model the following steps: 1. Create Paas with query; 2. Update Paas from step 1, remove query and add users to the group; 3. Group is not removed from whitelist; 4. Sync fails because the required ldap.uid doesn't match the groupname. (We don't fully test sync.)

Secrets

The capabilities are also used to test clusterwide quotas

Capability ArgoCD

What we test: creating a Paas with the ArgoCD capability enabled.

Scenarios:

1. A minimal Paas with ArgoCD capability enabled.
   
   Given a minimal Paas and ArgoCD capability configuration,
   when the minimal Paas is created with the ArgoCD capability enabled,
   then the list entry in the applicationset should have been created,
   and a namespace with the name paasname-argocd should have been created,
   and an ArgoCD Application should have been created in namespaces,
   and a quota with the name paasname-argocd should have been created,
   and the ArgoCD Application and quota conform to the points below.

ArgoCD Application points: 1. Assess gitUrl, path etc. exist in spec; 2. Assess RBAC .. determine how; 3. Assess Secrets exist in namespace and in argo...? 4. Assess Exclude appset is included in spec as ignoreDiff;

Quota points:
  1. Assess a quota with the name `paasnaam-argocd` was created;
  2. Assess that the `quota_label` label was used as selector on the quota;
  3. Assess that the quota selector was set in such a manner so that only the `paasnaam-sso` namespace is selected;
  4. Assess that the size of the quota equals the size of the default quota specified in the paas_config;

!!! Note TODO Assess default_permissions; 1. Rolebindings to service account etc. (TODO: can these RBs be created without the existence of a ServiceAccount?)

Post scenarios: reset environment to clean slate.

Capability Tekton

Check Quota

Capability Grafana

Check Quota

Capability SSO

What we test: creating a Paas with the SSO capability enabled.

Scenarios:

1. A minimal Paas with SSO capability enabled and no capability quota.
   
   Given a minimal Paas and SSO capability configuration,
   when the minimal Paas is created with the SSO capability enabled,
   then the list entry in the applicationset should have been created,
   and a namespace with the name paasname-sso should have been created,
   and a quota with the name paasname-sso should have been created,
   and the quota conforms to the points below.

Quota points: 1. Assess that the quota_label label was used as selector on the quota; 2. Assess that the quota selector was set in such a manner so that only the paasnaam-sso namespace is selected; 3. Assess that the size of the quota equals the size of the default quota specified in the paas_config;

1. The Paas from scenario 1 is removed.
   
   Given a the Paas remaining from scenario 1 above,
   when said Paas is deleted,
   then the associated Quota should have been removed,
   then the associated Namespace should have been removed,
   then the associated list entry in the ApplicationSet should have been removed.

Post scenarios: reset environment to clean slate.