E2E test cases for opr-paas

Below you will find a list of features that we want to test using end-to-end testing. For each feature, the setup and the assessments are listed.

Paas

What we test: CRUD for Paas

Scenarios:

1. A Paas is created that already exists.
   
   Given that a specified Paas already exists,
   when someone configures a new Paas with the same name,
   then the Paas related namespace must not be created
   and the operator must return an error.

2. A minimal Paas is created.
   
   Given that the Paas does not exist,
   when someone configures the minimal Paas
   then the Paas namespace must be created
   and the status of the Paas contains no errors.

3. A Paas is renamed.
   
   Given that a specified Paas exists,
   when the Paas is renamed in the configuration,
   then the related Paas namespace must be renamed as well.

4. A Paas is deleted.
   
   Given that a specified Paas exists,
   when the Paas is deleted,
   then the namespace belonging to the Paas is also removed.

Post scenarios: reset environment to clean slate.

PaasNs

What we test: CRUD for PaasNs

Scenarios:

1. A PaasNs is created for a Paas that does not exist.
   
   Given that a specified Paas does not exist,
   when someone configures a PaasNs to be created under specified Paas,
   then the PaasNs related namespace must not be created
   and the status of said PaasNs must contain the correct error.

2. A minimal Paas that is referenced in the PaasNs is created.
   
   Given that the PaasNs does not exist,
   and that the minimal Paas does not exist,
   when someone configures the minimal Paas
   and someone configures the PaasNs referencing the minimal Paas
   then the PaasNS namespace must be created
   and the PaasNS namespace has the quota_label label with the value of ... FIXME
   and the status of the PaasNs contains no errors.

3. A PaasNs is renamed.
   
   Given that a specified PaasNs exists,
   when the PaasNs is renamed in the configuration,
   then the related PaasNs namespace must be renamed as well.

4. A PaasNs is deleted.
   
   Given that a specified PaasNs exists,
   when the PaasNs is deleted,
   then the namespace belonging to the PaasNs is also removed.

Post scenarios: reset environment to clean slate.

Namespaces

1. A minimal Paas configuration without namespaces results in one namespace.
   
   Given a minimal Paas configuration without namespaces,
   when someone adds said configuration to the system,
   then a single namespace should have been created,
   and this namespace should be named the same as the Paas,
   and no PaasNs's or there namespaces are linked to th Paas.

2. Adding two namespaces to the Paas's spec.
   
   Given a minimal Paas without any namespaces exists,
   when the Paas is updated by adding 2 namespaces to the spec,
   then one namespace with the same name as the Paas must exist,
   and two PaasNs's must exist in the namespace of the Paas,
   and these PaasNs's each have a namespace,
   and these PaasNs namespaces must be named according to their spec.namespaces entries, prefixed by the Paas namespace name

3. Removing the namespaces from a Paas.
   
   Given a minimal Paas with two namespaces exists,
   when the Paas configuration is updated to remove the namespaces,
   then the PaasNs's should have be removed from the Paas namespace,
   and the Paas namespace was removed.

Post scenarios: reset environment to clean slate.

ClusterResourceQuotas

What we test: CRQ CRUD

Note

The spec.quota does not fall under not cluster wide quotas, hence a separate set of test scenarios.

1. Ensure the correct CRQ is created for a Paas.
   
   Given a minimal Paas exists,
   when someone adds a quota to spec.quota for the Paas configuration,
   then a CRQ with the name of the Paas must be created,
   and clusterquotagroup= followed by the Paas name should have been applied as label selector on the CRQ,
   and the size of the created CRQ equals the size as specified in the spec.quota.

2. The spec.quota for a Paas is updated.
   
   Given a minimal Paas exists,
   and a valid CRQ exists for this Paas,
   when someone updates the spec.quota section for the specified Paas configuration,
   then the CRQ should be updated,
   and the size of the updated CRQ equals the size as specified in the spec.quota.

3. Removing the Paas should remove the associated CRQ.
   
   Given a minimal Paas and its associated CRQ exist,
   when the Paas is removed,
   then the associated CRW with the name of the Paas should have removed as well.

Post scenarios: reset environment to clean slate.

Cluster wide quotas

What we test: cluster wide quota CRUD

Scenarios:

TODO

Post scenarios: reset environment to clean slate.

Groups => Users

What we test: managing users and group memberships through Paas configuration.

Scenarios:

1. Creating a group with a single user without a specified role.
   
   Given a minimal Paas with a single namespace,
   and a group with a single user without a specified role,
   when that Paas is created,
   then a Group was created with the correct name,
   and the user is a member of said group,
   and the correct labels were placed on the group,
   and the Owner Reference for the Group points to the correct Paas,
   and the rolebinding on the namespace points to the group to the default role,
   

2. Updating the Paas, adding a group with a role other than default.
   
   Given an existing, minimal Paas with a single namespace,
   when a group is added to said Paas,
   and said group has a specific role, other than default (see test_config),
   and a different user is a member of said group than in scenario 1,
   then a Group was created with the correct name,
   and the user is a member of said group,
   and the correct labels were placed on the group,
   and the Owner Reference for the Group points to the correct Paas,
   and the rolebinding on the namespace points to the group to the default role,
   

3. Removing the Paas.
   
   Given an existing Paas with a single group,
   when said Paas is removed,
   

Post scenarios: reset environment to clean slate.

Groups => Query

What we test: managing users and group memberships through an LDAP query specified in the Paas configuration.

Scenarios:

1. Minimal Paas with one namespace and a Group with a Query but no Role.
   
   Given no existing Paas,
   when a minimal Paas is created with a single namespace,
   and a Group with a Query, but without a Role,
   then the Group should not have been created,
   and there should be no users in said Group,
   and the correct labels were added onto the Group,
   and the rolebinding on the namespace points to the group, to the specified role,
   

2. Updating the Paas, adding a group with a role other than default.
   
   Given an existing, minimal Paas with a single namespace,
   when another query is added to said Paas (compared to step scenario 1),
   and said group has a specific role, other than default (see test_config),
   then the Group should not have been created,
   and there are no users in said group,
   and the correct labels were placed on the group, (no ldap things)
   and the rolebinding on the namespace points to the group to the specified role,
   

3. Removing the Paas.
   
   Given an existing Paas with a single group,
   when said Paas is removed,
   

Post scenarios: reset environment to clean slate.

Secrets

The capabilities are also used to test clusterwide quotas

Capability ArgoCD

What we test: creating a Paas with a capability named: argocd enabled.

Scenarios:

1. A minimal Paas with a capability named: argocd enabled.
   
   Given a minimal Paas and argocd capability configuration,
   when the minimal Paas is created with the argocd capability enabled,
   then the list entry in the applicationset should have been created,
   and a namespace with the name paasname-argocd should have been created,
   and a quota with the name paasname-argocd should have been created,
   and quota conform to the points below.

   Quota points:

   1. Assess a quota with the name paasnaam-argocd was created;
   2. Assess that the quota_label label was used as selector on the quota;
   3. Assess that the quota selector was set in such a manner so that only the paasnaam-argocd namespace is selected;
   4. Assess that the size of the quota equals the size of the default quota specified in the paas_config;

   Default_permissions points:

   1. Assess that a rolebinding for monitoring-edit is created
   2. Assess that the monitoring-edit rolebinding contains the argo-service-applicationset-controller service account
   3. Assess that the monitoring-edit rolebinding contains the argo-service-argocd-application-controller service account

Post scenarios: reset environment to clean slate.

Capability Tekton

Quota points:

1. Assess a quota with the name paasnaam-tekton was created;
2. Assess that the quota_label label was used as selector on the quota;
3. Assess that the quota selector was set in such a manner so that only the paasnaam-tekton namespace is selected;
4. Assess that the size of the quota equals the size of the default quota specified in the paas_config;

Default_permissions points:

1. Assess that a rolebinding for monitoring-edit is created
2. Assess that the monitoring-edit rolebinding contains the tekton service account
3. Assess that a rolebinding for alert-routing-edit is created
4. Assess that the alert-routing-edit rolebinding contains the tekton service account

Capability SSO

What we test: creating a Paas with a capability named: sso enabled.

Scenarios:

1. A minimal Paas with a capability named sso enabled without a capability quota.
   
   Given a minimal Paas and sso capability configuration,
   when the minimal Paas is created with the sso capability enabled,
   then the list entry in the applicationset should have been created,
   and a namespace with the name paasname-sso should have been created,
   and a quota with the name paasname-sso should have been created,
   and the quota conforms to the points below.

Quota points:

1. Assess that the quota_label label was used as selector on the quota;
2. Assess that the quota selector was set in such a manner so that only the paasnaam-sso namespace is selected;

3. Assess that the size of the quota equals the size of the default quota specified in the paas_config;

4. The Paas from scenario 1 is removed.
   
   Given a the Paas remaining from scenario 1 above,
   when said Paas is deleted,
   then the associated Quota should have been removed,
   then the associated Namespace should have been removed,
   then the associated list entry in the ApplicationSet should have been removed.

Post scenarios: reset environment to clean slate.

Configurable capabilities

What we test: adding a new capability with configuration

Scenarios:

1. Add cap5 in a Paas and check that it does not work when not yet defined in config