Say we have a dev environment, an integration, a pre-production and a production environment.
We have features A-G that are on everything, while new feature H is on dev+int only, so far.
We want to run our nightly tests across all environments (aside from production). We have a core set of 'reliable' tests that run, however, as we add new ones to it for feature H, it means that they'll be failing on pre-prod, as H isn't there.
How to mitigate this?
I see one option being having 'feature' tests that you turn on per environment, although given we currently tag with @proven tests or similar and pull it into Jenkins, I'm not quite sure how to handle this. Still mulling.