You need to adjusted the stress calculations on your configuration policy. You should also examine the oversized VMs policy.
Set the stress calculation to be appropriate given the workload type. If it showing false positive, adjust the demand %s, adjust % time, and set an appropriate period for the workload.
On the oversized VMs, you can drop the lower boundaries on the % utlizations and increase the % of entire range to make the recommendations more conservative.
Also allow the policy to have stress account for spikes and peaks.
Between those things you should see VMs no longer be considered "stressed" and also "oversized".