With more testing I saw that there is an interaction with the trigger function and the pass/fail function. So if you trigger on channel 4 you wont see the change for some reason, the beeper still works, though.
It makes sense the scope would get confused, since the mask stays put, so if the signal (here channel 1) is not triggering it will likely fail the test (in the case of a sine wave it obviously would due to the phase shift compared to the mask).
Your best bet is to record data instead of triggering on channel 4 to see what the signal looks like.
Yeah, it's definitely related to the trigger function, since the BNC on the back is also trigger out, and when you have no pass/fail functionality enabled it always shows a sharply rising edge at the trigger point, i.e., on the set x-axis/time-axis trigger location.
So it likely checks at each trigger if the mask and the currently captured waveform match or not, and if not couples it out to the trigger.
You can actually see that if you switch "Aux out" on, the normal trigger signal disappears and it will stay at 0V.
I suppose the best option is to use a secondary/independent scope to see the aux signal.