That’s seems odd and I’ve never heard of such an issue. The only thing I can think of would be a power-related issue. For example, if HDMI provides 5V to RPi, then that could explain the difference — once HDMI is removed, that 5V source is gone and hence the circuit is powered solely by the 24VAC to 5VDC converter on OSPi. If that happens to be insufficient, you will have a WiFi problem.
Most likely the issue is because your WiFi dongle is somewhat picky about the voltage. Although OSPi is rated to output 5V voltage, in practice the voltage may drop to 4.7V if the current draw is significant (this is due to the on-board PTC fuse, which progressively drops the voltage as the current draw increases). If the WiFi dongle is not tolerant to such voltage drop, it will perform poorly. There are several ways to address the situations, one is to change the WiFi dongle, the other is to bypass the PTC fuse to reduce the voltage drop.