New RPi got here yesterday, and has been running with the OSPi attached for 24 hours now, so it looks like the problem was actually the RPi.
I did try running the old RPi disconnected from the OSPi GPIO pins, and it worked without problems, which is why I was thinking it had to be the OSPi.
Anyway, the problem seems to be fixed for now.