Yeah, my MIDI routing is completely different from my audio path, too. If you haven't already, you might want to make sure you've got the latest software update (V3.0.0[19]) for the PF (well, all of them, actually). There have been some improvements to MIDI Clock implementation in PF, which might help in your situation.

There's a new System >> MIDI option, too, that has to do with clock smoothing. I can't remember if it's in all of the Factors now, or if it's just a few (I think it's all, but I couldn't find the reference on the PF update page), and I can't remember exactly what it's called (I'm not with my pedalboard at the moment), but it's pretty obvious.

I've seen the PF go back and forth between tempos before, like you describe, but it's never been completely random, always within about +/-10 bpm. Updating to V3 corrected this, especially after turning on the clock smoothing option.

If that doesn't help, can you provide all the System >> MIDI settings for all your pedals? That might help track it down.

Enjoy your Space!!