No, I'm pretty sure there's no way to have the TimeFactor not respond to MIDI start, stop & continue commands. My colleague, Russ, will know for sure though. He'll be back from vacation at the end of this week.
With Tempo Mode ON the TF Looper will follow MIDICLK sync but also MIDI START and MIDI STOP commands. There is no way to turn off the START and STOP behavior independently. The Looper should ignore CONTINUE commands though.