My suggestion: Clean out the Speed potentiometer with some Deoxit spray (or similar cleaner).  What I think is happening is that you’re getting intermittent, spurious values from the Speed pot. While it *could* be a circuit issue, cleaning out the pot is a cheap fix.

You say that it’s always the Speed pot, so it may be jittering between values.  The ModFactor “thinks” that you’re adjusting the Speed, so it switches to fine-tune encoder mode. Just a theory, but I’d try that first. You may be able to dislodge dust using compressed air.  DON’T use WD-40 (it leaves a sticky residue).  Turn the pot several times after applying the cleaner; fully clockwise and counter-clockwise.