"If he's trying to connect an AUX switch and expression pedal
simultaneously with a Y cable, shouldn't this be set to Both instead of
EXP?"

Indeed. He describes the way it is supposed to work.