It's not really clear to me what you are connecting to what, using what inputs or outputs.

If you re connecting to the 1/4" input, you must drive it from a guitar or other unbalanced source.

If you are connecting to the XLR input, both pins 2 and 3 must be connected to either signal or ground. If either of these pins is "hanging" you will get the systems you describe.

Similarly, if using the XLR out put, both of pins 2 and 3 must either be connected to a signal input, or connected to ground. If either of these pins is "hanging" you will get the systems you describe.

If this does not help, please give me a very specific description of one particular scenario, saying exactly what output is connected to what input with what cable. Try different cables to be sure it is not just a bad one.