Unfortunately, could be many things. A few questions:

1) If you run your Protools without connecting to the H8000, do you get the distortion ? If no, it is not the Avid I/O. If yes, it is not the H8000.

2) Have you installed the latest beta drivers ? These are also needed for Lion.

3) You say you are using the AES routing. What have you set as the H8000 sync source ?

4) Are the connected AES lines showing as "locked" on the H8000 ?

5) Does the Firewire show as "locked" on the H8000 ?

5) What are the AES lines connected to ? Are there any other digital inputs connected to the H8000 ?

You should not rush to down grade, both because V5.6 is probably not the problem, and also because it is needed for Lion.