I made some progress by changing the input sensitivity of the DAC and ADC but it’s weird that I have to do that because when using the digital inputs, there shouldn’t even be any DA or AD conversions. Anyways, it worked and I picked up a lot of db’s so I’m good.
I also found how to select ADAT for the digital input/output but it appears that one has to decide whether you want to use ADAT, AES or SPDIF but apparently only, one can be used at a time. I’m ok with that as I work in 48k, so 8 digital I/O is plenty.