You can set-up the encoder simulation in the motor wizard.
I set up a system, too and it seems to work fine. Have you measured the A and B signal as different signals.
Just to check the signal you can take two probes.
Probe 1:
GND: A\
Measuring point: A
Probe 2:
GND: B\
Measuring point: B
So that will give you two signal
In the end you will need encoder counting device. If the B-signal is in advanced then count positive, if B-signal is lagging count negative.
If you do a 4x decoding then every count equals the resolution. As example if resolution is set up to 10microne by 4x decoding every count is 10micron. If you do just 1x decoding every count is 40micron.
