From my testing, when using pulseaudio it seems the phone requests which mode the device should be in.
For example, when using music playback the device gets switched automatically to a2dp, while when making calls the device gets switched to handsfree_audio_gateway.
What I think may be the solution is to get a working setup with BlueAlsa and try to use Pulseaudio over bluealsa.
It may also be possible that using some combination of module-alsa-sink, module-alsa-source and module-alsa-card would be needed, considering there is both a sound input and output involved in a call.
The following page contains relevant information about the arguments available for these modules: Modules – PulseAudio