The kernel drivers for the bluetooth chipsets are part of the kernel modules provided by the backports package, so you already removed them.
However the kernel recipe provides the higher layers of the bluetooth stack. So you would have to remove the relevant elements from the kernel configuration.
Note that together with the kernel modules from backports the relevant firmware files get also installed. If your aim is to reduce the rootfs size then removing them would make sense too.
We regard our image recipes as demos and recommend that you write your own tailored image recipe rather than having loads of entries in your local.conf trying to tweaking a demo image to your production needs.