I think the problem most likely has to do with the 6 wires from the adapter to the ISP pins on the board. Make sure the side with a stripe on the adapter cable matches the white stripe (next to the ISP pins) on the circuit board, and the ordering of wires exactly matches the ordering as if you were plugging in the adapter into the pin headers.
Perhaps a more reliable way is to stack and solder another 2×3 male headers on top of the existing 2×3 header, that way the pins will stick out enough so you should be able to plug in the adapter.