>>1047611Clover, buckwheat, alfalfa, fava beans and BQ mulch come to mind as aggressive cover crops that may help break the clay by sending taproots down and breaking clay as well as scavenging nutrients from lower in the soil.
It'd be hard digging and pickaxe work, but if you have those worms like you say then trench composting might serve you very well.
Sand and organic matter like the compost will help too if you break the clay and incorporate them, just watch that the sand is coarse and not full of silt.