The concern we have with minors engaging in sexual activity with adults is due to how the child processes the relationship.
We can agree that a minor is mentally equipped to consent to a direct action. A direct action, in this case, is something they can immediately choose to perform or not perform. So a minor can choose to play chess, they can choose to have intercourse, they can choose to drive a car, etc.
The trouble comes from the minor's ability to process the consequences of that decision and interaction, along with any indirect results stemming from it.
Minors are building their understanding of their society. The relationships they experience help to build what they can expect from the rest of society. They cannot effectively protect themselves from the harmful parts of society, which is why they are protected as minors.
Since this involves pedos, I'll get specific. A child does not understand the importance of sex in society. They do not understand that someone enjoying your body is not the same as someone enjoying you as a person. They do not understand that a person who wants them for sex is not the same as a person they can trust, or a person who will protect them. But the sexual activity is pleasurable and they are rewarded with that pleasure, and maybe even praise from an adult when they engage in it. They are learning what to give and expect from society when they engage in these behaviors. They are not equipped enough to see this is an individual person looking for one thing from the child. Any rules or expectations that adult teaches the child will exist in the child for life, and may bleed into other relationships, sexual and non-sexual.
To an adult, sex is just sex. To a child, sex is many things, and there is no guaranteed way to prevent it from being those things if a child engages in the activity, especially with an adult.