If the only benefit is bragging rights and a chance to compete unofficially, then who REALLY cares?
However, as noted above, if there are financial considerations, gold/silver/bronze medals, or scholarship awards then considerations are very different.
In MOST cases, it is biological females/ciswomen who are being harmed when a biological male/trans-women is allowed to compete in the same category. In MOST cases, the biological male has innate advantages that cannot be discounted.
As for chess, my eldest sister was an incredible chess player. She won many national and a few international competitions. Her rating rivaled that of male players. She drew a grand-master in a well attended competition while every male there lost to him. Another sister is a genius mathematician who worked for NASA and later the IRS. She's smarter than most people, male or female.
Intelligence shouldn't be categorized by one's biological sex IMHO but it often is. The national Chess federation created the two categories to get women to join and to compete amongst other women who, by and large, had less experience playing chess so they'd learn and move forward. Nowadays, there's simply no reason to have the two categories because society no longer expects women to be less intelligent. (Well, intelligent, civilized society anyway)
Gymnastics is a category where trans-men would unfairly have an advantage over cis-men. Being smaller/shorter is a huge advantage in gymnastics so trans-men should NOT be competing with cis-men. Unfortunately, they really shouldn't be competing with cis-women either because of the testosterone hormones that build them up physically more than cis-women can expect.
So, a solution might be two-fold.
Create a trans-league. Yeah, it would be small, but so what? The volume of trans humans in sports is not very large since it is not very large in society either.
Ban the use of hormones in both male and female sports teams and only allow it in the trans-teams where it is expected and essential to their gender identities.
VOILA, situation resolved as fairly as possible.