I believe the main issue is BEGED ISH, wearing the other sex's clothes.
Pants as worn by men delineate the legs. This also leads to questions of
tzniut [modesty], with regards to both men and women, but that is for
another discussion.
It should be noted, however, that Dr. Lisa Aiken, who authored the book, "To
be a Jewish Women", wanted to include a prohibition against pants. Rabbi
Mordechai Tendler, who was her halachic reviewer, refused to allow such a
statement, since, although pants as they exist today are a problem from both
standpoints, namely men's garb and modesty, there exists the potential for a
pair of pants to be made of a style uniquely for women that would also allow
for modesty.