Marco Fabian equalized, Oribe Peralta scored the go-ahead goal and Javier Cortes added insurance in injury time for El Tri, which had never won a medal in Olympic soccer and is now guaranteed at least a silver.
The victory came three days after Mexico surrendered two second-half goals and needed extra time to beat Senegal 4-2 in the quarterfinals.
Mexico will next face Brazil in the Olympic final on Saturday at Wembley Stadium.
The result means CONCACAF will be bidding for both the men's and women's titles, with the United States beating Canada in the women's semifinals on Monday and advancing to the final against Japan.
Japan was bidding to become the first nation to send both a men's and a women's team to the soccer gold-medal match at the same Olympics.
Peralta tallied the winner in the 65th only moments after forcing a save from goalkeeper Shuichi Gonda.
Gonda rolled the ball out of the middle of the penalty area to Takahiro Ogihara, who was quickly challenged and turned back toward his own goal. Peralta charged in, stole the ball and dribbled to the edge of the arc before hitting a right-footed shot into the left side of the net.
Cortes, who replaced Javier Aquino only moments earlier, assured the victory in the third minute of injury time. Peralta backheeled a ball off the right touchline to Cortes, who sprinted into the penalty area, shouldered off a challenge from Daisuke Suzuki and finished low past Gonda.
Japan had not allowed a goal in its previous four matches in the Olympics. The last goal it did permit came in a 2-1 victory over Mexico in a friendly five days before the Olympics on July 21 in Birmingham, England.
Yuki Otsu gave Japan the lead in the 12th minute with his third goal of the tournament. Keigo Higashi played a ball between Carlos Salcido and Jorge Eenriquez to Higashi, who took one touch, turned and drove a right-footed shot from near the arc into the far-side netting.
Mexico equalized in the 31st from a corner kick. Giovani Dos Santos' corner kick was nicked by Enriquez at the near post and Marco Fabian climbed from inside a pack inside the six to deflect it past goalkeeper Shuichi Gonda.
Mexico played the second-half without Dos Santos, who was replaced by Raul Jimenez at halftime.
El Tri's previous best finish at the Olympics was fourth, when it hosted the Games in 1968.