Yuya Osako's 73rd-minute winner was a huge relief for the Japanese, who were thumped 4-1 by Colombia four years ago during a miserable campaign in which they failed to register a single win.
The complexion of the game in Saransk was changed radically when Colombia's Carlos Sanchez was shown the first red card of Russia 2018 for handball in the opening minutes of the contest.
Shinji Kagawa scored from the penalty spot and although Juan Quintero equalised late in the first half, Japan made their superior numbers count, with Osako heading in from a corner to seal the win.
"To score a goal at the World Cup has been a dream for me since I was little, so I was extremely happy," said the match-winner.
Akira Nishino was only appointed as Japan coach in April, replacing Vahid Halilhodzic during a tumultuous build-up to their World Cup campaign.
But despite the win, Nishino said nothing was decided yet.
"Our players were aggressive from the start and I think that worked very well for us. This is just one win and three points, so we will save our celebration."
Brazil 2014 Golden Boot winner James Rodriguez came on for the last half-hour after labouring in training with a calf strain but could not inspire his team to pull back on level terms.
The result was bitterly disappointing for Colombia coach Jose Pekerman, with Robert Lewandowski-led Poland, then Senegal to come in Group H.
"It is tough to lose a player (Sanchez) so early, especially one so important," said Pekerman.
"Despite that we came back in the second half, but with Japan having an extra player, it was very hard to recover possession."
Travelling Colombia fans turned the Mordovia Arena into a sea of yellow for the early kick-off but were soon stunned into silence.
When Osako fired in a shot from Japan's first attack, Sanchez blocked the effort with a raised arm and referee Damir Skomina showed him a straight red card after pointing to the spot without referring to the Video Assistant Referee (VAR).
The Colombians bitterly protested but Kagawa drilled home the spot kick to put Japan ahead in the sixth minute.
It was the second-fastest red card in World Cup finals history, after Uruguay's Sergio Batista was sent off inside the first minute in 1986.
Colombia poured forward and their captain, Radamel Falcao, twice went close.
Pekerman made a tactical switch on 31 minutes, with Wilmar Barrios replacing Juventus midfielder Juan Cuadrado.
The pressure paid off as Quintero's low free-kick flew under Japan's wall and crept inside the post shortly before half-time. The goal was confirmed by goal-line technology.
Japan pressed after the break, forcing Colombia goalkeeper David Ospina into a string of saves.
To a deafening roar from Colombia fans, James came on for Quintero with just over 30 minutes remaining, before Barrios earned a yellow card for clattering Kagawa from behind.
But Japan finally made their extra man count when Osako, who was a constant menace to the Colombia defence, leapt highest from a corner and guided his header in off the post with 17 minutes to go.
The goal jolted Colombia into life as James then Barrios went close at the other end but they could not find the finish they needed.