Mr Hancock apologized after images emerged of him kissing a close associate he named in his office at the Department of Health and Welfare (DHSC).
Labor said his position had become “hopelessly untenable” and demanded that he be sacked if he was not prepared not to resign voluntarily.
However, a Downing Street spokesperson said Mr Johnson accepted Mr Hancock’s apology and “considers the matter closed”.
It came as the assistant at the center of the scandal, Gina Coladangelo, 43, was pictured loading overnight bags into her car hours after learning her affair would be revealed.
She then left without her husband in her £ 70,000 Audi Q7 from outside her £ 3.6million family home.
A Cabinet source told The Telegraph that if Boris Johnson was “next to” Mr Hancock right now, “it could fall apart pretty quickly.”
The frontbencher added: “If there’s a Barnard Castle moment, it’s going to be under a lot of pressure.”
Mr Hancock said he was “very sorry” for letting people down after The Sun posted a CCTV image of him kissing Ms Coladangelo.
The newspaper reported that the Health Secretary was having an extramarital affair with Ms Coladangelo, whom he had known from their years together at the University of Oxford and whom he appointed to DHSC last year.
She was first hired as an unpaid advisor on a six-month contract in March last year, before being appointed non-executive director of the department.
In a statement, Mr Hancock said: “I accept that I violated social distancing guidelines under these circumstances, I have let people down and I am so sorry,” he said.
“I remain focused on working to get the country out of this pandemic, and I would be grateful for my family’s confidentiality on this personal matter. “
Labor Party Chairman Anneliese Dodds said that while Mr Hancock had secretly maintained a relationship with an adviser he had appointed to a taxpayer-funded post, it was “a flagrant abuse of power and a clear conflict of interest “.
She said his admission that he broke social distancing rules meant his post was no longer tenable.
“He set the rules. He admits he broke them. He has to go. If he does not resign, the Prime Minister should fire him, ”she said.
Despite Mr Johnson’s determination to stand up for his minister, Labor insisted it would continue to pursue the case and not allow the government to simply “cover it up”.
“Matt Hancock appears to have been caught breaking the laws he created while in a secret relationship with an assistant he appointed to a taxpayer-funded job,” a spokesperson said .
However, Metropolitan Police said they were not investigating Mr Hancock.
A spokesperson said: “Obviously, the MPS does not retrospectively investigate issues related to Covid.”
The force had also not opened an investigation into how the CCTV showing the embrace was made public.
During a grueling briefing for reporters in Westminster, a No 10 spokesperson was repeatedly stuck in the face of questions from reporters.
The spokesperson insisted that the “correct procedure” had been followed in regards to the appointment of Ms Coladangelo, but declined to go into details or say whether Mr Hancock had declared their relationship with any. senior DHSC officials.
Pressed repeatedly to find out whether their violation of social distancing rules amounted to a violation of the law, the spokesperson said: “I refer you to the statement of the Secretary of Health. I have nothing to add to this.
The row echoes the political storm that erupted last year when Mr Johnson’s top adviser Dominic Cummings made his infamous trip to Castle Barnard, County Durham, in apparent violation of lockdown rules .
On that occasion, the Prime Minister resisted calls for Mr Cummings to be sacked despite widespread public anger over his actions.
Mr Johnson has also dismissed calls for the sacking of Home Secretary Priti Patel when it was found that she had intimidated officials working for her, and he appears determined to do the same with Mr Hancock.
Professor Stephen Reicher, the government’s behavioral science adviser to the Spi-B committee, said the prime minister sticking to aides and ministers who might have broken the rules was what made the repercussions of following the restrictions “Toxic”.
He told the BBC Newsnight: “What made this Cummings affair really toxic was not what Dominic Cummings did himself, was when the Prime Minister stood up for it and so an individual indiscretion has turned into a systemic problem, the feeling that there is one rule for us and another rule for them.
“And so for me the Hancock case changed when again the Prime Minister walked in and defended him, again giving the impression that you are part of the government, an adviser or a minister, you are being treated. in a way, but the rest of us are treated differently.
“And once you get a feel for them and us it seriously undermines trust, you get the impression that the government is not looking after us, on our side, but as other, that ‘he imposes on us. “
The Health Secretary has also been accused of hypocrisy after strongly condemning Professor Neil Ferguson, one of the government’s top pandemic advisers, when it emerged last year that he had met his lover in violation of locking rules.
The latest revelations come at a difficult time for Mr Hancock after he was fiercely attacked by Mr Cummings, who blamed him for the early days of the pandemic’s problems with PPE and the spread of disease in the nursing homes.
The Health Secretary is also deeply unpopular with some Tory MPs who believe he has been an obstacle to easing restrictions on coronaviruses.
Mr Hancock, who is said to have met Ms Coladangelo while they both worked at Oxford student radio station, has been married to his wife Martha for 15 years and the couple have three children.
Ms Coladangelo is Director of Marketing and Communications at Oliver Bonas, a UK retailer founded by her husband, Oliver Tress.
Vaccine Minister Nadhim Zahawi expressed his support for Mr Hancock, telling reporters: “The secretary of state apologized and said all he had to say.”