Nowhere in the world is safe," Count Olaf said. Not with you around," Violet agreed. I'm no worse than anyone else," Count Olaf said.
Figuratively, they escaped from Cout Olaf and their miserable existence. They did not literally escape, because they were still in his house and vulnerable to Olaf's evil in loco parentis ways.
The cold never bothered me anyway.
Edward smiled, I smiled, even Bernardo smiled. Olaf just looked sinister.