House of Fraser Group swung to a £43.9m pre-tax loss last year as the Brexit vote, terrorist attacks and online competition weighed on the embattled department store business.