WHSmith has ended talks about buying troubled bookseller Borders UK, raising doubts over the specialist’s future.