I’ve done nothing wrong, insists James Corden, as he calls restaurant row ‘silly’