Several scholars to withdraw as ‘far-right fringe groups’ accuse event of being ‘anti-Hindu’