Nick Griffin, the BNP's leader, has shuttled to and from Brussels since European elections last month in a push to take a central role in a new grouping of far-Right MEPs.
But Mr Griffin accepted on Tuesday that there was "no possibility" of the BNP finding sufficient support. "We have failed to form a formal group. It's disappointing but not surprising," he said.
As an "unattached" party outside a European political grouping, the BNP's two MEPs will lose out on extra funding worth £1 million a year, will not get a party office or administrative staff, and will not possess the right to vote on the parliament's main committees.
The parliament's rules state that at least 25 MEPs from seven different member states are required to form a new bloc.
Mr Griffin, along with his colleague Andrew Brons, had been in talks with Hungary's far-right Jobbik party; France's National Front; Belgium's Vlaams Belang and Ataka, the nationalist Bulgarian party.
Their plans to forge links with other parties, such as Italy's Northern League, failed because other Right-wing groupings feared that the BNP's controversial reputation would lead to isolation.
"We will remain in the unattached members working informally with some of the groups such as the Front National and Vlaams Belang," said Mr Griffin. "We held discussions with several others as well but haven't had any joy."
Sajjad Karim, a Conservative MEP for North West England, said: "The BNP being unable to form a new group is good news. This proves how ineffective the BNP will be in the European Parliament."
The BNP won two seats in Britain's European elections last month. Mr Griffin was elected in the North West of England region after winning eight per cent of the vote, while Mr Brons, a 61-year-old retired politics and law lecturer, picked up a seat in Yorkshire and the Humber with almost 10 per cent of the vote.