Home Office data published for the first time, shows around 600 people who applied to continue living in the area by September 30 had their application rejected.
Applicants can challenge a negative EU Settlement Scheme application by launching an appeal.
But the3Million, which campaigns for EU citizens' rights, is concerned about the status of those who are left "in limbo" waiting for their appeals to be concluded.
The EU Settlement scheme launched in March 2019 to regulate the immigration status of European citizens who live in the UK.
Those who have lived in the UK for five years, and meet the criteria, can receive settled status and remain in the country indefinitely.
Others who have lived in the country for less time can receive pre-settled status, which allows them to remain for a further five years. They can later apply for settled status.
The figures show that since applications opened, 26,610 people applied to continue living in Bedford, with 25,100 receiving a conclusion by the end of September.
Of them, 14,160 (56 per cent) received settled status and 9,500 (38 per cent) pre-settled.
The highest number of applications came from citizens of Poland (7,530), Romania (5,900) and Italy (3,140).
Monique Hawkins, policy and research officer at the3million, said many people had lost their job or rental opportunity while waiting for application and appeal outcomes.
She said: "Many people report not being able to get through to helplines, and find it next to impossible to get progress updates on their applications.
"For those who have been refused, the administrative review and appeals process face their own lengthy delays.
"We are extremely concerned about the length of time it is taking to unite people with their lawful status, and thereby their rights to continue living and working in the UK."
Though the scheme officially closed on June 30, EU citizens with limited reasonable grounds for missing the deadline can still apply to secure their rights.
Around 670 applications were submitted after the deadline in Bedford.
The Home Office said people with a pending application, are protected while the outcome of their application is unknown.
A spokeswoman said the EU Settlement Scheme has been an "overwhelming success", with 6.3 million applications received and 5.5 million people being granted permission to stay so far.
She added: “Caseworkers will always look for reasons to grant rather than refuse.
"Individuals can be refused on eligibility or criminality grounds, and if a refused applicant disagrees with our decision, they can apply for an administrative review or appeal.
“We have published non-exhaustive guidance on reasonable grounds for making a late EUSS application and take a flexible and pragmatic approach to considering them, and we’ve made millions of pounds available in funding for organisations to support vulnerable applicants.”