The Cod Army took the lead through Jamie Vardy’s 10th-minute strike, but Hatters striker Amari Morgan-Smith was clearly fouled in the build up to the goal.
Referee A Davies elected to play on though, a decision which bemused the Hatters players and management staff as Brabin said: “I think everyone in the ground has seen it except the referee and they’ve gone on and scored.
“I know people are saying about decisions and stuff, but I keep saying we’ve got to be bigger than that and better than that.
“I don’t want to use that as an excuse and I thought we drove on from that.
Setchell feels Bedford Town deserved their opening point at Stratford
Bedford's women cricketers make it three wins in a row
TK Maxx launches free gift card giveaway to celebrate summer of Great British Sport
Clubs all set for big kick-off as Bedford Town, Ampthill Town and Real Bedford begin campaigns
George Bernard Shaw treated in Luton after tree fall
“Fleets (Stuart Fleetwood) is unlucky with one which has just gone wide, Robbie (Willmott) has some (efforts) across the six yard box which I’m disappointed we’ve got nobody on the end of.
“(Aaron) O’Connor’s got in and we haven’t got our shots off. Overall I thought the first half performance we were the better side and unlucky to be goal down.”
Brabin wasn’t happy with his side’s reaction to falling 2-0 behind though after Jamie Milligan netted a penalty midway through the second half when Morgan-Smith fouled Ryan Jackson in the box.
The boss said: “Probably the most disappointing thing was the last 20 minutes, because I thought the first half we did enough, created enough to have certainly gone in front.
“I thought we started the second half quite bright but as soon as they scored the second goal it seemed to deflate us and the most disappointing thing for me was we didn’t roll our sleeves up.”
Hatters made a host of changes to the side which disposed of Hendon in the FA Cup, with the likes of Keith Keane, Alex Lawless and Morgan-Smith all fit enough to return to the starting line-up, but Brabin didn’t feel that had an averse affect on his side.
He added: “We welcomed a number of players back. I’ve been saying we need to get our best players back and it was important to get them into the fold. I’m just disappointed that we haven’t got a result.”
For all the latest Luton Town FC news, reports, stats, live match updates, prediction table and a complete fixture list, visit our Hatters Microsite.