The manager bore the brunt of the Kenilworth Road crowd’s frustrations at the final whistle on Tuesday night as his side were held to a 1-1 draw by lowly AFC Telford.
Despite the barrage of abuse he received when walking down the tunnel at full time, Brabin feels he is still the right man for the job.
He said: “I’m as frustrated as the fans, I genuinely am.
“I can understand their disappointment but I just plead with them to hold on to their frustrations.
Setchell feels Bedford Town deserved their opening point at Stratford
Clubs all set for big kick-off as Bedford Town, Ampthill Town and Real Bedford begin campaigns
Bedford's women cricketers make it three wins in a row
Ampthill hit six to get the season off and running in style
George Bernard Shaw treated in Luton after tree fall
“If it means me coming out at the end of the game and them hurling it towards me that’s fine.
“We want to be back where we belong and I just don’t think at the moment that gives a positive vibe.
“I’d been lying to say I don’t hear anything, but I’m professional enough to realise that happens in football.
“I’m the manager, I make the decisions. It’s me who’s in charge of the team that’s nine points off the top of the league
“From my point of view, I’m working as hard as I can behind the scenes. I really believe I can get success here.
“I genuinely have got a bond with this club too as I know there’s been a couple of comments from people that I’m not a real Lutonian shall I say.
“But I genuinely am. I’d love to get this club back where it belongs and I genuinely believe I can.”
There was an air of unease surrounding Kenilworth Road before Robbie Willmott broke the deadlock on 41 minutes in midweek and Brabin admitted it could have an adverse affect on the players.
He said: “I think having them here in numbers, that’s a massive thing.
“When there was 11 minutes on the clock I sensed a little bit of anxiety. I totally understand the supporters though, I’m a supporter myself.
“I’m lucky enough to be involved in football, but it takes one tut in the crowd as 20 people do it and then 100 people do it.
“We’re lucky enough that we have a lot of fans here, but it only takes a few 100 tuts at the same time and all of a sudden it does filter on to the pitch.
“We’ve got a really good young side here and I just want to focus on the players being positive and working on enjoying their football.”
Brabin’s substitutions were also criticised last night when he withdrew James Dance and Danny Crow, while delaying the introduction of Adam Watkins until the 81st minute.
He said: “Both were injured. James Dance recently broke his cheekbone and was struggling with breathing through his nose.
“I thought he tired a little bit and Danny Crow was struggling with his hamstring.
“We asked him a couple of times as we were going to make another substitution, but we asked Danny to keep us informed and he was struggling, nothing other than that.”
Although the result saw Hatters remain in sixth place, Brabin remained confident the title was not beyond his side.
He continued: “I still think there’s a long way to go.
“You always find there’s a team that comes from nowhere and gets promoted.
“Yes I’m disappointed, yes I can understand the supporters’ frustration, but let’s not give up yet. Let’s not throw the towel in and I genuinely believe we can build on this.
“Everyone knows the history of the club and what’s happened in the last three or four years, but all I can try and get across is that’s not my fault.
“I know what’s happened and I’ll be trying my best to try and get them out of the league.”
For all the latest Luton Town FC news, reports, stats, player profiles, live match updates, prediction table and a complete fixture list, visit our Hatters Microsite.