Father Patrick Bailey was in charge of the Holy Child and St Joseph Catholic church in Midland Road for many years.
He was also once the Director for Education for the Northampton Diocese.
Two of the charges refer to indecent assaults that allegedly happened in Bedford between November 1989 and October 2001.
They relate to two different men, both over 16 years of age.
Police step up patrols after Bedford car workshop broken into more than 35 TIMES
Bodycam footage shows police breaking into hot car to free distressed dog stuck in high temperatures
Burst water main on outskirts of Bedford causes concern
Three members of the same Bedford family die in Italian road crash
Bedford commuters: Only travel if absolutely necessary this weekend
Father Bailey faces two further charges of intentionally touching another man in a sexual manner.
These charges relate to alleged incidents that happened in Chesham, Buckinghamshire, between December 2012 and August 2013.
Father Bailey now lives at a nursing home in Aylesbury.
A spokesman for the Northampton Diocese said the priest voluntarily withdrew from all active ministry following his police interview.
Bishop of Northampton, the Rt Reverend Peter Doyle, said the Diocese would cooperate fully with the police.