It will not be permissible in Shari?ah to offset this annual fee against the interest accrued to the card holder in the same bank. The reasons are two fold;
Firstly, it is not permissible in Shari?ah to open an interest bearing account. Surplus money should always be deposited in a current account where no interest is received. Secondly, if somebody has deposited his money in an interest bearing account erroneously, he cannot use the amount of interest for his own benefit nor can he use it to settle any rightful claim against him. Since the annual fee of a credit card is a rightful claim against the card holder, he cannot set it off by the amount of interest accrued on his account. (Contemporary Fataawa pg.132)
A similar verdict will apply to other types of back charges levied against a person.
and Allah Ta’ala Knows Best
Mufti Ebrahim Desai