Answer
The words of the hadith “The people of the grave are let loose” is not taken in the litreal sense, rather the meaning is that the punishment of the Qabr (grave) is eased during the month of Ramadaan. During the month of Ramadan multitudes of people are saved from the fire of jahanam.
It does not refer to any soul leaving the grave and making ziyarah of any person or place.
and Allah Ta’ala Knows Best
Mufti Ebrahim Desai