It is the duty of the parents to ensure that the children are brought up in a suitable manner and are not exposed to hostile environments. On the other hand the grandparents have a right to see their grandchildren and the maintenance of family ties is also important.
Thus the more suitable method for you to adopt would be for you to explain to the members of the grandparents household that their behaviour is not suitable for being displayed in front of the children. Either they ensure that they don’t behave in this manner when the children are there otherwise the children will not be brought there any more.
If they wish they can visit the children at your own home. However it is very important that wisdom and tact be employed in getting this message across to them as the maintenance of strong ties of kinship is of utmost importance.
and Allah Ta’ala Knows Best
Mufti Ebrahim Desai