There are three kinds of houses (in this regard): A house that was built from the beginning for idolatry is forbidden, it being written "And you shall raze." If he plastered and decorated it for idolatry, thereby adding to it, he removes the additions, it being written "shall you cut down." If he placed (idolatry) in it and then removed it, then it (the house) is permitted, it being written "and you shall cause their name (but not the house itself) to go lost from that place.
Variantly: "And you shall raze their altars": Once you have razed the altar, leave it. "and you shall break their monuments": Once you have broken it, leave it. ("and you shall cause their name to go lost) from that place." In Eretz Yisrael you must pursue it (its complete destruction), and not outside of Eretz Yisrael.