He is not liable unless the "damager" leaves his domain and causes damage, in which instance the damage is assessed (to be paid) from the best of his land — a fortiori (if the damage is) to consecrated property.
A Damager Must Pay from His Best Land
Mekhilta Tractate Nezikin 14:2