(Ibid. 20) "that it is not a fruitful tree": This is a barren tree. If in the end we include (to be cut down for the siege even) a fruitful tree, what is the intent of "that it is not a fruitful tree"? We are hereby taught that a barren tree takes precedence (to being cut down) to a fruitful tree (if only one is needed for the siege). I might think that this holds true even if the barren tree is worth more, as per the words of R. Elazar b. R. Shimon; it is, therefore, written "Only" (to exclude such an instance).
When Is a Barren Tree Not Truly Barren
Sifrei Devarim 204:2