A family of moochers comes along and pretends that they built it. To continue this farce they even take pictures with it...claiming it as their own.
As soon as the pictures are taken, along with the credit for building it...their desconstruction team kicks into action. They take painstaking efforts to destroy your castle grain by grain.
Now, don't you feel guilty about leaving it in the first place?