Sometimes when a geocacher finds a letterbox, they think they've found a geocache or a game which includes trading for the "swag" inside the box. So they may take the stamp and leave a trinket.

To help prevent this problem see:

How can I tell if there are any geocaches near the spot I want to hide my letterbox?
How can I help prevent my stamp from being traded out for a trinket?