If your facts are as stated and you adhered to all provisions in the offer contract, and you used the "standard" offer agreement format you are entitled to the deposit. Take it!! You lost the opportunity for exposure to other qualified buyers, the time value of interest, and your time was wasted on unqualified buyers.
If I remember the form correctly, your agent also gets a piece of the action, so you are still out some money.
As an aside, it doesn't seem that your agent is very skilled at bringing you qualified prospects or at closing deals; I was not there, just my opinion.