need help with lemon tree in pot
 

I have a Meyer Lemon in a pot - planted last fall, brought inside every frost, was doing great with lots of lemons - now they've all turned brown/black and fallen off, although I still have some blossoms.

Think I'm either over or under watering, not sure which. Anyone else have one, and how often do you water? I was told twice a week was about right. Am about to leave it for the summer where it will get watered twice weekly, plus any rainfall, and hope for the best, although I might ask my neighbor to take over for the summer.