My brother-in-law is a bluebird expert. He has hundreds of boxes in public areas and golf courses in Medina County Ohio. He replaces the nest while the mother is nesting her eggs. About six to ten days after the eggs are laid, he'll wait until he's sure the mother isn't in the nest. He washes his hands with Netrogena soap to get the odors off his skin. Then he reaches into the box and gently lifts out each chick. He thoroughly inspects each chick for blow worm larva. They attach themselves to the chicks and suck the blood out of them and kill them. He has a Kool Whip container with dried grass and he places each chick in the grass. He then takes out the nest and throws it away. He has an old paint brush and he uses it to get everything out of the box making sure there is nothing left in the box.
Then he reaches into a trash bag and grabs several handfulls of dried grass he has saved and stuffs it into the box. He uses his fist to make a pocket in the grass as you would to form a pocket in a baseball glove. When satisfied, he will place the chicks back into the nest.
His nests hatch hundreds of bluebirds a year. He put a box in my yard in Ohio and taught me how to do this. It is quite amazing.
So don't worry about your hubby tossing the nest. There is plenty of pine straw around. The two of you might consider washing your hands and taking some pine straw and building another nest. I'll be in TV by Thursday of next week. If you would like me to help you just send me a pm. Good luck and keep up the Bluebird work. As you may know, they were nearly extinct in the 60s and it's only through the good work of people like you that they are thriving now.