Join the club. We have not had any debt for probably 30 years, except credit card charges which get paid in full monthly. I think I had one late payment (oversight) about 25 years ago. We own two homes outright. Yet our credit score jumps up and down by as much as 20 points, with seemingly no change in anything we do. It has never been higher than 816. Their algorithms need work.
Knowing I will never use credit again, I have frozen all of our credit agency accounts for safety.