The men were actual gold diggers, that is, miners, panning for gold.
It was those women who hung around the miners (for their money, obviously), that were coined gold diggers.
Some of my 1on1 students in LA are very wealthy, and I’ve had to help them many times get rid of gold digger girlfriends or play defense against girls they are dating or seducing.
Here in LA, there are a lot of women out there who are specifically after your money.
In those days, times were rough for both men and women.