I was training my image model wrong for a solid year and didn't know it.

I kept feeding it these huge, messy datasets from random places, thinking more data was always better. The tip off was when a friend in Austin asked to see my training logs and pointed out my validation loss was basically flat after the first 50 epochs. I was just wasting compute time and getting nowhere. Has anyone else hit a wall with their training where the numbers just stop moving?

2 comments

2 Comments

anna_grant19d ago

Sounds like @dixon.felix's friend learned the hard way too.

dixon.felix19d ago

My buddy had the same thing happen with his text generator. He was training on a massive, unfiltered scrape of forum posts for months. Turns out the noise in the data was so bad the model just learned to output gibberish averages. He had to scrap the whole dataset and start over with clean sources.