The real question is what your lead hand meant by "finish." If he meant knocking off early so you're not lifting in the worst heat, sure. But if you're pouring concrete or laying asphalt, the heat is actually working for you. Concrete cures faster and sets harder in that window, and hot mix asphalt has to go down hot or it's trash. Sounds like his rule was about staying comfy, not about the work.