I think it's because cold water is already cold so being placed in more cold isn't too much of a difference, while it's the opposite with hot water. It's hot, so being put into a colder environment affects it even more.