Cold air might result from a drafting issue. It could also be a wind-induced downdraft and can be fixed with a wind cap. Depending on the location of your fireplace, there can be a negative air pressure. This is when the pressure in the upper levels of the house tend to be higher than the lower levels of the house due to the tendency of hot air rising, also known as the "stack effect". It may also be resolved with a fresh air intake.
