Mixing Single and Double Precision Floats

I've heard that there's a performance penalty with using both floats and doubles. Obviously mixing the two in a calculation would require a conversion. However, is there a penalty if you do some float calculations, then do some double calculations, without the floats ever touching the doubles?