Driving immediately can cause increased wear and tear on engine components due to the lack of lubrication at low temperatures. It can also lead to higher emissions before the engine reaches its optimal operating temperature. The extent of the damage depends on factors like the outside temperature, the age and condition of the vehicle, and the length of the drive.
In short, while you *can* do it, it's better for your car's longevity and efficiency to at least let the engine run for a short time (30 seconds to a minute is usually sufficient in modern cars) before driving, especially in cold weather.