If you look under the hood of your car you should see green or orange coolant in one of the plastic reservoir of your car. At times it may be water, but it is usually coolant. If your coolant is brownish or has brownish spots, there's oil mix with the coolant. (Unless your radiator is rusted)
If your not familiar with with coolant and engine oil, the rule is that they should never mix. Oil should never mix with the water or any coolant in the coolant system, and if it does, something is going wrong with your vehicle and it will need your attention.
What does it mean? To answer these questions, let’s first understand the basics of engine functioning, which will also help you to identify, which type of the problems might be causing these liquids to mix in together.
Oil and coolant shouldn’t ever mix. Oil is a lubricant that is used to cover all the different movable parts of your car’s engine. As you drive your car, the engine generates an immense amount of heat, and the oil can cover these move part even when very hot.
To cool down your engine, coolants are used to keep the temperatures of the engine steady and under control. In your car, four wheeler vehicle, or other vehicle, the engine is designed so that it is sealed, and can't be interact with outside elements. This help protect the engine from foreign elements that might cause wear or damage to the engine.

Via : https://www.trucknews.com/features/maintain-heavy-duty-engine-coolant-system/
As the cylinders in your car move, they generate lots of heat. The cooling system of your car helps to cool down your engine, allowing the combustion chambers to operate without getting too hot.
Between the engine block and the cylinder head is the head gasket. This gasket is important because it maintains the seal between the outside and inside areas of the engine. If you notice coolant in your oil, it's likely your head gasket needs to be replaces, since it's letting in coolant.
The head gasket seal may be broken or damaged because of excess heat due to several reason, including not enough oil, not enough coolant, and running the engine at excessive heat levels for to long.
How it Works
If you notice anything strange about the color of your engine oil, one conclusion you can make is that the oil and coolant of your vehicle’s engine mixing because the head gasket seal is broken. The head gasket (Amazon Link) may have malfunctioned or become broken.
Usually, if you open the radiator cap (when cool) or look at the reservoir tank, you'll notice the mixing of oil and coolant is the head gasket is broken. It may have a milky appearance or you'll see brownish dots of oil.
So, what should you do? Stop driving your vehicle and get it fixed. The longer you drive it the move damage will occur. It will eventually ruin your engine and may cause your engine to malfunction, which might be dangerous.
However, you may not notice problems immediately. If you notice your car is not quite working as it normally does, then it's a good idea to check the engine oil and radiator fluid once your car has cooled down (about 30 minutes).
Overworking your engine, improper servicing, accident impact, and age-related wearing are common causes of the mixing of the engine oil and the coolant.
Via : http://m.wikihow.com/Check-the-Fluids-in-a-Car
An engine’s head gasket (Amazon Link) failing may be one reason the oil and coolant have mixed. Lets look at other reasons that may be causing these fluids to mix. According to the leading mechanics and vehicular engine experts, some other reasons why your oil is ending up in the cooling reservoir are as follows:
If you have opened the reservoir or the radiator top and see that the coolant and oil have mixed, or you have noticed your vehicle has started to run oddly, it is time for you to take immediate action.
Unless you have experience taking apart engines, you should call an experienced mechanic who can inspect your vehicle and possibly repair any damage that has occurred. Check around to see if they have experience with this specific issue so you can get a proper diagnosis and a possible solution.
Depending on whether the problem is big or small, the mechanic will give you different a different diagnosis. Be prepared to make some hard choices. You may need to have an complete engine overhaul or you might want a second opinion.
The mixing of coolant (Amazon Link) and engine oil (Amazon Link) can cause severe damage to the engine of your car and lead to big problems. Getting regular maintenance on your vehicle is essential. Poor use and maintenance of you vehicles tends to put the engine of your vehicle at risk of getting damaged.
If damage takes place because fluids have mixed, the engine will not function properly. At times, sparking, fire, or even small explosion in the engine of your vehicle have been know to occur, putting your car and your life in danger. So, addressing this issue quickly is important to prevent major problems.
Oil in the coolant reservoir can be a huge issue but if caught quickly enough will save you time and money. It is highly advised that you get a mechanic to inspect your engine if you suspect or know of a problem. You could be saved from major issues and can drive with more assurance in the future.
Should I replace the oil and coolant if the two liquids mix?
Not at first. If you notice the coolant and oil are mixing then you need to stop driving the car and find out the reason why this is happening. It won't help to replace the oil and coolant unless you've fixed the problem first.
Should I use head gasket sealer to repair a broken head gasket?
Maybe? A liquid head gasket sealer that you pore into the radiator can help seal the gasket, but the results will vary greatly. If you want to be sure not to have more problems down the road, replace the head gasket.
Photo Credit: