Warm socks are a must, as are layers.
Bring a hat, gloves, and a scarf. You want to keep the wind off your skin when it's cold. This also allows you to more finely control your heat retention, because you have the option of taking off your hat for a second rather than necessarily needing to remove a layer.
The key with this is to not only have your layers, but to use them. When you're stationary, you want to be able to put everything on and stay warm. As you move and begin to warm up, you want to be able to remove layers to keep your temperature stable. When you stop again, you want to add the layers back on before you lose all that heat you just built up.
You're not trying to just stay warm. You're trying to regulate your temperature to keep it in a comfortable range.