It’s called the cold shock response. When the cold receptors in your skin are all suddenly stimulated they cause an involuntary gasp and, for about a minute after that, hyperventilation. If you fall into chilly water, the cold shock response will kill you long before hypothermia does.
Either that first gasping breath will fill your lungs with water (drowning you instantly), or the hyperventilation will make swimming almost impossible. In the UK, 67 per cent of drowning victims are strong swimmers, and over half of those are within 3m of the shore or the side of their boat when they drown.