The only downside? A LOT of people get very mad at the implications.
I wouldn't trust a human to generate enough entropy for any kind of key material. But I'd happily feed their output, and more importantly, the metadata around said output (like the ns delay between key presses) into the seed of a CSPRNG, (much more importantly, along with plenty of other sources of entropy).
The primary characteristic of a CSPRNG, is the inability to predict the next output, from the previous output. Once you get sufficient entropy to seed a CSPRNG, nothing you (correctly) mix into the state, can decrease it's security.
There is no folly in using human interactions to help seed a random number generator. Assuming you dont use the characters they type as the only seed input.