For the first 4500+ million years of Earth’s existence the human population was zero.
“Has there ever been a one million year span where the human population was greater than zero and less than 26,000?”, the answer appears to be no. Homo sapiens didn’t speciate until much more recently than one million years ago, so we simply haven’t been around long enough to have had a million year span of anything.
The genus Homo appeared somewhere around two million years ago, so if your definition of human is sufficiently broad you can at least find a million year span of something. It seems implausible, though, that the entire genus would have been constrained to fewer than 26,000 members for an entire million years, and some speculative analyses seem to confirm that. You probably wouldn’t recognize most of them as human, anyway.
For what it’s worth, world population estimates are between one and ten million at 10,000 BC. Estimates further into deep history get pretty speculative.