You do know that Drake himself estimated N=10,000. Didn't you?
You do know that modern scientists estimate the number could be millions.
We are not certain whether there is "just no way to get there, or for them to get here". That is far from proved. We only know that we can't conceive how to do it, for the nonce.
A wonderful little boy at a star party, in answer to the same argument said, "what if they live to be a hundred thousand years old?" Good thought kid. That would make a one generation space voyage encompass what we would consider access to most of the Milky Way galaxy. Thinking about this, why is it that an entire advanced civilization might tale to the skies with no intention of returning in a single generation.
When my father was born the fastest thing on the planet was an airplane that momentarily hit 163 miles per hour. The first commercial radio broadcast was still two years away. When I was born, 32 years later that speed had about quadrupled to 670 MPH. When I was born a television was wild technology. And not color. Today the Columbia reached an atmospheric, controlled flight of 17,000 miles per hour. When I was in high school we still took slide rule to succeed in math classes. There were rudimentary calculators but they were very expensive. Computers of any real utility were just getting started, were mainframes, and hard wired to boot. In 1989 my state of the art PC had an 80386SX processor that ran at a blazing 16 MHZ. I upgraded it to a DX33 MHZ version of the same processor with an astonishing 4 megs of ram and a 100 meg hard drive. People used to come to my house to see how fast it was. The internet was, at best, a very limited novelty. And one we accessed with a state of the art telephone modem that ran at between 110-300 bits per second. And access was very expensive. It was not until the 1990's, just over 20 years ago, that cable modems became available.
As a young college student the absolute state of the art science was to send a ship to the moon using vastly less computing power than I have in my Iphone. Actually less than I have ever had in a cell phone.
I am not going to get into the weeds about what are the absolute limits of physics as we understand them now. I have see far to many old ideas fall hard and fast to participate in the arrogance that says, "this far, no further".
God is a comedian playing to an audience that is afraid to laugh.