Does shaving your legs impact running speed?
Server location might impact SEO in terms of speed, primarily.
However, the fact “speed impacts SEO” is technically true, but often misunderstood.
In other words: there are far more important aspects to server location alone.
Website optimization and server performance often affect speed a lot more drastically than server location.
Using a CDN can further help.
Of course, website design and content quality are probably by far the most important in terms of SEO.
With all that in mind, regarding speed and server location effect on it:
website audience geo distribution plays a role (among other things) – if visitors come primarily from one area, server location may have a bit bigger impact.