Usually difficulty in an open world game either scales to your level or has set levels for areas.
As for your last part about direction, goal, and story, have you played Breath of the Wild at all? That game is one of the most open worlds on the market, it constantly has directions for where to go (if you choose to follow it), shows you your ultimate goal as soon as you start the game and shows you various other goals when you enter specific areas, and it has a story (although it can be a bit light in that regard). Open world gameplay doesn't have to mean the absence of good game design in the hands of a capable developer.
I will agree with you though, Pokemon doesn't need to be open world (in a mainline game), I don't think Gamefreak is capable of doing it justice and it doesn't feel necessary to me. Although I do think they should get rid of the unnecessary things blocking you from going to later cities, if I want to take on gyms in a different order I should be able to, even if it makes the game incredibly difficult early.