For me, what is right and what is wrong must be the same for everyone. That is, indeed, implied by the concepts of the rule of law and equality before the law. So, you cannot reasonably claim, of twin brothers Mo and Ahmed, neither of whom has ever committed a crime, that it’s legal for one to be in a particular place in the public space, and illegal for the other. Therefore, no-one may ban any individual, who has not committed and is not committing any harm to anyone, from any point of the public space.
With private property, of course, it’s different. You can say that Mo is rightly on your private property, because he’s a plumber, and you’ve invited him in to fix a leak. While Ahmed, if he was in the same place, would be trespassing, unless you chose to invite him in. But by calling one of them legal in the public space and the other illegal for being in the same place, you reject the idea of equality before the law. You also accept the dangerous notion that a government, unlike any other landowner, has a right to deny the passage of individuals across a piece of land it does not own.
Thus, there is no case for preventing any convivial individual from going anywhere he or she wishes within the public space. Otherwise said, visitors should be able to move freely, as long as they behave convivially.
Immigration, however – by which, I mean entry to an area with intent to reside there, either for a period or permanently – is a more complex matter. Someone who goes to reside in an area is, in effect, joining the community of people who live in that area.
Now, the criteria for joining a community are not as strict as those for joining a society. For example, there is no requirement to agree with a society’s particular set of goals. Nevertheless, the people already in the community must have the right to decide who is to be allowed to join their community. Shared culture is not necessary; but a willingness to accept the culture of the people already there is a requirement. And that is so, however small the community.
There is another aspect, too. If a small community agrees to let an individual join them, that individual should be able to travel to that community. He or she must not be prevented from doing so by a third party (e.g. a government) refusing permission for the individual to enter or to cross its land, such as at a border.
So, the current system of arbitrary borders and immigration controls is unjust and unworkable. This is made worse by deliberate encouragement of mass migration by some political factions, and opposition to it by others. The system needs radical change.