It depends how that character is used.
Loki is the best example of how it could be poorly done and also how damn lazy Marvel can be at incorporating it. You have a Loki that we’ve followed through multiple films all the way up to Endgame, seen him as a villain, seen him as an anti hero and eventually he ends as almost a complete good guy and, more importantly, true brother to Thor. It took hours, multiple films, ups and downs to grow to love THAT version of Loki and so it was genuinely heartbreaking when he eventually died while trying to do something right.
So next up is the Loki show with a new Loki fresh out of Avengers. As you said in your post, yes it’s Loki but it’s not THAT Loki. And what is the very first thing they do in the Loki show? They completely abandon that concept of him being different and within the first 5 minutes they show him the Hallmark highlights of his life and boom they’ve basically tried to tell us he’s now the exact same Loki we lost in Infinity War.
It’s pathetic storytelling. The new Loki, the trickster god himself, who just half an hour before seeing that video, was trying to kill the Avengers and take over Earth does a 180 because of a video. I emphasise trickster god because Loki would absolutely not take that video at face value. He, being devious himself, would likely assume that the TVA were just playing made up lies to him to make him think he’s good now, at the very LEAST he would not feel the emotional connection that he suddenly does. If you watched a video of an alternate you having a relationship you wished you had, it would be nice sure but not having lived it you are not going to take those emotions and adopt them yourself.
So yes there is a chance that they could utilise the multiverse that way in terms of bringing in similar characters but completely different but I have zero faith in them committing to that concept. For example if they bring in a new Stark, the maximum I’ll expect are a very quips about how the new Tony isn’t the same as the old. But in terms of story I would expect him to behave exactly as RJD would but with a different face.