"Rebuild" is the wrong word, I'd say. It's more like... reshaping. The other person still remembers those interactions, so they haven't disappeared. But now they need to frame their perceptions around a new person. When you reshape something, you don't expect it to turn out the way it was before, even though you start with the same base. It's the same with people.