Maybe Desmond remembers things more clearly because of how he is and for everyone else it's just kind of a vague memory. Desmond is the only one we have to base anything on though, until we hear from Alpert or Widmore again. Even then, Widmore might not make the connection or even see the people from the island again and we don't know how things work with Alpert.