you're right that the creation story doesn't come out and say that adam was the "only" man, at least not in those words. However, teh creation story does tell us that Adam was teh first man and that he was created on the fifth day. It never mentions other people being created and i don't think we can assume that there were other people. It seems very logical that Cain married and sister and the population grew from there. The genetic makeup of that day was almost flawless so incest wouldn't have been a problem