Otherwise the transfer from the host might not be finished when we try to read the data.
@ryanneph @zzyiwei