Maybe Azumane decides that Oikawa is too much hassle to deal with. Or maybe he's not a very good cook, or he snores too loud or he has weird taste in movies or-

[He cuts off there and just looks down at his food.]

Or maybe he's sure he's going to screw things up at some point and that screws everything up.