He's halfway to the door when she asks, stopping to growl in annoyance at her use of 'tolerate'. He doesn't think he's that much of a problem and this is some kind of conspiracy to blackball him from these outings for an unknown reason. The robots may be behind it.

"Fuck you. I'm gonna go even if no one wants to tolerate me."

And he will, even if he has no allotted provisions. He's getting the fuck out of the vault somehow.