Whats the hardest thing about being veg*n?

For me its when people are eating meat and I'm just eating chips or other sides.. I hate eating side dishes cuz then it feels like I'm eating less.