You should still go. Allergies are understandable and even overlooked with the right words. Use the opportunity to set something up for a later time. It could still be a good party for you.