Part of what's getting to me is the thought that, "I'm being so picky I'm even ruling out my lovers! How unreasonable is that!" But you have a cat, and a cat-allergic sweetie, and the conflict doesn't seem unreasonable when you're doing it.

That's one of the complications of poly. You aren't so much ruling out your lover as your lover's lover, who she lives with. (I like having a cat, and have a commitment to this cat, but Andy is the one who needs to have a cat.)