I believe the original form was zingiber. Checking Wiktionary, I see it has gingiberi as a later Latin form, and zingiberi as an alternative to zingiber.
I think the phrase would logically be panis zingiberis, "bread of ginger," or panis zingiberi (the later form doesn't have different endings for different cases). At least if you want to say "ginger bread does so and so." The form panem is the one you use when something is being done to the gingerbread: eating, baking, whatever. But the form panis is what's called the "citation form."
But I'd also suggest that since gingerbread is a cookie/biscuit sort of thing, the noun that would convey the idea to an actual ancient Roman is crustulum rather than panis. So crustulum zingiberis might be the most authentic form.