1) We can accept that the Baron was so obsessed with extracting spice that he had no interest in anything else. This is an analogue to people in our era who began extracting oil swiftly and who had no concern or interest in what would happen 100 years down the road. But how could we accept that no one knew about the spice cycle prior to Pardot Kynes, or had any inclination about the relationship between spice and worms before that? It seems to me inconceivable that if the spice had been in supply for a very long time, given its importance, that scientists and other parties wouldn't have been all over Arrakis doing studies like Pardot did. Pardot appears to have been the first, and I take that to mean spice had only recently been on the galactic market. Even the Fremen didn't know about the spice cycle until Pardot taught them!
2) The BG, even as of 10,191, were still using a 'truth drug' for their truth trance and not the spice. Perhaps they hadn't updated their Truthsayer methodology yet to include the spice?
3) The factions in the Duniverse appear very concerned about the spice flowing smoothly, and there seems to be a lot of worry about disruptions. But if the spice had been around for a very long time, i.e. millenia, you'd think they would have already established a reliable system and wouldn't have to worry about basic logistics any more.
4) If the spice was so important, why is it that factions like the Bene Tleilaxu only began trying to synthesize it themselves after Paul's time? If we accept that Leto II was actively encouraging the BT to create it themselves, then why wouldn't they already have been trying that for thousands of years if spice had been known for all that time? I would suggest it's because spice hadn't been around for very long.
5) The people of the Duniverse seem perplexed by Paul's prescient abilities and we can probably take it for granted that a prescient of his power had never come before; the Guild Navigators were probably the strongest prescients previously. If the spice had been around for thousands of years, what are the odds that no one in all that time would have taken large quantities of it and had strong prescience? It seems more like the spice's full capabilities were only being discovered for the first time during Shaddam IV's reign.
6) Here's a passage from Dune:
Dune wrote:He thought of the filmbook Yueh had shown him--"Arrakis: His Imperial
Majesty's Desert Botanical Testing Station." It was an old filmbook from before
discovery of the spice. Names flitted through Paul's mind, each with its picture
imprinted by the book's mnemonic pulse: saguaro, burro bush, date palm, sand
verbena, evening primrose, barrel cactus, incense bush, smoke tree, creosote
bush . . . kit fox, desert hawk, kangaroo mouse . . .
Granted that future tech may allow 'filmbooks' to last a very long time, but you'd think if Yueh had showed him an ancient relic from 5,000-10,000 years earlier that would have been mentioned. It feels more like "old filmbook" may mean centuries, but not millennia. I also never got the impression in Dune that the botanical testing stations were ancient, from thousands of years earlier; it just all seems to fit better that they were constructed fairly recently, maybe during Elrood's reign, but that the Harkonnens didn't keep them up so well.
These details all seem to point to me to the fact that spice was a fairly recent discovery and that research on it was only in its infancy. If the spice was so important that the entire galaxy was brought to its knees by Paul's threat to destroy it, then I really can't accept that no one ever bothered to learn simple facts about it like how it's created. We could believe that the Baron might have decided to not care about that, but I can't really believe that whoever had the Arrakis-fief before him didn't either.
The one fact that seems to go against my theory is that the Guild seems to think they won't be able to navigate anymore without spice; this would suggest that they don't have alternative drugs they can use. But is this a case of spice just being superior to the other drugs and them not being able to go back to the old ones, making their case the same as that for the BG? Or is spice really the only drug they can use? If the latter, it means that spice was around since before the BJ. There is one passage that can help with this, but it doesn't help much:
Children of Dune wrote:melange (me'-lange also ma,lanj) n-s, origin uncertain (thought to derive from
ancient Terran Franzh): a. mixture of spices; b. spice of Arrakis (Dune) with
geriatric properties first noted by Yanshuph Ashkoko, royal chemist in reign of
Shakkad the Wise; Arrakeen melange, found only in deepest desert sands of
Arrakis, linked to prophetic visions of Paul Muad'Dib (Atreides), first Fremen
Mahdi; also employed by Spacing Guild Navigators and the Bene Gesserit.
-Dictionary Royal fifth edition
If Shakkad was a Corrino then it means this was post BJ, and therefore there was no spice before the BJ. If that is so, then it means the Guild did manage to operate prior to that without spice. If this wasn't a Corrino Emperor, i.e. it took place before the BJ, then maybe the Guild always used spice for Navigation right from the start.
My best guess is that Shakkad was a Corrino, and that he was maybe Elrood's father or grandfather; and that the Guild did use non-spice drugs before, just like the BG, but seem now to believe that they can't do without the spice.