“What would happen if you stuck in a top of the line graphics card like a n Nvidia [sic] GTX 1080 in an old motherboard from 2006?”
The best version of PCI Express that a 2006 mobo would be able to support is version 1.1. PCIe 2.0 wasn’t released as a spec until January 2007.
One of the nice things about the PCIe specification is backwards-compatibility through all versions (in theory). That means that you can put a card designed for a PCIe 3.0 bus into a PCIe 1.0a slot and have it work. You can also put a PCIe 2.0 card in a PCIe 3.0 slot and have it work.
Specifically for video cards, the caveat for backwards compatibility is that the slot must be a x16 (i.e., 16 data lanes wide) slot. It *may* work in an x8 slot but I doubt it, plus x8 slots are fairly rare.
Also, you’d need to make sure that the PSU in the system can A) provide the necessary output on the 12V rail(s), and B) have the proper PCIe dedicated power connectors for the card.
You will not get the full performance out of the higher-spec PCIe video card on the lower-spec bus; there just isn’t enough bandwidth between the northbridge chipset and the GPU with mismatched PCIe specs to allow for it.
So, to directly answer your question: Nothing much. It’ll work. You’ll be able to play games that your 2006-spec’ed PC would be able to handle in terms of memory and CPU power. It’ll just be a waste of GPU processing power (and probably cause cooling issues inside the case).
Edit: Cleaned up a couple of grammatical errors.