Replacing Rear window Glass: PREP?

I've got an old 89 GMC SIERRA C2500 and when I bought it it had the 4-pane sliding glass in the rear, with one of the panes broken. I've removed the ENTIRE rear window. Glass, Seal, Everything. And I've bought a used non-sliding window and removed all the old seal. My problem is that the old seal On my truck has been extremely hard to remove and there are tiny bits of the old polyurethane seal still on the metal i've scraped for hours trying to remove these and im wondering how crucial is it that this Surface is Perfectly smooth? im using polyurethane glue/caulk to seal the new glass so will this cause a problem with my sealing and bonding??