Early Stellar Flybys are Unlikely: Improved Constraints from Sednoids and Large-q Trans-Neptunian Objects