Turns out a couple of bolts are missing or fell out. The bolts tighten the power steering rack to the car, and the torquing was making the noise. With the bolts in place, the noise is gone!
Next up is window rattle... they ordered a new window regulator assembly. What a pain!
After that is the rear axle clunk which they will tighten once I get a new set of tires.