Yeah, immigration gives foreign, low-skilled labor access to markets (construction/services, mostly) that they can't get through international trade, but the reverse is also true for manufacturing. Plus immigrants add to local demand, so I'd imagine it's a wash.
In any case, you can't simultaneously advocate for 1)
intervening to stop the offshoring of US jobs, 2) protecting US citizens from competing with immigrants in the domestic labor market, and 3) improving living standards in the developing world to ease the demand for labor movement. If the world's poor are going to see any gains, it will be through access to developed markets; labor or goods/capital,
something has to be allowed to cross borders at some point.