It depends on the social environments of the countries the factories are moved to. If workers’ protections and environment regulations in those countries are lacking, the move will not help those countries or the US. The move will just serve as a way for companies to avoid regulations in the US. It will not help grow the consumer markets in those countries. The world economy will not grow as a result.
If workers’ protections and environment regulations are adequate in those countries, the consumer markets in those countries will grow, the world economy will grow, and the US economy will grow as well.
So free trade agreements, if set up improperly, will be harmful, but can be a good thing if set up properly.
By the way, other than avoiding regulations, there are two more reasons for American companies to move factories to other countries: to increase competitiveness by reducing cost, and to maximize profits. So many American companies will move factories to other countries whenever it is possible whether people like it or not. There is nothing much that anyone can do about it. The US government can only use taxation to reduce the benefit of increasing profits American companies will receive by moving their factories to other countries to offset the immediate cost to the US economy brought about by the move.