We discuss the formation of a vortex polycrystal in type II superconductors from the competition between pinning and elastic forces. We compute the elastic energy of a deformed grain boundary, which is strongly nonlocal, and obtain the depinning stress for weak and strong pinning. Our estimates for the grain size dependence on the magnetic field strength are in good agreement with previous experiments on NbMo. Finally, we discuss the effect of thermal noise on grain growth.