Transcription factors (TFs) play an important role in regulating the expression of genes. The accurate measurement of transcription factor activities (TFAs) depends on a series of experimental technologies of molecular biology and is intractable in most practical situations. Some signal processing methods for blind source separation have been applied in the prediction of TFAs from gene expression data. Most of such methods make use of statistical properties of the gene expression data only, leading to the inaccurate detection of TFAs. In contrast, network component analysis (NCA) can provide much improved result through utilizing the structural information of the gene regulatory network. However, the structure of the gene regulatory network, required by NCA, is not available in most practical cases so that NCA is not directly applicable. In this paper, we propose to use particle swarm optimization (PSO) to find the most plausible network structure iteratively from the gene expression data, with the assistance of recently developed fast algorithm for network component analysis (FastNCA). This novel approach to TFA inference can thus take advantage of NCA, even when the required network structure is unknown. The effectiveness of our novel approach has been demonstrated by applications to both simulated data and real gene expression microarray data, in the sense that TFAs can be inferred with high accuracy.