Deblurring is a problem that has been studied for a long time. Extant works have primarily focused on deblurring real-world images. However, face images are different from real-world images. Because face images have fewer textures and weaker edges than real-world images, the deblurring of real-world images focuses on restoring the overall texture of the image; however, restoring the particular face structure (e.g., eyes, nose, and ears) is essential for face images. Recently, a convolutional neural network(CNN)-based deblurring network has been proposed. There are various types of CNN-based deblurring networks. Recently, multiscale architecture has been widely used; however, these types of networks need large amounts of resources. Further, because of the multitude of parameters, it requires a significant amount of time for inference. In this study, we developed a end-to-end network for face image deblurring, wherein novel CNN-based feature attention (FA) blocks are adopted, and a low inference time is achieved. Moreover, discrete Fourier transform (DFT) is employed for high-quality deblurring. FA blocks combine channel attention layer and pixel attention layer for feature extraction. The spectrum obtained using DFT is used as a loss function by comparing the ground truth image with the deblurring image. Experimental results show that the ours network is comparable to other deblurring networks in terms of performance as indicated by the PSNR, SSIM. Moreover we also demonstrated performance improvement by measuring the mean Intersection over Union (mIoU) of the deblurred image using a face-segmentation network.