The investigation into age and gender identification has been receiving more attention from researchers since social and multimedia networks are becoming more popular nowadays. Recently published methods have yielded quite good results in terms of accuracy but have also proven to be ineffective in realtime applications because the models were too complicated. In this paper, we propose a lightweight model that can classify both age and gender. The number of parameters used in this model is 5 times less than existing models. Experiment results show that the accuracy of the proposed method is equivalent to state-of-the-art methods, while the speed of age and gender recognition decreases by 4 times on the Audience benchmark.