Facial micro-expression is a superposition of micro-expression features and identity information of a subject. For the problem of identity information interference in micro-expression recognition, this study proposes a new method for facial micro-expression recognition by de-identity information, called two-stream difference network (TSDN). First, a two-stream encoder-decoder network is trained by a convolutional neural network, where the input of the micro-expression stream is a micro-expression image, and the identity stream is a facial identity image. The micro-expression image is the apex image, and the identity image is the onset image in the micro-expression sequence. The identity information and micro-expression features are recorded in the intermediate layer of the micro-expression stream, while the intermediate layer of the identity stream contains only the identity information of a subject. Then, the identity information is removed by the difference network, but micro-expression features are stored in the intermediate layer of the micro-expression stream. Given the sequence of the micro-expressions, the TSDN model of de-identity information learns the difference that stores in the expression stream. Two public spontaneous facial micro-expression data sets (SMIC and CASME II) are employed in our experiments. The experiment results show that our model can achieve a superior performance in micro-expression recognition.