Digging characters in ps ancient books

First of all, your pixels are not very high. No matter what matting method you use, the edges will not be smooth. In addition, if you want to add it to other backgrounds, put this layer on the background without matting, and then change the blending mode to filter or overlay the positive film. I really want to use channel matting, blue or green channels. Remember to adjust the contrast between black and white with the color scale before.