How you would reverse engineer a certain IC depends on what is known about it. Relatively much is and was known about the NES CPU from the get-go. The CPU core is a second-sourced version of MOS Technologies' (a.k.a. Commodore's) 6502 core, with the BCD portion blocked out to prevent a patent intrusion, so you get that for free. The custom parts are the video and audio. These were described in programmers manuals, which may have been leaked. One big advantage here is that you can run your own code on the CPU, so you can hypothesize about, and test corner-cases about the various IO functionality. You can also use the game library for testing. If you want to get really serious about it, you can input all digital IO pins into to your own FPGA and xor it with the internal representation of the same pins and trigger on any difference.
Let's say you don't have this kind of prior knowledge, though. Then your first path of attack will probably be to reverse engineer the peripheral circuitry. Sometimes this can be done by logic induction, for example if the chip under reverser engineering is connected to a memory chip or other chip with a known pinout. Then you may for example hook up a logic analyzer to it to try to make sense of the signals. Then you work from there. One interesting example of this, though not made specifically for cloning the ting, is Natashenka's reverse engineering of tamagotchis, which turned out to use a 6502 clone as well, Sunplus branded.
If all else fails, you resort to nitric acid, a scanning electron microscope and and probing needles. This is what Chris Tarnovsky is doing. Again, the focus here is not to replicate the function of a chip, but it tells you a lot about the reverse engineering process. Another option at this stage is to image the chip layer by layer and simulate it to figure out something about it, or perhaps resynthesize the internal circuit for an FPGA.