I removed the factory stereo to my 96 avalon but us stumped by the wiring in behind. It is nothing like the manual (or hans manual) says it should be. So I bought a toyota harness but that doesn't fit with the one that's currently there. What's the trick to the wiring?
As well, were would the amp be located?

Actually there is no correct wiring harness that I know of. Yes, there is a Toyota harness that's listed to work with the 1st generation Avalon (95 to 99, I believe), but your Avalon has a factory amplifier installed. As such, all of your wiring can't be accessed easily directly behind the head unit as can be done with other cars. When I replaced my stereo over a year ago, one of my good friends helped me "ring out the wires" to see which ones are power, signal, speakers, etc. Your average car installer will charge around $70-$90 to bypass the amp, and this fee is in addition to vehicle dash kits, as well as installation fees. If you still want to do it yourself, let me know and I can look at the wiring we did on my stereo and draw up a diagram. Just a note on the bright side once you've bypassed the amp, if you ever have to replace a stereo again it will be much simpler.