I agree that not all recording engineers know high fidelity when it comes to capturing that "You are there" sound, but it is complex to get. A lot of it has to do with the engineer’s ability to correctly choose miking techniques and also consider room acoustics carefully. All of my recording experience comes from the days of analogue and tape, there was no such thing as digital anything except for digital delay units. So I can't comment on the digital recording domain.
Just for reference, the equipment at most record label level studios is of very high quality and top notch gear. The studio I worked at in the late 70’s had 16 and 24track 2" tape machines which at that time cost $80K - $90K each, the recording console $150K, 2 channel master tape machines at $30 - 40K each, microphones each with their own sound from a few hundred dollars each to over $2K each. Then there’s outboard equipment like huge overhead monitors, huge amplifiers, compressors, limiters, noise gates, delay units, reverb plates...big bucks, and this was a middle of the road studio!
One reason for miking a kick drum inside the drum itself is to isolate the kick from the rest of the kit so the engineer can EQ it to the producers liking. Miking this way is most likely to get only the kick drum as the prominent sound. This also goes for the rest of the kit where any number of mics can be used. I've seen sessions where 7 mics were used on one kit. Most mics, except for the overheads are placed within an inch of the drum head...again to isolate each drum as much as possible and allow the engineer to pan (select the balance left to right or location of the drum in the recording) to their (or the producer's) liking. Personally I like and prefer distant miking.
Another aspect of close type of miking is that it eliminates the affect or contribution of the room acoustics to the sound of the kit. Now keep in mind this can and is also applied to a lot of other instruments as well, such as close miking a guitar amp. And let’s not forget about direct boxes, where you take the signal right off of the electronic instruments output and feed it directly to the console.
There are a lot of techniques being used as well as complex binaural far field miking. Far field miking captures more of the room acoustics along with the original instrument. You see more of this with acoustic instruments than you do with electronic and are gaining use in live recordings, particularly orchestral type settings. If a recording engineer were to use far field miking in a recording studio, it would require a large room with very good acoustics. Either way what the engineer wants to do is record the original signal as dry and flat as possible to the multitrack tape. These are original tracks. Think of it like this, if you record a signal with a lot of EQ, too much compression/limiting and processing, and it turns out that the next day the track sounds lousy...it becomes very difficult to undo what you already recorded on tape without introducing other problems. After the signal is recorded to the multitrack, then it can and usually is EQ'd and processed (sometimes to death) when it is mixed down to two channel master tape. Very often two channel master tapes are also compressed or limited one more time to keep the signal within the signal to noise of the tape. This really kills a lot of the dynamics of the original recording. There's one more hurdle, when the master tape gets to the record lathe, it is limited once again due to the record limitations and the RIAA is applied to the cutter head electronics and encoded into the record grooves. Yeah there's a ton of processing going on by this time...
A recorded signal should hit a tape at a certain level to avoid two things, too low a signal will increase the noise floor of the tape itself, and too loud a signal will saturate a tape or cause a distorted signal. Just because a level meter is in the red does not always mean it is distorted. Some tapes can handle a hotter signal and others cannot. +3 and +10 dBm seem to hit my memory. When I used to set up the noise reduction and console levels, it always was dependant on the tape and the engineer’s preference. Recording a performance that has a lot of dynamic range is difficult. That's why DBx and Dolby noise reduction is necessary...because of the tape limitations to handle both very quite and very loud signals. This in combination with room acoustics and or close miking as apposed to distant miking challenges the engineers ability to use good techniques.
This is a complex issue and very hard to understand in a few paragraphs. But like most of you, I am also still out for the reason why certain recorded material sounds so unlike the real thing. Most of the pop and rock stuff is highly compressed from the start. I do believe in very good playback equipment to allow the signal to be reproduced as well as we can afford to. However we have to consider our own room’s acoustics also. After all we do read about of some listening at near distance to avoid room interaction acoustics or others who develop very well dampened rooms to minimize the room’s contribution to the overall sound. On a personal level, and this was mentioned in someone’s in home piano playing, I also believe that a lot of power is needed to reproduce all the sound pressure that an acoustic instrument can produce live. Example, my 11 year old daughter practices her clarinet and although she is young I always listen to the volume and harmonics it can produce and am always amazed at how much richer it sounds than when I play a prerecorded CD. I feel it in my chest when that little girls lungs blurt out a note! Is it my system, probably...it can stand a few upgrades yet. But I've heard good systems too and even some of them can't even reproduce the sound my 11 year old gets out of that wind instrument!
So yes a lot of it has to do with the complex nature and decisions the recording engineer, recording studio acoustics, and also with the definitive say-so the Producer decide sounds best for the record company and what will sound best on the radio. Most of the time recordings are not made to satisfy audiophiles...that takes to much time to do. I can and do appreciate a well recorded disc, whether tape, vinyl, or disc. There's so much to this.