Speedskater
Well-known member
The linked article is totally lacking in methodology detail.
The Merits of 'DBT'
- Thread starter twich54
- Start date
Help Support MartinLogan Audio Owners Forum:
asindc
Well-known member
Speedskater
Well-known member
"Hi Fi Plus" magazine issue #34 has a 2004 date and I don't see this test listed om their web page. I believe that the messenger brought up the subject of methodology.
asindc
Well-known member
From the article to which the supplied link directs you:
"How the test was done…The problem of course, is in devising an appropriate test
or set of tests. Way back in Issue 1 we carried out a similar exercise with budget leads – with notable success.
As a result, I opted to use a development of the same
format, employing the same checks and balances.
The basic set-up involves a blind, panel test,
subsequently repeated with the same test products but a
different system, thus allowing us to compensate for and
also investigate the issue of system matching. Sounds
simple but it’s not without its complications. The test
group consisted of five
expensive cable sets as
well as a budget
combination, the
intention being to
assess the audibility
of differences
between cables, as
well as whether or not
the expensive cables outperformed
the basic pairing.
Secondly, there was a spread of
price and approach, with different
materials and constructions in an effort
both to maximize the sonic differences and see
to what extent the cost/performance equation held
true. The tests themselves involved a panel of three
listeners who sat en bloc. It would have been better to
conduct separate listening but it would have been
unwieldy to say the least. The sessions took all day as
it was! Instead we simply banned any discussion during
the testing in an attempt to prevent undue influence.
For each test series, the panel were first played one of
the expensive cables, without its identity being revealed, to
set a level of expectation from the system and act as a
control. The six samples were then played, each one with
three different pieces of music, the control set appearing for
a second time. The first of the six was marked as a 10, each
following sample being marked up or down in comparison.
This differential marking system gets over the issue of
establishing a scale for what is, after all, a completely
arbitrary judgement, indicating as a primary response preference as opposed to rating. In terms of relative
performance, it’s a far more meaningful gauge. On the
second series we adopted the same approach but a
different running order and choice of initial control as
well as the different system.
In both cases we went to great lengths to ensure that
the identity of the cables was concealed, and whilst the
drapes between the speakers and in front of the system
would have had their own influence on the
sound, they’re an unavoidable evil if blind
testing is to be achieved.
The scores were entered on preprinted
response sheets, along with
comments regarding the listener’s
response to each cable. These written
notes deliver a valuable expansion to
the simple numerical scores, often
helping to explain apparent anomalies."
Mods, please feel feel to edit if directly quoting from an article is against the rules.
Gordon Gray
Well-known member
As you can tell from my previous posts on this topic, I wouldn't base any audio decision on DBT. These type tests remind me of judging a movie by its promo trailer versus watching the entire film.
Having said that, it was an interesting read.
I found it suspicious that all speakers were behind drapes. Gee, I wonder how that impacted the test results. The brighter cable would seem to have an advantage if that cable is pumped at the top end. The converse is true for a mellower sounding cable, which would likely be over damped in this arrangement.
GG
Having said that, it was an interesting read.
I found it suspicious that all speakers were behind drapes. Gee, I wonder how that impacted the test results. The brighter cable would seem to have an advantage if that cable is pumped at the top end. The converse is true for a mellower sounding cable, which would likely be over damped in this arrangement.
GG
asindc
Well-known member
The 'behind the drapes' thing also threw me off, which is the main reason I would not employ this test to make a decision. I suppose they could have blindfolded the judges, but then they would not have been able to take notes. Aside from setting up six identical but separate rooms, I don't know what they could have done to provide a more realistic environment without revealing the test cables to the judges.
Gordon Gray
Well-known member
Frankly AS, the drapes seem to be a fundamental / significant flaw in the apparent results.
Why would one run a test using this compromised environment?
GG
Why would one run a test using this compromised environment?
GG
Gordon, I didn't read it that way. It says: ". . . and whilst the drapes between the speakers and in front of the system would have had their own influence on the sound, they’re an unavoidable evil if blind testing is to be achieved."
That sounds to me like the drapes were placed between the speakers to conceal the system components, but not in front of the speakers themselves. In my opinion, this would probably have a beneficial effect more than anything as it prevents some sound reflections off the system itself from reaching the listeners. But I don't expect it would really dampen the sound coming out of the speakers themselves. I can't imagine it would have had that much of a detrimental effect on the results.
That sounds to me like the drapes were placed between the speakers to conceal the system components, but not in front of the speakers themselves. In my opinion, this would probably have a beneficial effect more than anything as it prevents some sound reflections off the system itself from reaching the listeners. But I don't expect it would really dampen the sound coming out of the speakers themselves. I can't imagine it would have had that much of a detrimental effect on the results.
Gordon Gray
Well-known member
Rich,
Sorry if I misinterpreted the text. If that is the case, there would likely be no substantial negative effects.
Gordon
Sorry if I misinterpreted the text. If that is the case, there would likely be no substantial negative effects.
Gordon
asindc
Well-known member
I think the drapes still matter in the sense that putting anything between the speakers could 'possibly' have an effect on the sound, whether that be positive or negative (can't imagine that the effect would be positive). More than anything else, the drapes would not be part of a 'real-world' listening environment, so in that respect they detract in some way from the ultimate goal of the test, IMO.
shakham
Well-known member
I'm in favor of DBT...or any type of testing that removes the bias of price and brand out of the equation.
I also understand that the test may need to be longer than a 5 minute clip of the same song. I also understand it may require the equipment to be switched back and forth multiple times to get a true comprehension of which sounds better. I also understand that the equipment needs to be as equally setup as possible.
Look....
If you are buying it for yourself, then you can test the equipment however you want. If DBT doesn't float your boat, aka you want other variables than the sound itself, that is your choice.
However...
If you are testing with the purpose of helping others in their purchase decision, then it has to be about the sound. To be honest, the differences between many of these audiophile level equipment is so subtle that you need to take away other variables/distractions/biases in order to get a fair comparison. If not you'll always hear more things from your favorite brand.
The drapes issue...
At the end of the day, no equipment is getting an edge over the other.
Maybe I'm not audiophiley enough, but if you have to look that hard for an improvement, then honestly, one should be willing to say two pieces of equipment are almost the same.
Blasphemy..yes..maybe....
I also understand that the test may need to be longer than a 5 minute clip of the same song. I also understand it may require the equipment to be switched back and forth multiple times to get a true comprehension of which sounds better. I also understand that the equipment needs to be as equally setup as possible.
Look....
If you are buying it for yourself, then you can test the equipment however you want. If DBT doesn't float your boat, aka you want other variables than the sound itself, that is your choice.
However...
If you are testing with the purpose of helping others in their purchase decision, then it has to be about the sound. To be honest, the differences between many of these audiophile level equipment is so subtle that you need to take away other variables/distractions/biases in order to get a fair comparison. If not you'll always hear more things from your favorite brand.
The drapes issue...
At the end of the day, no equipment is getting an edge over the other.
Maybe I'm not audiophiley enough, but if you have to look that hard for an improvement, then honestly, one should be willing to say two pieces of equipment are almost the same.
Blasphemy..yes..maybe....
Gordon Gray
Well-known member
Hi Shak,
Appreciate your insight but I have to disagree.
Regarding third paragraph, If DTB doesn't float your boat aka you want other variables than the sound itself, that is your choice., you infer that the only way to make a decision based on sound is DTB and that long term listening, absent DTB, introduces biases other than sound is, from my thirty odd years in this hobby, just silly and unsupportable. There is no way one can pick up on all the subtleties that a piece of hardware (cables or otherwise) reproduces in a snap shot of time.
Drapes, or anything else that dampens the sound, will likely favor an over bright cable and cause a warmer cable to be over-damped.
Last paragraph. Once your system reaches a certain level of transparency (no insult intended), the essence of hi end audio, and your personal preferences is all about minor / subtle differences.
GG
Appreciate your insight but I have to disagree.
Regarding third paragraph, If DTB doesn't float your boat aka you want other variables than the sound itself, that is your choice., you infer that the only way to make a decision based on sound is DTB and that long term listening, absent DTB, introduces biases other than sound is, from my thirty odd years in this hobby, just silly and unsupportable. There is no way one can pick up on all the subtleties that a piece of hardware (cables or otherwise) reproduces in a snap shot of time.
Drapes, or anything else that dampens the sound, will likely favor an over bright cable and cause a warmer cable to be over-damped.
Last paragraph. Once your system reaches a certain level of transparency (no insult intended), the essence of hi end audio, and your personal preferences is all about minor / subtle differences.
GG
Hocky
Well-known member
I agree that long term listening you will notice things that you miss otherwise, but long term listening is a double edged sword. Before too long, what you're hearing becomes "right" because it is what you're used to hearing. In that sense, you may be able to get a better feel for 2 systems if you're unfamiliar with both and are doing A/B. And in doing so, you're probably subconciously comparing them to what you hear as "right," aka what you most commonly listen to. A great example is a friend of mine that always runs his $200 DIY sub way overcooked in his combo - it sounds terrible to me, but right to him. When he comes and listens to my system, even with my now $6000 sub, he tells me that my sound is too bass shy and thin. lol
All methods will have their inherent plusses and minuses, but it is my opinion that if you can't consistently pick out the "better" option while blind folded (even if this means hours of listening with potentially lots of source material per A/B/x), neither piece of equipment is "better."
Last edited:
Gordon Gray
Well-known member
Hocky,
Let me give you one example.
I auditioned a VTL 150w per channel tube amp for one week. Listening to certain material, I loved the sound. Other material, not at all. I never got used to and was not able to get comfortable with its sound. It never sounded "right" within the context of a broad range of musical material.
Needless to say, I returned to product and ended up with my Pass Labs.
If people like using DBT and use it as a basis for purchase, have at it. I personally think this type of test has limited value at best.
Gordon
Let me give you one example.
I auditioned a VTL 150w per channel tube amp for one week. Listening to certain material, I loved the sound. Other material, not at all. I never got used to and was not able to get comfortable with its sound. It never sounded "right" within the context of a broad range of musical material.
Needless to say, I returned to product and ended up with my Pass Labs.
If people like using DBT and use it as a basis for purchase, have at it. I personally think this type of test has limited value at best.
Gordon
Last edited:
Hocky
Well-known member
That is true and definitely a fair statement, things could be missed in such a brief test. I also think, though, that if someone tests appropriately with the types of material that they will be listening to, a fair amount of those cases could be avoided. But no, a few 5 minute samples won't do it.
Audiophiliac
Well-known member
I have always had mixed feelings about these listening "tests"...DBT, ABX...whatever you want to call them.
It seems too simple in my mind. If you are looking to buy a new audio component, simply listen to it in your system. If you think it sounds better than what you currently own, buy the new thing already! Why should it matter if your mind, somehow, got tricked into thinking it sounds better? As long as that trickery continues, is this not the epitome of blissful ignorance?
I do not consider myself an "audiophile" by the definition implied by the hifi magazines, etc. I simply enjoy listening to music. I have heard thousands of systems. I form opinions on all of them while listening to them. Some are good, some are bad, all are different. There are so many different opinions out there on what sounds good and what does not. The only one that matters to me is my own. If I bought a $10000 pair of speakers because I think they sounded better than my $2000 speakers, who is to tell me that I made a mistake because I did not to a DBT or an ABX or a WWJD test? And if I later found a pair of $1000 speakers that I thought sounded better than my $10000 speakers, sold the old ones for $8000, took the extra $7k and went to the Monaco Grand Prix for vacation, who is to tell me I made a mistake either the first time or the second time I switched speakers? The fact is that the sounds of F1 at Monte Carlo would sound better than any pair of speakers no matter how much they cost.
It seems too simple in my mind. If you are looking to buy a new audio component, simply listen to it in your system. If you think it sounds better than what you currently own, buy the new thing already! Why should it matter if your mind, somehow, got tricked into thinking it sounds better? As long as that trickery continues, is this not the epitome of blissful ignorance?
I do not consider myself an "audiophile" by the definition implied by the hifi magazines, etc. I simply enjoy listening to music. I have heard thousands of systems. I form opinions on all of them while listening to them. Some are good, some are bad, all are different. There are so many different opinions out there on what sounds good and what does not. The only one that matters to me is my own. If I bought a $10000 pair of speakers because I think they sounded better than my $2000 speakers, who is to tell me that I made a mistake because I did not to a DBT or an ABX or a WWJD test? And if I later found a pair of $1000 speakers that I thought sounded better than my $10000 speakers, sold the old ones for $8000, took the extra $7k and went to the Monaco Grand Prix for vacation, who is to tell me I made a mistake either the first time or the second time I switched speakers? The fact is that the sounds of F1 at Monte Carlo would sound better than any pair of speakers no matter how much they cost.
Gordon Gray
Well-known member
Well stated.
Many ways of getting to the end of the journey.
I guess the lesson from this thread is there are many divergent ways to accomplish the same end result.
That is, of course, connecting with the music.
GG
Many ways of getting to the end of the journey.
I guess the lesson from this thread is there are many divergent ways to accomplish the same end result.
That is, of course, connecting with the music.
GG
slowGEEZR
Well-known member
I do not use DBT to assess gear, as I think it is a flawed concept. Does any piece of music sound the same to you every time you listen to it? I think Martin Colloms said it best with his Objective Subjective Review Debate column way back in 1991.
http://www.hificritic.com/downloads/Archive_A1.pdf
http://www.hificritic.com/downloads/Archive_A1.pdf
