The test of the testing

2021. augusztus 27.
Stiftung Warentest

Nowadays, everyone tests everything and as a result, you can find countless ratings on the internet. However, for the most part, the credibility of these results is highly questionable and can be seen as subjective opinions rather than tests. To gain an insight into credible testing and its more important details, we turned to Stiftung Warentest, one of the most prestigious research institutes. We spoke to Dr. Holger Brackemann, head of the organisation’s testing department, about the testing of technical products.

Which household appliances product groups have you tested? What was the most difficult category to test? Electric can opener, kettle perhaps?

In general, we test all household appliances as long as we think that they are useful for customers. Egg-cookers, for example, are not being tested, because we feel that they are not necessary to have. The most difficult tests are our lifespan tests, i.e. for washing machines, dishwashers or vacuum cleaners. They take a long time, they are expensive and not all of our testing institutes are capable of lifespan testing.

How long in total does it take to test an appliance from process definition to publication of the results?

The average time from process definition to publication is about six month. Because a lot of products like smartphones, cameras or tvs are quickly being replaced by new models, we have established databases with the results of several test results. Whenever a new product is introduced in the market, we buy it, test it and add the result to the database. This way our online users have access to up to date test results.

How is it determined which characteristics are tested during the testing?

For many products, we check the areas of functionality, usability, safety and environmental properties. The test requirements must, of course, be defined individually for each product type. Often the testing is performed according to national or international standards. For us, it is important that we examine all properties relevant to consumers. This applies in particular when conflicting objectives arise: For example, it is easier for a dishwasher to clean dishes at high temperatures. But this also results in high energy consumption. So both properties must be taken into account in the test.

How do you determine what rating is given to the test result for the characteristic being tested? E.g. If you are looking at whether a washing machine actually heats water to 60 degrees Celsius, how much better is a 1 °C difference than a 5 °C difference?

There is no simple and universal answer to this question. Basically, the project manager develops an evaluation function that takes two perspectives into account. First is the external benchmarks such as legal limits, requirements resulting from industry standards or the regulations of quality labels (eco-label), scientific or analytical limits, and quality requirements developed by experts with the foundation, as well as. The second part is the “relative” aspects such as the differences found in the test between the products examined. If laws or standards are not complied with, there is usually only one rating for this property: ” insufficient”.

Is the significance of the characteristics tested weighted? In a test result, does it make as much difference if the display is pixelated as if the drum does not rotate as it should?

Yes, every single test criterion is weighted individually and these weights are all different. We first weight the groups, such as function and usability, and then the individual criteria. The function is usually in the foreground, for example with a weighting of 50%. We usually give safety a low weighting. This surprises many people, so here is a brief explanation. We expect every product to be safe, as is also required by the European directives. We do not want to give a high weighting to a matter of course. Instead, we use downgrading effects that cause a bad verdict if safety is poor.

Does it matter what a device offers? For example, can a model be rated excellent if its performance is below average, but its existing features are perfect?

This question cannot be answered universally either. For many properties, we are guided by the product’s claim. For example, we expect only real vanilla to be used for flavouring in a vanilla ice cream. This is different with power consumption: even if an appliance states a high power consumption on the label, we always rate a low power consumption better than a high one. So it depends on the circumstances of the individual case.

Stiftung Warentest

“In many household appliances, the biggest advances are in reducing energy and water consumption. In contrast, we cannot see that the quality of tools has changed significantly in the last 10 years.”

Dr. Holger Brackemann, head of the Stiftung Warentest’s testing department

Is there a difference in quality, construction, durability between the machines of 10-15 years ago and today?

That’s a good question, and unfortunately we can’t answer it so easily. We are seeing significant progress in many high-tech areas, such as smartphones, televisions and computers. In many household appliances, the biggest advances are in reducing energy and water consumption. In contrast, we cannot see that the quality of tools has changed significantly in the last 10 years.

Are there models without defects?

Yes, there are.

When you buy for yourself, do you rely on your experience? Do you look at a product through different eyes than the average buyer?

Like any other consumer, I also buy spontaneously. But for important purchases I rely on the results of Stiftung Warentest, because I know that I can trust in them. Many of the products I use at home were tested before. That’s a good occasion to compare my own experiences with our test results. And from time to time I can derive proposals to develop our test protocols further.

What is your opinion about consumer tests and reviews? Especially in the light of the rise of the internet. Do they enhance the credibility of expert tests such as yours or, on the contrary, blur the boundaries between different tests?

In Germany a very high percentage of consumers (75 %) trust in our test results. However, in order to see our complete test result you have to buy it online or purchase one of our magazines. Other so called “consumer tests” either don’t test at all or it is not transparent how the “tests” have been conducted. But they are free of charge for the consumer and the websites get money from affiliate links. As a result we have to constantly communicate to the public, that we are the real testing institute, how much effort we put into testing and that we are the ones to rely on. 

Stiftung Warentest
Most buyers of any product always want to buy a good brand and a good machine. Based on your personal experience, is there such a thing? Alternatively, if someone asked you this about a product that you have tested, could you give an answer?

We are always very cautious about drawing conclusions from our test results for products that we have not tested directly. Of course, there are some brands that very often achieve a good test result. But that is no guarantee that all products of that brand are good. There are also always outliers. That also has something good: We don’t run out of work and we can always test new products.

Share This