Have you ever seen any of the first series with Ian Hendry?
As a child, I was sometimes allowed to stay up to watch The Avengers but I never saw any of the early episodes until much later (apparently only 3 of the first series survive now, so not sure how many I saw, but they were shown again.) They are quite different in character; no outside location shots, not so much martial arts action, Ian Hendry was the main character. I found them quite dull. So, I thought something must have changed quite radically after the first series when it went on to become a '60's icon. The first series also explain the series name, which I could never understand. Hendry and Macnee were initially "avenging" the death of Hendry's fiancée.
Between the first series and the Diana Rigg era (which would be when I began watching) the scripts did become more and more, weird and wacky and yes, I would consider many of those later episodes science fiction. However, even the first series did have Hendry and Macnee investigating a Nazi war criminal frozen in a cryogenics experiment, so maybe science fiction was there from the very start. It was also the only TV programme I remember with women in a dominant role - Honor Blackman, Diana Rigg, Linda Thorson.
The New Avengers also had an episode with "Boys from Brazil" cryogenic Nazis and many of it's plots were recycled. I remember that series much better, but it couldn't capture the spark of the original.