This is a long shot.
"The Golden Helix" by Theodore Sturgeon
The
possible matches are:
"ape-like creatures"
Humans hijacked to be prisoners on a very green planet, Viridis, exposed to the planet's radiation field, regress with each generation to earlier forms of primate.
"a strange temple (?)"
Their captors fly in a double helix formation as a salute and tribute to the humans they force-captured.
"...surgically-modified apes (I don't think they were humans) brought to the planet to be participant-observers"
Not a match, except that humans, devolving with each generation to apes, monkeys, lemurs, are definitely not observers, but forced, imprisoned participants.
"ape-like creatures lived in nests in trees"
At the very end of the story, almost fully devolved monkeys descended from the humans are hanging around in the trees, watching, without comprehension, the next race of aliens captured and brought to this planet.
"the story ends with mysteriousness and wonder"
From Sturgeon's point of view, it does. He waxes grandiloquent about the precious DNA being sampled from each race forcibly brought to the planet.
It's Sturgeon science fiction, written before the DNA double helix was discovered, he later said in a foreword.
It is also Sturgeon sentimentality and sappiness.
I am much more repulsed than Sturgeon intended readers to be.
