Five Worlds Trilogy by Al Sarrantonio. Book 1, Exile, the only one I read, and I didn't really like it all that much. It's funny, because I was looking for it on goodreads so I could rate it, and I couldn't remember the name and searched for it, and came across this thread. My husband remembered the title, and from that I was able to search the author. It's probably far too late, and you're not coming back to check on it, but now others who remember what I did from it will find this thread and know.