No, they don't have a museum, it's true ... but HPL is not yet considered a major literary figure -- look how long it took to get a museum established in Poe's honor! However, Brown University's John Hay Library has a marvelous collection of Lovecraftiana, both his own work (original magazine publications, later editions, manuscripts, letters, etc.) and by others with Lovecraftian associations, such as Clark Ashton Smith, August Derleth, E. Hoffmann Price, J. F. Morton, W. Paul Cook, etc. And there are various sites now designated as historical sites connected with him; and a monument was set up in honor of him, as well to celebrate his centennial ... so it's getting there.
Considering Lovecraft's sepciality was the incorporeal and the undescribable, monuments of brick and mortar are probably a bad idea, and something the reclusive gent would have been uncomfortable with.
Always wonder how the Old Gent would feel if he were here now to see the influence he has had on books, comics, games, movies and so much more. However, like GOLLUM, I'm seriously planning on visiting Lovecraft's haunting grounds.
I actually live right outside of Providence. Although there is no actual museum, you can walk the streets, see his house and the other locations. I've never actually done it myself, but have a friend who is a maniac about HPL and has done a number of walking tours.
Also, if you were to be in the area and rent a car, you can extend a trip around New England and visit a number of other sites that he references within Massachusetts and New Hampshire.