What about RPGs?
The first two titles I'm working on are both in the strategy/simulation space, with a bit of nostalgia twist (hmmmm...).
But what about an RPG? After all, Spiderweb does well with them, and I love RPGs (Ultima, Morrowind, and Skyrim being all-time favorites).
Well... the first, most obvious challenge is that big 3D RPGs are [obviously] the products of huge development teams. Which begs questions of funding, business models, etc.
The other, less obvious challenge is "what unique thing can be done?" Here's a great article on some of the challenges with innovation in RPGs.
For a small studio to tackle the RPG space, there has to be some real innovations to make it interesting. I have some ideas, but it's not a first- or second-title challenge.