So here's a really great question: Why do good people make bad games? Here's a really insipid article written based on that premise.
There are a LOT of reasons good people make bad games. More than it's reasonable to put in a single article. Still, there are really three big reasons this happens:
1.) Resource limitations
2.) Creative conflicts
Resource limitations are the most obvious culprit - the main ones being a lack of time, a lack of people, and a lack of money. In most games' development, some of these are known from the start. You can have a pitch with a budget and ship date before the content of the game has been locked down. On more than one previous occasion, I've known when the game was supposed to be done, the team size, and the overall budget before even knowing what the game was. It's crazy, but it happens more often than you'd expect.
This causes some pretty serious problems. If the ship date is inflexible, and your time is limited as a result, if you have to push for more resources, you can only push for more people and more money. But often, hiring more people requires more money (obviously) and worse, more time - it takes time for someone to "ramp up" on a project. So, what usually happens is that the resources don't change - the game design is "scoped" to fit into the available resources.
"Scoping" is a hard process - it's where you take the beautiful, detailed, well-integrated design that's full of awesome, revolutionary, genre-defying innovative genius, and you hack it to pieces so that it can be made with five guys, a box of socks and an old hamburger.
It's where you have to find the "core" of your game, and throw almost everything else away. If there are other features you absolutely must have, you find a cheap, fast way to do it, and hope it all holds together. Often it doesn't - we'll come back to that when we talk about innovation.
The second point - creative conflicts - is a tricky one. Game development is a creative process - a unique combination of art and engineering that requires tremendous collaboration between people with wildly different skillsets, working in a new medium where the end user's input can radically change the experience. Getting everyone on the team on the same page is fantastically difficult, and there are often huge creative differences between members of the team.
"Fun" is a really subjective thing. If someone thinks collecting 300 flags in Assassin's Creed is "fun," I think they're completely out of their mind. But I know people who have tried to do it, and who think that it brings genuine value to the game. In some sense, it's just a creative difference - a difference in what someone thinks is fun. There's a huge amount of that in game development. Is jumping "fun"? Is death "fun"? What *kinds* of death are fun? What aren't?
Worse, the longer you've been working on a game, the harder it is for the developer to really understand what's fun and what's not. If I've been working on an action game for two years, I have *no* idea what it's like to play that game for the first time. I take a lot of things for granted - the controls, the "intention" of a level, etc. I have a lot of information that the novice player won't. But because development is such an iterative process, and it takes so long before a game is really *playable* in a way that's polished and working, that it's often difficult to playtest with "new" eyes and hands on the project until relatively late in the development process.
That mentality's slowly changing - people are learning to playtest with less "fully-developed" games, but the simple fact of the matter is that the people making the decisions about what's "fun" or what goes in the game are people who are so deep into the game that they no longer even understand what it's like to play it. So, a combination of creative differences between the developers, and the inability to actually make good judgements often lead good people to make bad decisions. If there's a reason that you've seen something that appears to be totally unreasonable, that's probably the main reason.
Last, though, the reason that things are difficult is because games continue to be innovative. Trying new things is difficult. Making something fun is a hard process. Making something new fun is astonishingly challenging. It's an extremely iterative process, where you need to be able to try a number of approaches, scrap the ones that don't work, and polish it up until you can tell whether in its final form, it will be fun or not. Even when well-managed, this kind of iteration can't be guaranteed to result in a fun, new idea. When poorly-managed, this kind of iteration can spiral out of control, consuming huge amounts of resources and resulting in little that can be applied to a fun, new experience.
Thing is, while in game development the main development process is referred to as "production," it's nothing of the sort. In manufacturing, "production" is cranking out a million widgets. In game development, "production" is creating something entirely new. That's research & development. In R&D, budgets can't be set to the dollar. Timelines can't be set to the day. You can try to reach certain goals, you can make educated guesses, and you can work hard to meet targets - that's how you can keep the process focused and efficient - but the end result is never guaranteed.
You can't give someone six months to make something fun. Adding another week doesn't make a bad idea good - it takes good ideas to turn the bad one around - but finding that good idea is an unpredictable process. I've seen people work at an idea for years - literally - and not be able to breath that spark into it to give it life. Not for lack of trying, and not for lack of them being a quality person with good ideas. Sometimes it just doesn't come together how you thought it would.
Everyone who's worked on a bad game knows these things - the lack of resources, the creative conflicts and the iteration required to take something new and make it work. Hell, everyone who's worked on a good game knows these limitations, too, and how sometimes the difference between one and the other is almost intangible. Good, even great people can make terrible games - for me, I'm much happier seeing someone try for something new and fail than I am seeing a dozen cookie-cutter games all molded from the flavors of the day. Innovation is hard, sometimes it sucks, but it's the only thing that makes us better.