It Has Been Done
We all like to think and believe that when we come up with an idea, it is an original idea. Chances are however that idea is most likely not as original as you think or believe. In most situations given similar environments and conditions, other people will come to the same conclusions. Games are no exception to this rule. That doesn't mean you shouldn't try to create something unique and new, but don’t be shocked if a similar design exists. Game Designers in a way are like storytellers, although we are telling the same story we tell them in different ways from different perspectives.
That doesn't mean something new can't be created. Every once and awhile a game designer does come up with a unique mechanic. Gravwell from Renegade Games is an example of something that is unique (at least I haven't seen it used but I haven't played every game in existence yet). It only utilizes 26 cards and the alphabet to determine player order and resolution, everything else is based on the position of the ships around your own. Depending on the card played, it determines how many places you move based on the closest ship to you at the time. You can move X spaces in the direction of the nearest ship, Y spaces away from the nearest ship or you don’t move but everyone else moves Z spaces towards you. Players pick the card to play in secret, then reveal at once and they resolve based on the card letter. Keep in mind there are only 26 cards, the card you picked might have resolved one way but because someone went before you, it now can affect how your card ultimately resolves. Great game, simple, fun and fairly quick to play.
It is ok, if your idea isn't unique so don't try to force something just to make it different just because you want something different. If it feels awkward or clunky, don't keep using it because just to be different, it should feel natural. In the end no matter what you design or how unique you think it may be, it will always end up being compared to something else. If you make a deck building game, it will be compared to other deck building games even if there are a lot of differences. That isn't necessarily a bad or a good thing but it is something you should prepare yourself for.
Almost every board game at the core can be broken down based by their mechanics into similar categories. The most common ones known are worker placement, co-operative play, area/territory control, dice rolling, etc. You can find more information about the various mechanics and games that use them here and here. A game could have multiple mechanics associated with it so it doesn't just fit into one category, but don't make it too complex. Remember K.I.S.S. (Keep It Simple, Stupid).
Just like board games, miniatures gaming all share commonalities that can break them down their mechanics into IGO.UGO, alternating turns, action/reaction games, and hidden movement to name a few. The heart, the base core of each of these games are all the same. They all contain miniatures, that we play on a table with terrain and utilize some device to measure distances like a tape measure or ruler. We create armies or have unit lists based on stat cards and/or profile sheets. Each player takes turns activating, moving their units. We then roll dice to determine successes of those actions. Some games have variations but still the same mechanics like using cards to determine success, instead of dice rolling. Movement/Actions may vary from IGU.UGO, to alternating turns or alternating model activation. The results are all the same. There is even grid or zone type of mechanic as well.
Why doesn't someone simply take Game A, replace the miniatures and have instant success? It is a simple and yet complicated process. If you look at the majority of games, a good portion of them simply just do this. However the rules aren't what will completely make a game successful. There are also minor changes that need to change in terms of terminology because of trademarks. Themeing a game properly can make all the difference. All game rules have their pros and cons. Great games aren't just determine by good rules or good miniatures or great lore. Good games are a combination of them all, married together and built with a good community. There is no one rules system that is better than everything, it is just that there are rule systems that are better depending on players preferences.
Copyrights, Trademarks, Patents
I won't go into a lot of detail talking about Copyright, Trademarks and Patents because it isn't a simple issue even for lawyers. For the most part you don't have to worry about a patent, nor should you really worry about copyrights or trademarks. There is one thing you should check on trademarks, make sure a term isn't used. A lot of games use generic definitions and terms, you can't trademark or copyright many common definitions and terms like strength, agility, action, reaction, etc. Some terms are trademarked and you should make sure you aren't in violation or it can cause problems elsewhere. One example is when Upper Deck started to re-release Vs calling it a Living Card Game. That term is trademarked by Fantasy Flight Games. Although many types of games are similar to what a LCG is, they hold the trademark so avoid using that in your title.
I'll summarize Copyrights by quoting Lisa Steenson from Gut Bustin' Games. She said during a panel at a convention, "Although it can be difficult to protect a game design, most publishers are not in the business of stealing designs - their reputation is on the line. If you are self-publishing, consider registering the copyright on your rules and game components. You have copyright on your intellectual property upon the moment of creation, but registering that copyright allows you greater protection in a court of law in cases of infringement. Typically registration requires 1 copy of the game to be sent to the Library of Congress if published, along with an application fee of about $45. This has inherent advantages and does not cost an unreasonable amount - just the fee and the cost of sending off a single game. For more information, read about copyrighting games in the US on the official government copyright office website. Don't bother with seeking a patent. The short answer is that game designs cannot be patented and it is a very expensive and complex process."
When you are creating a game, don't be afraid to share the mechanics of how a game works because you are afraid someone will steal your idea. There are many times I’ve seen people try to ask what you think about a game, but are being really secretive on how it is played because of this fear. Overall it will hurt how people view it if they don’t know more about it. It is better to know ahead of time what people think, before investing a lot of time and effort into what you may be great but others think is a bad idea. There are quite a few game ideas that I think are going to be great, then after sitting down and going through the motions I’ve found that they didn’t seem interesting or fun to others.
The game industry itself is a small industry, a multi-million dollar market but overall the players involved are well known. When attending conventions like GenCon or Origins many publishers and designers meetup, get together and talk. One example was when someone approached one publisher to publish a game. When looking over it, it was discussed with another publisher on what they thought and discovered that it was also being pitched to them as well. Not only was it being pitched, they had asked them to not show it to other publishers as they were considering a legitimate offer. Although not completely common, both parties did agree to it, so the designer was violating the agreement. In the end neither party picked it up. Often over meals or after hours game sessions, multiple publishers get together to play and talk. It really is a small world, at least with board games.