If you'd like to join the mystery quilt group to see how it goes, by all means, do so. That quilt is yours. If after the first real clue or two you feel that you can't afford the fabrics needed or that it isn't something you want to continue with, you can always leave the group and no harm done to anyone. I am in the mystery quilt group, but once we get instructions on what fabrics are needed I may drop out because I want to be able to use up some of my stash and not have to buy new fabrics. If i have to purchase fabrics then I'll just drop out.

Swaps are a bit different in that you are committing to doing something for another person so you really can't just leave the group.