Feta cheese is a staple for many Greek food recipes, so you'll need it to be as authentic as possible. Good feta is made from goat and/or sheep milk, is best in whole large blocks, and is unsalted. Buy it in large quantities, and store it wrapped in paper, in brine, or in olive oil.
A simple way to pit olives for your Greek food is to smack them with the flat of a large knife; the meat will then peel off the pit easily. If you're planning to use a large number of olives, roll over them a few times with a rolling pin. This will loosen the meat from the pit as well.