As Hnrutt says, you're quite likely going to be using peoples names or just inferring "you" from the context.
But I wouldn't exactly say avoid them in spoken Japanese.
"Anata" isn't really going to be considered rude, even if it's hardly heard in conversation. "Anata" is probably your best bet if you just want to sound polite, and don't know the person's name. It's the standard "you" and it's used by both males and females.
"Kimi" on the other hand is used mainly when addressing equal partners and subordinates, often suggesting familiarity. So you're probably not gonna address your boss with this one. When it's used in songs/anime/manga/dramas it's often used to address only a female, though this is not the case in real life.
"Omae" used to address guys. Used only by guys. Very informal.
"Temee/Kisama". Another one of those wonderful "yous" that's throughout anime etc. Very rude. Don't use it.
There are plenty others, but those are probably the main ones. Again, people tend to either imply the "you" from context or use the person's name directly, but there's no real problem with using the Japanese pronouns for "you". Just be sure of your status relationship with the person, so you don't end up calling for your employer as "きさま！”
In fact, I might just put the "I's" in here to to avoid potential queries.
Watakushi: Super formal. Business keigo.
Watashi: Polite. Used by men, but moreso by women.
Atashi: casual for females. It's kind of cutesy; the "wa" changes to "a".
Boku: Plain form of "I". Used in familiar company.
Ore: Cool, rugged way to say "I" for males. Very informal.
If you want to see all the possible ways to say "I" and "You", you could check out