1) Biyur Halacha brings a machlokes regarding an ignoramus, and one should be machmir.
2) If the "not frum" Kohein is classified as a rasha, there is also a machlokes. However, today most not frum people are considered tinok sh'nishba and he would be obligated to honor a Kohein.
3) A mechalel Shabbos publicly is not necessary to honor even according to the lenient opinion in #2.
4) If the Kohein transgresses the Mitzvos specific to Kohanim, such as he marries a divorcee or enters a cemetery, one may not honor him at all.