Well, my father still speaks poor English and has been living, legally and as a citizen, all his life. So if you met him on the street you would probably think he was illegal and you would be dead wrong.
I understand your frustration and anger has lead you to believe a certain way about some people. I've had plenty of caucasion kids knock on my door asking if they could mow my lawn. Are they guilty of taking jobs away from contractors as well? And they speak english!!!
There are plenty of legal residents who do not speak fluent english you know.