I wonder how much his beliefs were influence by his own religious tradition. I'm not sure he was a Reform Jew, but his beliefs seem similar to what they hold to. Reform Judaism has probably the weakest sense of a personal God who chooses to relate to people, whereas Orthodox and Conservative Judaism are stronger in this regard.
I'm not sure, however, how far physics, math, and chemistry can reveal a God who, indeed, reveals Himself to man. The order of the universe CAN reveal the existence of a designer, but not much beyond that. We call this "general revelation," a form of God's self-revelation that is available to every person.
It is the job of religion and theology, however, to get at who and what exactly this designer is. Some of God's qualities (his love, justice, holiness, etc.) are not available to be observed in nature. They must be given to us in scripture -- there's no other way to know about them.
So ... I would be interested to know where, exactly, Einstein got these ideas about a distant, impersonal god.