You don't invite people you don't love to your house, right? You don't like them to come close to you, and you don't engage with them unless it is necessary.
Well, the Almighty does not draw those whom He does not love close to Him. Each time you pray is the very proof of His love for you.
Cherish this, enjoy this and be grateful. Show Him that your love is mutual by knocking frequently on His door, not only five times a day.