I think the problem is that most people believe in magic and for them, prayer is just a socially acceptable way of casting spells. In that case, if the things we pray for do not come to pass, the spell didn’t work, or, to say it as they do, the prayer wasn’t answered. In fact, there is no magic. All prayers are answered, it’s just that some are answered ‘no.’
God is omniscient, He knows what we want before we ask, He gives good things before we request them, just as Jesus fed the multitudes who had not asked for food. We do not pray to enlighten God, as if an omniscient God would bask in ignorance without our information, we pray so that we might discern God’s wisdom in the way He is provident. Without prayer, God’s providence is present, but our spiritual growth is stunted, because we cannot see His providence or conform ourselves to His will.
To give an example, a child asks his mother for a cookie and doesn’t get one. So his little friend says, “You request didn’t work.” That is how the world views prayer. What actually happens is the child asks for a cookie, and his mother says, “No.” If the child is insightful, he realizes that if he got the cookie, it would spoil his appetite for dinner, which he cannot see being prepared, and not having eaten a proper dinner, he would awaken hungry in the night. In that way, prayer is our spiritual eyesight. Through prayer, our discernment is perfected and our spiritual maturity is hastened.
God, like our mothers, sometimes says No out of love, when it impedes a greater providence that lies ahead.