Pamela, I saw you asking a question re Tea Olives on the chat session but you weren't there any longer.  Any how, I have the following Osmanthus in my gardens here: Osmanthus fragrans 3-Aurantiacus, Beni Kin Mokusei,  & Fudingzhu Osmanthus  heterophyllus 2-Goshiki, & Ogon. One of them is quite rare.  But, you asked about falling leaves. They are evergreen, so shed leaves typically in the spring and the fall. Your plant picture looks like it might be overly dry where you have it planted - so if it's losing more leaves then normal it might be distressed and in need of moisture. I have mine in flat beds, sloped beds, mostly sun, partly shady and even one in a fair amount of shade and the all do well. I'm watering 2 times a week now and as it gets hotter and more humid I will be watering 3 times a week. We probably don't live too far apart - I'm on Lake Oconee Georgia. Hope this helps.