You aren't. It really is that simple. I don't care what your age is, as an experienced college-level educator I can assure you that you are not too old, because there are many nontraditional students on college campuses today (and likely some of those are older than you are), your life experience gives you many advantages over more traditional students, and with the growing nontraditional population many colleges have programs and services especially tailored for the nontraditional student.
I went back to school in my 30s and today I teach college. Yes, I have many traditional students in my classroom but every semester I have a large percentage of nontraditional students as well. I have studens in their late 20s as well as 30s, 40s, 50s, and up. I have students who have retired from one career and are looking to move into another. I have students whose children (or grandchildren) have left the nest so they are looking to enter a new stage in their life.
I also have many students balancing school with work and family. I have students who are the traditional age but are in nontraditional circumstances including children and family, work and military service, as well as sports and other activities. You are a unique person, but your situation is not as unique as you might think.
In many ways, your age, or rather your life experience, will be a tremendous asset for your return to school. Nontraditional students understand much better than traditional students how to manage their time and prioritize tasks. In addition, nontraditional students are often much more motivated and goal-oriented than their more traditional counterparts. Finally, your life experience also gives you a great deal of knowledge and experience to fall back on or pull from when it comes to understanding, applying, or adapting the new knowledge you gain through college. I regularly see my nontraditional students outperform traditional students in many ways, but it ultimately comes down to a maturity of thinking and reasoning that can only come with growing up. I know when I returned to college as a student after working for a number of years that I did much better in the classroom and also handled my work load much better than I did when I was a more traditional student.
Today colleges recognize they have a changing student population and offer classes in a variety of formats including on campus, off campus, televised, and internet as well as a range of schedules including days, nights, weekends, and accelerated. In addition, there are now support services available for students who fall outside the traditional student role. Many financial aid and scholarship programs also exist specifically for the nontraditional student.
In the end, it really comes down to your own gut feeling. Do you really want that degree? Are you ready to change your life? You can do it and there will be help and support available for you. Don't use your age as an excuse not to pursue your dream.