You really should try to get over your fear of the subway.
The subway is by far the fastest, easiest and most efficient way to get around. Visitors from all over the world, including many who don't speak English, use the subway to get around. You have a leg up in that you speak English and you can obviously read. Everything is extremely well-marked.
The E train stops very close to your hotel at 53rd Street - entrance on Madison Avenue. It is extremely safe - very very secure.
The E train goes to the World Trade Center and terminates there. You can't go too far - you can't miss your stop.
When you go downstairs at the subway station by the hotel, you'll have two choices, there are Queens bound trains and there are Downtown trains. You do not want the track with the trains to Forest Hills and Jamaica Queens. You want the track with the downtown trains.
Both the E and the M train stop there but they're very clearly marked - they're marked in the front and on both sides of every carriage. It will say E train to World Trade Center. You simply take the E train to the last stop. Then you're there.
Say hypothetically you get on the wrong train - you boarded a Jamaica Queens bound E train. No need to panic. The next stop is Lexington Avenue - simply walk directly across the platform (10 feet) and board the train going in the other direction.
Say hypothetically you board a downtown M train instead of an E train - don't panic. Stay on the M train to West 4th Street and transfer to the E train upstairs.
There are uniformed police and other uniformed MTA (transit) workers in most subway stations - they can answer questions. Most random people can answer your questions too.
The subway is easy, it is fast and safe and I really think once you use it once, you'll be confident and will use it for the rest of your trip. It will save you a lot of time and a lot of money.