What is the recommended size of luggage when traveling Europe by train? Would a large back pack be easier than a small rolling suitcase?
Whatever "you" can carry on/off and load/unload is the best size for you. But if you do choose a "large" backpack or any backpack at all please don't wear it strapped to your back while walking around in the carriages. They inevitably end up banging into the other passengers and can be quite annoying.
A suitcase that can stand up in the aisle or lay flat on a shelf is the most versatile on trains, as you never know what luggage space will be available. And as small as possible for ease and versatility. A back pack is the worst choice - gets in people's way, easier to break into, harder to squeeze on to luggage racks.
Just having made quite a few trips by train in Switzerland and Italy -- take a medium size "rolling" suitcase that you can lift easily onto the train. We did not encounter any trains with low level entry between Martigny/Sion/Venice or Rome/Milan/Sion. We also used small back packs ("daypacks") instead of normal carry on shoulder luggage. So we had two hands free at all times - very useful.
Second - be careful to put your suitcase in a place where you can keep an eye on it. Don't leave a laptop on a table to go to the toilet. Trenitalia had a very good video screening on our train Rome/Milan showing all sorts of problems one could have with luggage (some of which I have to admit I had never thought about)
The most important thing for me while travelling by train is how easy the bag is to carry around, including to and from the platform and on and off the trains.
I have managed individual trips with rather big wheeled luggage, but it's been inconvenient and a struggle. For multiple trips I'd definitely want something that is very portable, quite small and light. I have found that a medium backpack or over the shoulder bag and a small rolling case works well for me. I have learnt that two smaller pieces are easier to stow than one big piece. As much as I'd love to have two hands free, I never seem to manage it!
i agree that it' important to keep an eye on your luggage. Theft is very uncommon, if not rare, but being sensible about taking precautions helps a lot. Needless to say, keep your valuables safe and with you at all times.