It was all in Marshack spiel to the son during his Bar Mitzvah. "Be a good boy." Larry kept saying throughout the film re: his misfortune, "But I didn't do anything!" It's not enough to just "not do anything" wrongg, you have to actively do good.

Larry lead a pretty damn blessed life (as illustrated by the scene with his brother pool side by the Jolly Roger), but all he really seemed interested in was making his life better for himself (getting tenure, making sure his neighbor didn't build on his property, etc.) Though he wasn't actively doing bad, he wasn't "good" either.