It is notorious for being almost 40 minutes long and yet only containing less than 30 minutes of "unique" music. Basically, the title track (very much a composition bTW and not a "jam") which starts and ends side 2 is exactly the same take. And in "shh/peaceful", there's a long stretch of music that is repeated towards the end to give it a semblance of form.

Also, regarding the thing about the musicians just playing what they want, that's not entirely true because the record clearly has a "vibe" that the musicians all adhere to and there are basslines and riffs that repeats a lot for the soloists and keyboards to play over. It ultimately comes down to new methods of organizing sounds in the context of improvisation that aren't the old "chord progression in a 32/12/whatever form" but not formless either.

It's like in the late 50's, you had the "modal" style breaking through with soloists using a mode or a key or whatever over a static harmony in terms of composition rather than a preset progression as a basis for improvisation. And Ornette Coleman was VERY important in that era as well because he made songs where the function of the theme was NOT to set up a chord-progression or even a mode for the soloists to adhere to but instead a "vibe" or "temperament". Like, if the theme is hectic with a lot fast runs and shit, the soloists follow that or if it's relaxed and bouncy, they follow that and so on. The theme as a catalyst for the general rhythmic and intervallic nature of the solos in other words (note that Don Cherry had a tendency to switch it up a bit at times as evidenced by the "This is our music"-album but Ornette played like that).

By the time "Ina silent way" came out, musicians had been experimenting with new ways of organizing sounds and new forms for a decade and those guys were all modernists well-versed in both traditional styles as well as the most cutting-edge shit.

IMO, the biggest challenge instead must have been for those virtuosos to combine this with a more R&B/rock/populist mindset that Miles was at least partially aiming for. If you listen to Tony Williams drumming for examples, he plays SO minimalist by jazz-standards whereas he previously spazzed out a lot. And bassists playing riffs and Miles playing more bluesy as a opposed to modernist and so on... I think that was really the difficult part because many of those guys-Zawinul and McLaughlin excepted-hadn't really been playing like that for many years but had instead been playing fairly abstract stuff...

Also note that a shitload of music Miles recorded in that era including entire sessions were initially scrapped only to come out years later because he didn't think it was good enough and that's another important thing:

Previously, a recording-session in jazz essentially meant that the band walked into the studio with a group of compositions and then they knocked them out and picked the best takes for release (note that many times, they didn't even do alternative takes).

With records like "In a silent way" on the other hand, the method of working became closer to how rock-guys like Brian Wilson (think "Smile") approached recording:You walk into the studio with little else but a specific vibe and "concept" in mind and then let the music take shape there by means of experimenting and editing and then shaping the albums that way by discarding the bad parts and so on.

If you think about it, there wasn't a long tradition of this method of recording in jazz even if Mingus "Black saint and the sinner lady" partially points in that direction but it's of course MUCH more composed and arranged even if he allegedly rearranged and cut out and put in a lot of stuff with post-production methods...