You have to keep IM, Cap, and Thor because they are the true core. The rest you can use to punctuate drama around these guys.
Example: Ultron - The Avengers foremost threat. He captures Widow and just leaves her alone (and let's not add in that he leaves her with a comm's device). He should have murdered her, she wasn't being used for any leverage. Problem is, Scarlett Johanson is so popular among women aged 15-30 they wouldn't dare risk killing off that money. It's good business sense but the movie becomes weak.
Example 2: Civil war - Rhodey falls about 10,000 feet and the worst thing that happens is that Tony gives him robot legs. That's 10 types of garbage. He should have been murdered by Winter Soldier to raise the dramatic rift between Tony and Steve and give real pay off at the end, not "hey, yeah, I knew the Manchurian candidate did it, but it's all cool, 2 minutes ago you were OK with his innocence and his mind control so you know. Wait, what, we're fighting now?".
Example 3: Thor Dark World - well, well, they actually kill Thor's mother but he just ignores it, so they completely write around any impact the murder of his mother in front of his eyes would have because the levity is too much. So Thor's character arc is just hollow and no one cares for that movie.
Example 4: IM3 - Pepper gets killed. Oh wait no she doesn't, she's saved by the magic formula. So instead of Tony going into a spiral of depression where his IM technology could have led to the death of Pepper, and led him into down a dark path of creating Ultron and going through a redemption arc, they just write that all's cool again. No one is in danger ever, have a few laughs, next movie.
Now they're hugely successful, and will be dearly remembered, but they're thin as f***