I feel Phantom Blood is the weakest in the series (at least first three of the four so far I've read/seen) because it's an entirely different style/story than the later entries. Phantom Blood is basically torture porn of Jonathan until he gets Hamon and then he kicks Dio's ass (yay!). It has the horror elements that Jojo's entire series gets but here it's more... dreadful? Less played for comedy, which isn't a bad thing but you notice it as the series goes on that Araki changes a lot.
I can see why you'd dislike Battle Tendency but that's where the series sort of picked me up (beyond Dio's "I'VE REJECTED MY HUMANITY JOJO!" and Zapaelli's hamon demonstrations in Phantom Blood) even if the Pillar Men in retrospective kind of fall flat.
We get more information on Hamon (which is sadly dropped next part/Stardust Crusaders for stands), see the organization for that (in a way), get bigger/better battles. Joseph gets tension with the "biological clock"/poison put in him.
The nazi's were a misstep, I feel. Even if the "GERMAN ENGINEERING IS BEST!!!!" character was amazing IMO for the sheer ludicrousness.
A lot of the fanbase feel people should just skip the first two parts (especially Phantom Blood!) and go straight to Stardust Crusaders, but I disagree. The first two arcs each have their strengths and Joseph is (IMO) one of the better "Jojo's" if only because he's a hotblooded ass. Jotaro is similar, but can be a complete asshole, while Joseph cares about people like Jonathan but doesn't have the "gentlemen's agreement" that Jonathan was raised by.
The biggest disappointment though is dropping Hamon for Stands. A lot of the fanbase enjoys "Stands of the Week"/chapter in the manga and that's fine and all but I really miss Hamon and "oh I can't breath/calm down so I can't use this power" tension.
But Araki has grown over time. The nazi's a little less... "glorified?" in later entries and transphobia and the like is dropped because Araki sees the fanbase/diversity and sees what bothers folks and agrees/grows. (Not sure if that's making sense) Which is more than can be said for certain manga artists (*ahem*Toriyama...) in their work.
Edit: IIRC, Lisa Lisa is better written in the manga (but still has a sparse history) than the anime. The anime kind of glosses over her for the Joseph/Kars battle(s)
