Scotsman Francis Bannerman - an arms dealer, supposedly owning one of the largest if not the largest arsenal in the word, collected anything historic, especially from the civil war era. His original warehouses were in NYC. The ordinance was very dangerous and over time, unstable. The city told him to find somewhere else to keep it - so Pollopel Island it was. "There wasn’t a rifle, uniform, howitzer, sidearm, mortar, canteen, blanket, tomahawk, bayonet, machete, pith helmet, or suit of armor that he wouldn’t buy for a hundredth of a penny on the dollar. If he didn’t melt the metal down as scrap, he resold the stuff to just about anyone—from civilians who walked in off the street to Far East armies and banana republics. Sometimes he dealt with assumedly more sophisticated buyers; in 1917, he tried to sell the U. S. Navy 30 six-inch naval rifles for $450,000, which he’d bought for $78 each. (He lowered the price to $150,000 when several Congressmen threatened an investigation.)"

https://hvmag.com/life-style/bannerman-island-castle/

His dream was to live in a castle. Not being a builder or architect, the arsenal building that he built on the island was not the soundest. After he died, there was a black-powder explosion in a magazine that took out just about all the windows on the island. Vandals wrecked the place, weather further battered it - and a fire which burned for 3 days (no one would go near it to put it out and by then it was abandoned) finished most of the island off. His wife had planted gardens there.

Today they are trying to salvage the place. The arsenal building is too dangerous to enter as its too unstable. You can go inside his rather gloomy 'castle' residence - and I mean gloomy. Beautiful sunsets can be seen from the island though.

He had a harbor there at one time - most of it is rotted away/sunk. The entrance to the harbor is still there - towers of sorts.

Waters of the Hudson have strong currents at times (even more so down by Fort Lee/Edgewater). Rocks are close to the surface in places. Hudson is a lot cleaner in my neck of the woods - you can swim in it around here.