"I've always known him to be a person who'd been alone," DeJesus said. "I never pried into his personal life."
Some neighbors suspicious
But is there anything that could, or should, have tipped neighbors and police off that Castro was hiding something?
It's not as if he had ever been convicted of any crime. Neither firefighters nor ambulances were called to his house, police say, and there were no building code violations or complaints.
Authorities say they went to his house twice. The first time was in 2000, when Ariel Castro himself called to report a fight outside. Four years later, police came to follow up on a report that he had left a child on his school bus at the depot. They found no criminal intent, Deputy Chief Ed Tomba of the Cleveland Police Department said.
One neighbor, Nina Samoylicz, said that one night two years ago, she and some friends saw a naked woman in Castro's backyard. They tried to speak to her, and a man told the woman to get down, she said.
Samoylicz said she called police but said they weren't taken seriously, and Castro erected tarps in his backyard a week or two later. But Sgt. Sammy Morris -- a police spokesman -- denied to CNN on Tuesday that any such call about a naked woman was made.
Neighbor Lugo said he heard yelling in the house in November 2011.
But Marti, who said he was always outside, didn't hear anything, not even a "pin drop around the house" in the decades he lived near Castro.
Reflecting back, though, Marti said Castro would start walking toward the street or the front of his house -- away from earshot of the young women -- when they talked.
"Now that I think of it, he didn't want nobody back there."
Neighbor: 'Breaks my heart that I could have done anything'
While some characterized Castro as upbeat and outgoing, his uncle described him as withdrawn.
The uncle, Julio Castro, said Ariel isolated himself from his extended family after his father died in 2004. The uncle lives just half a block from the house where the women were rescued but had not spoken to Ariel in years.
On Monday, he saw a commotion down the street and walked over there, where police told him about the rescue.
His first thought was that it was impossible that his nephew had anything to do with it. But, Julio Castro told CNN en Español, "perhaps, he was the type of person who was living two lives."
Marti also suspects, now, that there may have been more to Ariel Castro than what he'd believed.
As someone with grandchildren himself, who prided himself in being a good neighbor who watched out for others, Marti said that it hurts that he didn't notice the signs and act to help those three young women.
One of them, Georgina DeJesus, is the daughter of a friend he's known since the 1970s. For years, Marti said he prayed for her -- and finally his prayers were answered, just not how he ever envisioned.
"It breaks my heart that I couldn't have done anything for them girls," Marti said.