Members of a secret police unit that illegally gathered information on grieving families are unlikely to face action, Derbyshire’s chief constable, Mick Creedon, has said.

Creedon is responsible for overseeing Operation Herne, an investigation into the Special Demonstration Squad (SDS), an undercover unit attached to London’s Metropolitan Police. An adjunct of the secret intelligence service MI5, it operated from 1968 to 2008, gathering information on left and right-wing parties. It was closed down after it was revealed that it had used the names of 80 dead children to create fake identifies for its undercover operatives who often went “deep cover” for years, even fathering children with unsuspecting activists.

Operation Herne reported earlier that undercover units had either stolen or concocted 106 identities as part of their infiltration.

Last year, former SDS operative Peter Francis revealed that the unit had spied on the family of Stephen Lawrence, murdered in a racist attack in 1993. The aim was to smear the family, who had led a long campaign to expose police inaction and corruption in the failure to prosecute their son’s killers.

Amid public outcry, Home Secretary Theresa May announced the internal inquiry in 2011. It is still ongoing.

Last week, Creedon revealed that the SDS had compiled information on many justice campaigns for murder victims and those who had died following contact with the police. In addition to the Lawrence campaign, evidence had been found of the SDS holding covert information on a further 17 groups, Creedon revealed, although he would not disclose their names. The information had been gathered between 1970 and 2005.

Among those thought to be targeted are the Justice for Jean Charles de Menezes, the young Brazilian electrician executed on the London Underground on July 22, 2005. Plainclothes police officers shot him repeatedly in the head during a covert shoot-to-kill “anti-terror” operation. No police officer was ever charged with his murder.

Information was also gathered by the SDS on the family of Cherry Groce, whose shooting by police in 1985 led to the Brixton riots that year; the campaigns for justice for Harry Stanley, shot dead in Hackney, east London, in 1999 after police mistook a table leg he was carrying for a shotgun and for Wayne Douglas, who died in police custody in 1995.

The unit also placed the family of Ricky Reel under surveillance. Reel, a 20-year-old student, went missing in 1997 during a night out with friends. The group had been forced to split up after being racially attacked by white youths and Reel disappeared. His body was found one week later in the River Thames. His family has waged a long-running campaign against the police’s insistence that Reel’s death had nothing to do with the assault.

Creedon states that the unit had “lost [its] moral compass” and became “a force within a force.”

“My report is very clear that criticism must be levelled at the MPS [Metropolitan Police] for keeping information, which had been gathered by undercover officers, which served no purpose in preventing crime or disorder.”

Despite such statements, he told the Guardian that it was unlikely any police officer would face action. “I don’t think anyone sees this as a misconduct issue,” he said, as they had merely been carrying out instructions.

Amid claims that the intelligence on the families was “hoovered up” accidentally, a source told the newspaper that this was the outcome of “collateral” intrusion as the SDS sought to infiltrate left-wing groups. The SDS was first formed as the Special Operations Squad in 1968 under the Labour government of Harold Wilson to spy on the antiwar movement. Massively expanded in 1972—at a time of rising working class militancy against the Conservative government—it was renamed the SDS.

Creedon states in his report, “Over the 40 years that the unit existed, senior Metropolitan police management of the day either knew nothing about the existence and activities of the unit or, when they did, they appeared to have allowed the SDS to exist in secret isolation in a manner that was complacent and possibly negligent.”

Claims by senior management that they knew nothing cannot be taken at face value. The unit was allowed to operate as a law unto itself because its task was to protect the state and political establishment against any opposition deemed to be a political threat.

The SDS was considered such a “success story” that Paul Condon, Metropolitan Police Commissioner between 1993 and 2000, praised the unit personally.

According to the Guardian’s source, “It was worse than out of control. It was actually a force within a force, operating to set of standards and ethics more suited to guerrilla warfare than modern policing.”

SDS operatives concentrated on infiltrating “extreme left wing/extreme right wing” groups and “animal rights/environmental” activists. Hundreds of organisations were targeted over the four decades. Likened by Creedon to the “human equivalent of a covert listening device,” they spent years posing as activists. They not only gathered information but organised provocations against the groups they were deep cover in.

Among those known to be targeted are the Socialist Party of England and Wales’ predecessor organisation, the Militant group, and its youth organisation, Youth Against Racism in Europe (YRE), in the mid-1990s.

Peter Francis was “Officer A” from the SDS who infiltrated the YRE after the murder of Stephen Lawrence. Details of his activities only became public after Francis sued the Metropolitan Police and won an out-of-court settlement after claiming he had suffered post-traumatic stress disorder as a result of his undercover activities.

He said that anger at police inaction over Stephen Lawrence’s murder meant the state feared “losing control”. Given a new identity and apartment, he lived the life of a “deep swimmer” for four years, becoming a YRE branch secretary, where he had access to details on the organisation’s membership, campaign plans and finances.

Officer A also admitted organising provocations, ensuring that protests soon became violent and open to police attack.

SDS recruit Mark Jenner, who went under the identity “Mark Cassidy”, infiltrated the Colin Roach Centre in the 1990s. Roach was a 21 year-old black man who died in police custody in 1983. “Cassidy” formed a four-year relationship with a woman unaware of his undercover activities and fathered a child with her.

She is one of five women claiming damages from the Metropolitan Police after being tricked into sexual relationships with undercover operatives. Earlier this month, the five won an important legal victory in their campaign for compensation when the High Court ruled that police could no longer rely on issuing a “neither confirm nor deny” (NCND) response to their claims. One of the women has described being duped into a relationship with one officer as being “raped by the state.”

The latest report confirms the massive extent of a police surveillance network, monitoring political protesters, activists and organisations critical of the state. It comes on top of the revelations by National Security Agency whistleblower Edward Snowden as to how the US and British intelligence agencies secretly spy on the electronic communications of people throughout the globe.

On July 18, the British Parliament fast-tracked into law the Data Retention and Investigative Powers Act (DRIP) giving the UK government even greater powers of surveillance, including compelling Internet and phone companies to store all the communications data of British citizens for 12 months and to make the information freely accessible to police and intelligence agencies.