The city of Dallas had no safety net to prevent an employee from accidentally deleting huge amounts of data. This is the main takeaway from the city’s investigation into how over 20 terabytes of police files were deleted during a migration from a cloud storage system to an internal server. The city of Dallas has never implemented a formal storage strategy, although it has posted it on its website for the public to see. The report presents managers as absent or apathetic and points out to the city gaps in training, culture and management.
In August, we uncovered incidents in March and April where a computer scientist had the power to wipe out millions of police data while migrating from the cloud to a local server, which the city was doing to save money. silver. The city council and the district attorney were not notified for four months; the media learned about it after public prosecutor Jean Creuzot alerted county defense lawyers.
The liberated city his report on what happened Thursday. (And posted it in… Calibri police? Really?) It’s a chess chronicle: The employee ignored city system warnings while deleting files. The city has “no defined rules” on how to archive backed up data, as it hasn’t implemented anything from a strategy document that explained how to archive backed up data. And then three IT managers signed off on the individual’s work, but they apparently didn’t understand what they were signing.
“[T]Those responsible for ITS infrastructure services did not understand the actions to take, the potential risk of failure, or negligently reviewed the change request before providing authorization and approval to proceed with the change request. modification. “
“We understand the scale and gravity of the situation and will continue to work diligently with DPD to recover as much data as possible. To date, the recovery team has recovered 140,353 potential files that have been deleted. “
Six digits! That sounds like a lot until you see the true scale of the deletion: The city estimates that 8.7 million files have been deleted, which is over 20 terabytes. Of these, 4.6 million could be recoverable if stored elsewhere, such as on a police laptop. Most of what was defeated concerned the Domestic Violence Unit, “information gathered by DPD detectives for ongoing, tried and adjudicated cases”. The prosecutor’s office is still unsure of the potential impact on these cases, but the report says 17,494 were affected. The prosecutor’s office has classified 1,000 of them as priority cases.
Some of the findings from this 131-page report are mind-boggling: The city has a strategy paper on how it should handle data on its external website, but that document is “outdated” and, oh yeah, “it doesn’t. has not been implemented as a formal activity and process in the City’s data environment.
The city does not regularly check its huge volume of data to ensure that it is properly backed up.
“The lack of a comprehensive, centralized and enforced data management capability managed and enforced by a data governance committee is a factor in the conditions that have led to an environment in which data loss is possible. This failure, especially when it came to unstructured data, was one major factor, among many others, that resulted in the loss and the resulting inability to recover the data from the Dallas Police Department. .
The kicker here? The city knew what to do. The data management provider used by the city, Commvault, provided “detailed documentation” on how to migrate the data. The employee did not follow him and his bosses “had insufficient monitoring of the migration”. This “inadequate monitoring … directly contributed to the loss.” What’s more, “[i]Instructions were never presented, read, or reviewed by technicians and management prior to actions leading up to data deletion, ”and IT executives did not need routine training to stay up to date on“ data deletion ”. latest knowledge on these functions. “
The entire report is filled with these anecdotes: bad oversight, bad culture, bad management and very few guarantees. There was a fire in one bedroom and the city closed the door, entered the living room, and turned on the Mavericks game. Then they acted surprised when the hallway was on fire.
The report points to the problem at the top of the governance structure of the town hall:
“There are gaps in documented management guidelines and clear expectations regarding management control systems instancing and commitment to the core values of excellence, ethics, empathy and fairness of the City of Dallas. “
It is not finished. The FBI is investigating because Dallas Police lacked the knowledge to determine if a crime had been committed. (Which is fair. Strange situation.) The city is going to hire a third party to investigate and recommend changes which, judging by the city’s report, should probably start from scratch.