Report Details Government’s Ability to Analyze Massive Aerial Surveillance Video Streams

Yesterday I wrote about Dayton Ohio’s plan for an aerial surveillance system similar to the “nightmare scenario” ARGUS wide-area surveillance technology. Actually, ARGUS is just the most advanced of a number of such “persistent wide-area surveillance” systems in existence and development. They include Constant Hawk, Angel Fire, Kestrel (used on blimps in Afghanistan), and Gorgon Stare.

One of the problems created by these systems—which have heretofore been used primarily in war zones—is that they tend to generate a deluge of video footage. A 2010 article says that American UAVs in Iraq and Afghanistan produced 24 years’ worth of video in 2009, and that that number was expected to increase 30-fold (which would be 720 years’ worth) in 2011. Who knows what that’s up to this year, or where it will be by, say, 2025. The human beings who operate these systems can’t possibly analyze all that footage.

In an attempt to solve this problem, Lawrence Livermore Labs has created a system for the military called “Persistics.” It can be used in conjunction with drone (or manned) camera systems such as ARGUS to help manage the vast oceans of video data that are now being generated. The system is

designed to help the Department of Defense and other agencies monitor tens of square kilometers of terrain from the skies, with sufficiently high resolution for tracking people and vehicles for many hours at a time.

That’s from a May 2011 report that I recently came across with the faintly ominous title “From Video to Knowledge.” Produced by Livermore Labs, it contains a lot of interesting detail about Persistics and the problems and solutions involved in massive aerial video surveillance.

The Persistics system consists of algorithms that “analyze the streaming video content to automatically extract items of interest.”

Its analysis algorithms permit surveillance systems to “stare” at key people, vehicles, locations, and events for hours and even days at a time while automatically searching with unsurpassed detail for anomalies or preselected targets.

With Persistics, the report boasts, “analysts can determine the relationships between vehicles, people, buildings, and events.” Among the capabilities touted in the report are:

• “Seamless stitching” together of images from multiple cameras to create “a virtual large-format camera.”
• Stabilizing video (“essential for accurate and high-resolution object identification and tracking”).
• Eliminating parallax (the difference in how an object appears when viewed from slightly different angles).
• Differentiating moving objects from the background.
• The ability to automatically follow moving objects such as vehicles.
• Creating a “heat map” representation of traffic density in order to “automatically discern if the traffic pattern changes.”
• Comparing images taken at different times and automatically detecting any changes that have taken place.
• Super-high “1,000-times” video compression.
• The ability to provide all the locations a particular vehicle was spotted within a given time frame.
• The ability to provide all the vehicles that were spotted at a particular location within a given time frame.

Technologically, according to the report, the Persistics program relies heavily on the explosion in the power of consumer Graphics Processing Units (GPUs) used in video games and the like.

The report also says that the system “is being further enhanced” to work with ARGUS, and includes new details about that system:

Persistics can simultaneously and continuously detect and track the motion of thousands of targets over the ARGUS-IS coverage area of 100 square kilometers. ARGUS-IS can generate several terabytes of data per minute, hundreds of times greater than previous-generation sensors.

Previous reports said that ARGUS could cover 15 square miles; here it reports 100 square kilometers, which is 38.6 square miles. (I suppose we should expect Moore’s Law-like expansion in the capabilities of these systems.)

Of course, the system is designed to store and retrieve all the records and data about everything that it surveils:

Persistics supports forensic analyses. Should an event such as a terrorist attack occur, the archival imagery of the public space could be reviewed to determine important details such as the moment a bomb was placed or when a suspect cased the targeted area. With sufficiently high-resolution imagery, a law-enforcement or military user could one day zoom in on an individual face in a heavily populated urban environment, thus identifying the attacker.

As with every privacy-invading technology designed and/or sold as helping foil terrorists, we have to wonder how long it will be before it’s applied to tracking peace activists.

Future work on Persistics is focused on the kind of behavioral analytics that have been discussed in the context of programs such as “Trapwire.” Livermore scientists, according to the report, are now working on automated methods for identifying “patterns of behavior” that could indicate “deviations from normal social and cultural patterns” and “networks of subversive activity.”

Also under development are efforts to allow the three-dimensional viewing of targets, as well as “methods to overlay multiple sensor inputs—including infrared, radar, and visual data—and then merge data to obtain a multilayered assessment.”

Of course, much of this is unobjectionable from a domestic civil liberties point of view when it’s used as originally intended: on foreign battlefields. The problem comes when the government brings the technology home and turns it inward upon the American people. In fact, at the close of the report, Livermore contemplates exactly that:

Unmanned aircraft have demonstrated their ISR [intelligence, surveillance, and reconnaissance] value for years in Afghanistan and Iraq. As U.S. soldiers return home, the role of overhead video imagery aided by Persistics technology is expected to increase. Persistics could also support missions at home, such as monitoring security at U.S. borders or guarding ports and energy production facilities. Clearly, with Persistics, video means knowledge—and strengthened national security.

Among the federal agencies most interested in the technology, the report says, is DHS.